Format of options file

[RS]

Is the format of the options file different in Windows and Linux? I am
setting up my machines to dual boot between Windows 10 and ubuntu 16.04.
On my main machine I wanted to copy the history file. I thought while
I was doing that I might as well copy the whole hidden .get_iplayer
directory including the cache and options files. I expected to have to
change the --output option since the path is different, but some of the
options don't work, and others do.

The criterion seems to be that options with a numeric or boolean value
like --subtitles or --expiry work.

Options with a string value like --tvmode, --radiomode or
--ffmpegloglevel don't. For example with --tvmode=hlshd I get

INFO: Downloading tv: 'Collateral: Series 1 - 2. Episode 2 (b09sym6m)
[original]'
) available for this programme with version 'original'
INFO: Available modes:
hlshd,hlsvhigh,hvfhd,hvfsd,hvfxsd,hvfhigh,hvfxhigh,hvfstd,hvflow

When I re-enter --prefs-add --tvmode=hlshd
--ffmpeg-loglevel=info gives an error

INFO: Downloading tv: 'Collateral: Series 1 - 2. Episode 2 (b09sym6m)
[original]'
') - using defaultalue for --ffmpeg-loglevel ('info

If I enter the options again with --prefs-add they work correctly, so it
is no big deal. Any ideas what is happening?

[Ralph Corderoy]

Hi Richard,

Post by RS
') - using defaultalue for --ffmpeg-loglevel ('info
If I enter the options again with --prefs-add they work correctly, so
it is no big deal. Any ideas what is happening?

That line above seems to be overprinting the «')» at the start of the
line suggesting there's an ASCII CR, carriage return, after the `info'
as if a Unix system was reading a POSIX text file of lines ending in
ASCII LF, linefeed, but being fed a DOS one with CR LF pairs.

`od -c foo' will show them, and they can be removed with `sed -i
's/\r$//' foo'.

[RS]

Post by Ralph Corderoy
Hi Richard,

Post by RS
') - using defaultalue for --ffmpeg-loglevel ('info
If I enter the options again with --prefs-add they work correctly, so
it is no big deal. Any ideas what is happening?

That line above seems to be overprinting the «')» at the start of the
line suggesting there's an ASCII CR, carriage return, after the `info'
as if a Unix system was reading a POSIX text file of lines ending in
ASCII LF, linefeed, but being fed a DOS one with CR LF pairs.
`od -c foo' will show them, and they can be removed with `sed -i
's/\r$//' foo'.

Thanks, Ralph. That does indeed seem to be the problem. Thanks also
for the command to remove the unwanted CR characters. That seems to
have cleared everything up.

Best wishes
Richard

[Colin Law]

Post by Ralph Corderoy
Hi Richard,

Post by RS
') - using defaultalue for --ffmpeg-loglevel ('info
If I enter the options again with --prefs-add they work correctly, so
it is no big deal. Any ideas what is happening?

That line above seems to be overprinting the «')» at the start of the
line suggesting there's an ASCII CR, carriage return, after the `info'
as if a Unix system was reading a POSIX text file of lines ending in
ASCII LF, linefeed, but being fed a DOS one with CR LF pairs.

Are we certain that the problem has been correctly analysed? The line
shown above appears to have a CR after info and before the quote sign,
which would not be valid on either system. In practice it may work on
Windows if the code just ignores CR characters wherever they are. Is
it certain that a correctly formatted windows file will definitely not
work on Linux?

Post by Ralph Corderoy
Hi Richard,

Post by RS
') - using defaultalue for --ffmpeg-loglevel ('info
If I enter the options again with --prefs-add they work correctly, so
it is no big deal. Any ideas what is happening?

That line above seems to be overprinting the «')» at the start of the
line suggesting there's an ASCII CR, carriage return, after the `info'
as if a Unix system was reading a POSIX text file of lines ending in
ASCII LF, linefeed, but being fed a DOS one with CR LF pairs.
`od -c foo' will show them, and they can be removed with `sed -i
's/\r$//' foo'.
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
_______________________________________________
get_iplayer mailing list
http://lists.infradead.org/mailman/listinfo/get_iplayer

[Ralph Corderoy]

Hi Colin,

Post by Colin Law

Post by Ralph Corderoy

Post by RS
') - using defaultalue for --ffmpeg-loglevel ('info

That line above seems to be overprinting the «')» at the start of
the line suggesting there's an ASCII CR, carriage return, after the
`info' as if a Unix system was reading a POSIX text file of lines
ending in ASCII LF, linefeed, but being fed a DOS one with CR LF
pairs.

Are we certain that the problem has been correctly analysed?

Yes, I did. :-)

Post by Colin Law
The line shown above appears to have a CR after info and before the
quote sign, which would not be valid on either system.

Because the value is «info\r» and it's being printed surrounded by
«('...')».

WARNING: invalid value for --ffmpeg-loglevel ('$opt->{ffmpegloglevel}') - using default\n

becomes

WARNING: invalid value for --ffmpeg-loglevel ('info
') - using default

becomes

') - using defaultalue for --ffmpeg-loglevel ('info

[Colin Law]

Post by Ralph Corderoy
Hi Colin,

Post by Colin Law

Post by Ralph Corderoy

Post by RS
') - using defaultalue for --ffmpeg-loglevel ('info

That line above seems to be overprinting the «')» at the start of
the line suggesting there's an ASCII CR, carriage return, after the
`info' as if a Unix system was reading a POSIX text file of lines
ending in ASCII LF, linefeed, but being fed a DOS one with CR LF
pairs.

Are we certain that the problem has been correctly analysed?

Yes, I did. :-)

Post by Colin Law
The line shown above appears to have a CR after info and before the
quote sign, which would not be valid on either system.

Because the value is «info\r» and it's being printed surrounded by
«('...')».
WARNING: invalid value for --ffmpeg-loglevel ('$opt->{ffmpegloglevel}') - using default\n
becomes
WARNING: invalid value for --ffmpeg-loglevel ('info
') - using default
becomes
') - using defaultalue for --ffmpeg-loglevel ('info

Yes, I agree. For anyone who knows the code it should not be
difficult to ignore CR characters after the text. I expect it is
already ignoring white space after it. Any patch takes time to work
out and to test though.

Colin

Post by Ralph Corderoy
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
_______________________________________________
get_iplayer mailing list
http://lists.infradead.org/mailman/listinfo/get_iplayer

[RS]

Post by Colin Law
Yes, I agree. For anyone who knows the code it should not be
difficult to ignore CR characters after the text. I expect it is
already ignoring white space after it. Any patch takes time to work
out and to test though.

For history (I haven't yet found where it processes the options file) it
uses chomp at line 3717 (v3.12) to strip off any unwanted characters.
If I have understood this article correctly
http://perldoc.perl.org/perlport.html#Newlines
chomp should work portably with default settings. I am not sure I have
fully understood the article it links to about chomp VARIABLE, but I
can't see anywhere the variable is set.

Best wishes
Richard

[MacFH - C E Macfarlane]

Please see below ...

Post by Ralph Corderoy
Hi Colin,

Post by Colin Law

Post by Ralph Corderoy

Post by RS
') - using defaultalue for --ffmpeg-loglevel ('info

That line above seems to be overprinting the «')» at the start of
the line suggesting there's an ASCII CR, carriage return, after the
`info' as if a Unix system was reading a POSIX text file of lines
ending in ASCII LF, linefeed, but being fed a DOS one with CR LF
pairs.

Are we certain that the problem has been correctly analysed?

Yes, I did. :-)

Post by Colin Law
The line shown above appears to have a CR after info and before the
quote sign, which would not be valid on either system.

Because the value is «info\r» and it's being printed surrounded by
«('...')».
WARNING: invalid value for --ffmpeg-loglevel ('$opt->{ffmpegloglevel}') - using default\n
becomes
WARNING: invalid value for --ffmpeg-loglevel ('info
') - using default
becomes
') - using defaultalue for --ffmpeg-loglevel ('info

Seems valid reasoning to me.

Some people may be misunderstanding the precise meaning of 'Carriage
Return'.  I'm not 100% up on my telecoms history, but AIUI ...

CR, LF, etc are ASCII acronyms, it itself being one for American
Standard Code for Information Interchange,
https://en.wikipedia.org/wiki/ASCII, which was a coding designed
originally to allow early telecoms equipment to inter-operate. 'Carriage
Return', CR, meant precisely that  -  in other words, move the slidable
teleprinter carriage holding the paper to the right so that the fixed
print head is at the left margin position.  LineFeed, LF, also meant
exactly that, rotate the carriage roller to move the printing position
to the next line.  Thus to print a new line of teleprinter output, both
a CR and a LF are required.  DOS followed by Windows have always stuck
by the original meaning of these codes, and thus both use both codes at
the end of lines of text.  Arguably the oddballs are Linux and perhaps
Macs, using only one.  This would mean that if their text was sent
directly to a teleprinter, the output would not be what is required, but
of course these days that would seldom if ever be done.  However, when
printing CRs to the Linux console, as Ralph has explained above, Linux
still retains the original meaning of CR, and moves the cursor back to
the beginning of the line, and in the absence of a line feed immediate
following, any other text following overprints  the beginning of the
same line.  Linux could be argued to have subverted both codes, because
not only does a LF move the cursor to the next line of text, but an
implicit CR is also performed, which is why having only a LF at the end
of each line of text works in Linux, but what really throws the spanner
in the works is that Linux counts a CR preceding a LF as part of the
text of the line, not as part of the line termination, hence the
original problem and subsequent rather convoluted thread.

[Bernard Peek]

Post by MacFH - C E Macfarlane
Seems valid reasoning to me.
Some people may be misunderstanding the precise meaning of 'Carriage
Return'.  I'm not 100% up on my telecoms history, but AIUI ...

This is a long-standing problem for any organisation that exchanges text
files between systems using different formats. When I wrote interface
formats I mandated that a receiving program will handle any permutation
of <CR> <LF> in any permutation. It's even a problem for places that
think XML solves all standards problems.

[Ralph Corderoy]

Hi MacFH,

'Carriage Return', CR, meant precisely that  -  in other words, move
the slidable teleprinter carriage holding the paper to the right so
that the fixed print head is at the left margin position.  LineFeed,
LF, also meant exactly that, rotate the carriage roller to move the
printing position to the next line.

Those of us that used manual typewriters, that pre-date fangled
teleprinters, will recognise the two actions that could be done by the
lever to the left. It would feed the paper to the next line when
pressed, and when continued to be pushed it would slide the carriage
along so it returned to the start. It was the `Line feed/Carriage
return lever'.

Arguably the oddballs are Linux and perhaps Macs, using only one. 
This would mean that if their text was sent directly to a teleprinter,
the output would not be what is required, but of course these days
that would seldom if ever be done.

But in those days of Multics and Unix that's precisely what was done
with Unix text files, terminated by just LF, and yet the text didn't
stagger across the page because a device driver was correctly the thing
that knew what a particular device needed to be sent as bytes; the file
format wasn't the place for that. Other characteristics include a
timing delay to allow for the carriage to return before sending more
bytes.

Linux could be argued to have subverted both codes, because not only
does a LF move the cursor to the next line of text, but an implicit CR
is also performed

This happens in a terminal window for the same reason it happens on a
teleprinter; the TTY is so configured and the device driver obeys. I
can say this particular device doesn't need a CR sent to it before every
LF.

$ stty onlcr; printf 'foo\nbar\n'; stty -onlcr; printf 'foo\nbar\n'
foo
bar
foo
bar
$

[MacFH - C E Macfarlane]

We could argue about this for hours, but I have no desire to do so  - 
there are things I like and others I detest about both OSs, and the
thread's becoming increasingly OT.  I answer one specific point below,
but anyone not interested in computer history and debugging programs can
safely ignore ...

Post by Ralph Corderoy

Arguably the oddballs are Linux and perhaps Macs, using only one.
This would mean that if their text was sent directly to a teleprinter,
the output would not be what is required, but of course these days
that would seldom if ever be done.

But in those days of Multics and Unix that's precisely what was done
with Unix text files, terminated by just LF, and yet the text didn't
stagger across the page because a device driver was correctly the thing
that knew what a particular device needed to be sent as bytes; the file
format wasn't the place for that. Other characteristics include a
timing delay to allow for the carriage to return before sending more
bytes.

Don't forget that the earliest teleprinter style devices date back as
far as at latest WW2, for example the German Lorenz machine which also
applied cryptography  -  and I don't believe the concept of
'device-driver' existed then; nor am I questioning the use of device
drivers, obviously they make a lot of programming sense.  What I'm
questioning is the wisdom, perhaps that should be rank stupidity, of
using LF as the line-terminator, while treating a preceding CR as part
of the line text.  That completely f*d backward compatibility with what
had gone before.  You say that it is C based, but my recollection of C's
internal representation is that strings are NULL terminated, and that
also might have been a more logical choice of line-terminator for Unix.

At any rate, we have the mess we have, and I join with Bernard in
deploring it.  I can think of at least two baffling examples that have
afflicted me in the past, one in each direction ...

The first concerned a Psion Gold modem PCMCIA card, installation of the
Windows drivers for which broke not only use of that card, but any
previously similar card using the same INI file.  I wasn't the person
who wrote this original installation script, from what I can recall that
had been, shudder, a contractor, but the mess was given to me to sort
out.  After I'd corrected all the usual contractor horrors, like failing
to get the Windows and Temp directory names from the environment, and
the hard-coding of them and other non-constant-across-different-machines
values repeatedly all the way through the script, I was still left with
a script that didn't work.  Now that I had cleaned it up, the script
appeared to be doing everything it should have, and should have worked
on any machine in the firm, and the resulting INI file looked alright,
yet nothing was working.  It occurred to me that perhaps there was some
hidden character in the INI file supplied by Psion, which we were
editing via the script, so I loaded it as hex into TextPad and, lo and
behold, Psion had supplied a *Windows* INI file with *Unix* line
endings!  Obviously, I could have just loaded the file as text into
Textpad and saved it with Windows line-endings, but this would not have
helped the many people who had already tried the installation and had it
fail.  I had to adapt the script further to clean up any pre-existing
mess from a previous installation by correcting the line endings in
their existing INI file.  Thus this incompatibility between line endings
created a ridiculous amount of lost productivity and extra work.

The second occasion was that I created a Linux bash script on a Windows
machine using TextPad, but failed to remember to save it with Unix line
endings.  Because it was interpreting the CRs as part of the line, the
errors given were baffling.  I tried posting the script in one or more
Linux newsgroups, but nobody could suggest what might be going wrong,
and I moved on to other things for a while.  It was only when I came
back to it and started trying things almost at random, that after a few
hours I came to realise what the problem was.  Again, so much time
wasted, for so little a reason.

[Ralph Corderoy]

Hi MacFH,

nor am I questioning the use of device drivers, obviously they make a
lot of programming sense.  What I'm questioning is the wisdom, perhaps
that should be rank stupidity,

You're saying Unix's core designers are stupid for choosing to stick
with text files as lines terminated by LF. There's some Sherlock Holmes
quote about `It takes talent to recognise genius...'. :-)

of using LF as the line-terminator, while treating a preceding CR as
part of the line text.  That completely f*d backward compatibility
with what had gone before.

Much of Unix broke orthodoxy, and Unix wasn't the first to punt device
dependencies out of the logical file format and into a device driver
that needed to know about the exceptions and oddities of the physical
world. Nor was it the first major OS to use LF-terminated lines.

You say that it is C based, but my recollection of C's internal
representation is that strings are NULL terminated,

They are, but that it not relevant.

and that also might have been a more logical choice of line-terminator
for Unix.

Only if a C string was to be prohibited from holding more than one line
of text. C allows the bytes "line 1\nline 2\n\0" as a NUL-terminated
string. And only if a C string "foobar" could no long be split into
two, "foo", "bar", without losing the fact that they are not both lines
that have been read.

C's standard library widely expects LF-terminated lines, and translates,
as I've said twice now in replies to Richard, from the host's
representation to that on reading and back on writing where it differs,
e.g. CP/M and its descendants.

You're drawing conclusions from sketchy knowledge and thus not able to
follow through their implications. Sorry, but there's no point to this
conversation.

[RS]

Post by RS
Is the format of the options file different in Windows and Linux?

...

Post by RS
The criterion seems to be that options with a numeric or boolean value
like --subtitles or --expiry work.
Options with a string value like --tvmode, --radiomode or
--ffmpegloglevel don't.

Looking at the files with xxd, the lines in the Windows file end with
CRLF (hex 0D0A). In file copied to Linux with re-entering of some of
the options with --prefs-add there is a mixture of lines ending CRLF and
LF, even though the lines have been shuffled by --prefs-add. There does
not seem to be any correlation between the errors and the presence of CR
characters.

I have not yet looked at the code to see how the options file is parsed.
What is puzzling is that the options display correctly with --prefs-show.

The Windows download_history file seems to work correctly. I hope I
don't need to edit it.

[James Scholes]

The Windows download_history file seems to work correctly.  I hope I
don't need to edit it.

You've probably edited your options file by hand in the past, resulting
in a mixture of weird line endings from a Windows editor, and
appropriate ones when get_iplayer itself has updated it. That's just my
guess after going through exactly the same thing recently. I'd bet you
probably don't go editing your download_history file by hand very often
so it's most likely fine.

_______________________________________________
get_iplayer mailing list
http://lists.infradead.org/mailman/listinfo/get_iplayer

Regards,

James Scholes
https://twitter.com/JamesScholes

[Bernard Peek]

Post by James Scholes

The Windows download_history file seems to work correctly.  I hope I
don't need to edit it.

You've probably edited your options file by hand in the past,
resulting in a mixture of weird line endings from a Windows editor,
and appropriate ones when get_iplayer itself has updated it.  That's
just my guess after going through exactly the same thing recently. 
I'd bet you probably don't go editing your download_history file by
hand very often so it's most likely fine.

For what it's worth I noticed that under Windows the Wordpad program
seems to understand UNIX style terminations and retains them.

[MacFH - C E Macfarlane]

Please see below ...

Post by Bernard Peek

Post by James Scholes

The Windows download_history file seems to work correctly.  I hope I
don't need to edit it.

You've probably edited your options file by hand in the past,
resulting in a mixture of weird line endings from a Windows editor,
and appropriate ones when get_iplayer itself has updated it.  That's
just my guess after going through exactly the same thing recently. 
I'd bet you probably don't go editing your download_history file by
hand very often so it's most likely fine.

For what it's worth I noticed that under Windows the Wordpad program
seems to understand UNIX style terminations and retains them.

Yes, so does the excellent tryit-ware program Textpad, which is
relatively cheap to license fully anyway, and for that you get just
about the best Windows programming editor there is.

But as the OP is changing over to Linux, perhaps the de facto standard
'dos2unix' would be a more appropriate recommendation. There are Windows
versions, but almost certainly it would be available from the original
Linux distro repository.  I can't recall what flavour of Linux the OP is
running, but in Ubuntu and similar distros ...
        sudo apt-get install dos2unix
... should install it, and ...
        man dos2unix
... should give instructions for using it.

[RS]

Post by MacFH - C E Macfarlane
Please see below ...

Post by James Scholes

The Windows download_history file seems to work correctly.  I hope I
don't need to edit it.

You've probably edited your options file by hand in the past,
resulting in a mixture of weird line endings from a Windows editor,
and appropriate ones when get_iplayer itself has updated it.  That's
just my guess after going through exactly the same thing recently.
I'd bet you probably don't go editing your download_history file by
hand very often so it's most likely fine.

Thanks for all the comments. I don't remember ever having edited the
options file manually. Until today I did not even know the format. In
any event I have now looked at the original Windows options file. It
uses 0x0D0A (CRLF) as a line terminator throughout. The 0x0A
terminators have been introduced by running --prefs-add in Linux but it
does not replace all the 0x0D0A terminators.

I have also just looked at the last few entries in the Windows
download_history. They have 0x0A as a line terminator. I used gedit to
save a manageable number of entries, so it is possible gedit changed the
line terminators. To check that I edited and saved the Windows options
file with gedit. The 0x0A0D line terminators were preserved.

If download_history can use 0x0A as a line terminator in both Linux and
Windows I can't see why the options file should not also. The options
file is internal to get_iplayer. Users do not need to edit it directly.

Ralph mentioned POSIX but as far as I can see from the Wikipedia article
there is a lot more to POSIX than converting 0x0A0D to 0x0A. Cygwin is
given as one example.

Post by MacFH - C E Macfarlane
But as the OP is changing over to Linux, perhaps the de facto standard
'dos2unix' would be a more appropriate recommendation.

Thanks for the suggestion but I don't think I need to convert the
download_history file (although gedit complained about invalid UTF-8
characters) and thanks to Ralph's command I now have an options file
which consistently uses 0x0D as a line terminator.

Best wishes
Richard

[RS]

Post by RS
If download_history can use 0x0A as a line terminator in both Linux and
Windows I can't see why the options file should not also.  The options
file is internal to get_iplayer.  Users do not need to edit it directly.

I would add that now Linux fully supports NTFS it is not unreasonable to
want to share data between Windows and Linux versions of programs and to
go backwards and forwards between whatever happens to be convenient at
the moment. Thunderbird, Libre Office, Softmaker Teamviewer and
browsers all make it possible, as long as you don't try to do it on the
Windows c: drive.

I am writing this with Thunderbird sharing its message store between
Windows 10 and ubuntu 16.04. All I needed to do to set it up was to run
thunderbird -p
to tell it where to store the messages.

Best wishes
Richard

[michael norman]

Post by RS

Post by RS
If download_history can use 0x0A as a line terminator in both Linux
and Windows I can't see why the options file should not also.  The
options file is internal to get_iplayer.  Users do not need to edit it
directly.

I would add that now Linux fully supports NTFS it is not unreasonable to
want to share data between Windows and Linux versions of programs and to
go backwards and forwards between whatever happens to be convenient at
the moment.  Thunderbird, Libre Office, Softmaker Teamviewer and
browsers all make it possible, as long as you don't try to do it on the
Windows c: drive.
I am writing this with Thunderbird sharing its message store between
Windows 10 and ubuntu 16.04.  All I needed to do to set it up was to run
thunderbird -p
to tell it where to store the messages.
Best wishes
Richard

This is OT but the other way of running Linux and Windows on the same
box is to use virtualisation. Virtualbox

https://www.virtualbox.org/wiki/Downloads

is a free way of doing that and certainly works with Linux. My
experience is using Linux as host and Windows as guest. Long time since
I did it but I do know W7 works, I have not used W10. The advantage
over dual boot is that you can access both OS's at the same time, and
you can certainly share directories between the two. The disadvantage
is that the virtual machine may lose some functionality ie its not
suitable IME for playing video games or doing heavy processing of things
like video files. But for pretty much anything else it works fine.

Disclaimer, I am by no means an expert in this so please don't direct
questions to me. VB is very well documented and IME both openSUSE and
Linuxmint have extensive community support where you can ask questions.
I will assome that Windows users will be able to access similar forms of
support.

As I said OT for this list but maybe germane to what has been said thus far.

M

[Bernard Peek]

Post by michael norman

Post by RS

Post by RS
If download_history can use 0x0A as a line terminator in both Linux
and Windows I can't see why the options file should not also.  The
options file is internal to get_iplayer.  Users do not need to edit
it directly.

I would add that now Linux fully supports NTFS it is not unreasonable
to want to share data between Windows and Linux versions of programs
and to go backwards and forwards between whatever happens to be
convenient at the moment.  Thunderbird, Libre Office, Softmaker
Teamviewer and browsers all make it possible, as long as you don't
try to do it on the Windows c: drive.
I am writing this with Thunderbird sharing its message store between
Windows 10 and ubuntu 16.04.  All I needed to do to set it up was to
run thunderbird -p
to tell it where to store the messages.
Best wishes
Richard

This is OT but the other way of running Linux and Windows on the same
box is to use virtualisation.  Virtualbox
https://www.virtualbox.org/wiki/Downloads

I mainly use Linux with get_iplayer running on a SuSE server but also
use a Mint workstation accessing the server using SSH. I have virtual
machines on the workstation using VB. I mostly run virtual XP machines
but I have tried W7, W8 and W10 successfully. I'm thinking of setting up
some virtual NT4 systems as they are likely to use less resources and
will run most of the programs I am interested in.

[RS]

Post by RS

Post by RS
If download_history can use 0x0A as a line terminator in both Linux
and Windows I can't see why the options file should not also.  The
options file is internal to get_iplayer.  Users do not need to edit it
directly.

I would add that now Linux fully supports NTFS it is not unreasonable to
want to share data between Windows and Linux versions of programs and to
go backwards and forwards between whatever happens to be convenient at
the moment.  Thunderbird, Libre Office, Softmaker Teamviewer and
browsers all make it possible, as long as you don't try to do it on the
Windows c: drive.

The discussion in this thread
https://forums.squarepenguin.co.uk/showthread.php?tid=1690
suggests it may be possible to put the profile directory somewhere it
could be read and written to by both Windows and Linux (not in c: or
/home). The incompatibility in file formats for the options file
between Windows and Linux would prevent that idea from working.

Best wishes
Richard

[MacFH - C E Macfarlane]

Please see below ...

Post by RS

Post by MacFH - C E Macfarlane
But as the OP is changing over to Linux, perhaps the de facto
standard 'dos2unix' would be a more appropriate recommendation.

Thanks for the suggestion but I don't think I need to convert the
download_history file (although gedit complained about invalid UTF-8
characters) and thanks to Ralph's command I now have an options file
which consistently uses 0x0D as a line terminator.

You seem to be confused about which line terminators are used by each
operating system, so to confirm:
    OS        Mnem    Hex code
    ---------------------------------
    Linux    LF           0x0A
    Mac     CR           0x0D (I believe)
    Win     CRLF       0x0D0x0A

I can't recall ever trying to copy windows GiP data over to Linux, so I
can't comment on whether the 'wrong' line terminator will make any
difference, but, as the engine interpreting the files is Perl rather
than the underlying OS, perhaps it won't matter to GiP at all.  However,
if you try to edit the file in a native editor such as GEdit, then yes,
you should expect problems.   I think I'd be inclined to make a backup
copy of the entire GiP data directory, I mean the one usually called
.get_iplayer containing the options and download_history files that
we've been discussing, and then run dos2unix on options,
download_history, and either delete the caches so that they are remade,
or run dos2unix on those as well.

Regards Charles.

[RS]

Post by MacFH - C E Macfarlane
Please see below ...

Post by RS

Post by MacFH - C E Macfarlane
But as the OP is changing over to Linux, perhaps the de facto
standard 'dos2unix' would be a more appropriate recommendation.

Thanks for the suggestion but I don't think I need to convert the
download_history file (although gedit complained about invalid UTF-8
characters) and thanks to Ralph's command I now have an options file
which consistently uses 0x0D as a line terminator.

You seem to be confused about which line terminators are used by each
    OS        Mnem    Hex code
    ---------------------------------
    Linux    LF           0x0A
    Mac     CR           0x0D (I believe)
    Win     CRLF       0x0D0x0A

You're right I am a bit confused, but not about the line termination
conventions for each OS, although I can't speak for the Mac.

What I said the download_history file was wrong. I have now looked at
it with a hex editor which allows me to view the whole file. Most of it
was written in Windows and has CRLF as a line terminator. The most
recent records, appended to the end, written in Linux, have LF as a line
terminator. The important thing is that the discrepancy does not stop
it working. The entries are indexed by pid. The script is able to find
the pid whether it is preceded by LF or CRLF, and identify the programme
as already in history. The mixed file works both in Linux and in Windows.

In the case of the options file the discrepancy in line termination does
prevent some options from being interpreted correctly. To find out why
I need to compare the Perl code for parsing the download_history file
with that for parsing the options file. I have not yet managed to find it.

As for using win2unix to convert, if you look at the win2unix
documentation you will see it is not that simple. It is possible to use
it just to change the line terminations. You also need to think about
character sets. Some of the comments in the code say UTF-8 is used for
Linux and cp850 for Windows. If I remember from 30 odd years ago that
was the oldest MSDOS code page. Should that be converted too? If I
want to go backwards and forwards I don't want to keep converting.

Best wishes
Richard

[Ralph Corderoy]

Hi Richard,

Post by RS
What I said the download_history file was wrong. I have now looked at
it with a hex editor which allows me to view the whole file. Most of
it was written in Windows and has CRLF as a line terminator. The most
recent records, appended to the end, written in Linux, have LF as a
line terminator. The important thing is that the discrepancy does not
stop it working. The entries are indexed by pid. The script is able
to find the pid whether it is preceded by LF or CRLF, and identify the
programme as already in history. The mixed file works both in Linux
and in Windows.

It probably doesn't.

Let's forget Mac for the moment. Linux text files are POSIX text files;
zero or more lines, each terminated by a LF. See ascii(7). DOS ones
use CR followed by LF at the end of each line.

Thus a DOS text file looks like a text file to Linux, but one where the
last character at the end of each line, just before the LF, is a CR,
which is just another possible byte that could be within the line. If
the operation you're doing doesn't mind that the CR is present then it
works; reading the PID from the start of the line and using it as a
key. But if you're using the data that runs to the end of the line then
you'll pick up the unwanted CR and that may do things like corrupt the
output.

$ seq 3
1
2
3
$ seq 3 | grep '^2$'
2
$ seq 3 | sed 's/$/\r/'
1
2
3
$ seq 3 | sed 's/$/\r/' | grep '^2$'
$
$ seq 3 | sed 's/$/\r/' | sed -n l
1\r$
2\r$
3\r$
$

The first grep works. Appending CR with sed doesn't change the apparent
appearance, but it's there as a byte so the second grep fails to find
the line that contains just `2'. sed's `l' command shows up the
problem.

So the new lines added by Linux to the end of the history file might
well look like one long, unterminated as there's no CR LF, line to
Windows.

[RS]

Post by Ralph Corderoy
It probably doesn't.
Let's forget Mac for the moment. Linux text files are POSIX text files;
zero or more lines, each terminated by a LF. See ascii(7). DOS ones
use CR followed by LF at the end of each line.
Thus a DOS text file looks like a text file to Linux, but one where the
last character at the end of each line, just before the LF, is a CR,
which is just another possible byte that could be within the line. If
the operation you're doing doesn't mind that the CR is present then it
works; reading the PID from the start of the line and using it as a
key. But if you're using the data that runs to the end of the line then
you'll pick up the unwanted CR and that may do things like corrupt the
output.

Hi Ralph

Essentially you seem to be saying I can't have a CR in a Linux text file
because it is non-standard. It seems a strange standard that prevents
data interchange rather than facilitates it. My approach would be to
ignore the CR or to recognise three possible end of record markers, but
it seems that is not allowed.

One article I read mentioned that errors can be caused by strict
adherence to a standard so that the last line of data is ignored if it
does not end with a newline.

I can exchange data in elaborate formats like DOCX, ODT or H.264 but I
can't as a text file.

What is the answer? How can Windows and Linux exchange data in text
files without requiring the user to run them through external conversion
programs?

Best wishes
Richard

[Alan Milewczyk]

What is the answer?  How can Windows and Linux exchange data in text
files without requiring the user to run them through external
conversion programs?

I've been following the gist of the thread but not the detail. I am
wondering whether the answer to your second question is that they
can't!!! Although GIP can be run on Windows, MacOS or Linux, each one
looks like it treats certain issues differently.  Not a problem when
you're sticking with the one OS but if you're trying to share the data
on different OSs, maybe you're trying to do something that cannot be
done seamlessly and that you have to use a conversion program.

Just my pennyworth..

Alan


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[Roger Bell_West]

I've been following the gist of the thread but not the detail. I am wondering
whether the answer to your second question is that they can't!!!

It would be possible to rewrite get_iplayer to make it agnostic with
respect to line endings.

This is, of course, only useful for those users who transfer its
working files between Windows and Linux machines.

If you are one of those users, feel free to write a patch.

[Alan Milewczyk]

Post by Roger Bell_West

I've been following the gist of the thread but not the detail. I am wondering
whether the answer to your second question is that they can't!!!

Sorry my statement should be qualified  by the caveat "with GIP as it
currently stands".

Post by Roger Bell_West
It would be possible to rewrite get_iplayer to make it agnostic with
respect to line endings.
This is, of course, only useful for those users who transfer its
working files between Windows and Linux machines.
If you are one of those users, feel free to write a patch.

This could always be raised as a request in the main GIP forum, but I
wouldn't hold your breath - I've seen many other requests receive short
shrift.... but you won't know unless you ask.

A

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[Ralph Corderoy]

Hi Richard,

Post by RS

Post by Ralph Corderoy
Let's forget Mac for the moment. Linux text files are POSIX text
files; zero or more lines, each terminated by a LF. See ascii(7).
DOS ones use CR followed by LF at the end of each line.
Thus a DOS text file looks like a text file to Linux, but one where
the last character at the end of each line, just before the LF, is a
CR, which is just another possible byte that could be within the
line. If the operation you're doing doesn't mind that the CR is
present then it works; reading the PID from the start of the line
and using it as a key. But if you're using the data that runs to
the end of the line then you'll pick up the unwanted CR and that may
do things like corrupt the output.

Essentially you seem to be saying I can't have a CR in a Linux text
file because it is non-standard.

Nope, I'm not saying that. `Linux text files are POSIX text files; zero
or more lines, each terminated by a LF.' Here's a one-line POSIX text
file.

$ sed -n l oneline
one\rtwo\rthree$
$ cat oneline
three
$

CR is just another character, a perfectly valid one, that's data as
opposed to the LF that's the line's terminator and meta-data.

Post by RS
It seems a strange standard that prevents data interchange rather than
facilitates it.

It is an excellent standard and ensures data interchange between the
hundreds of Unix text-processing tools that made Unix very popular.

Post by RS
One article I read mentioned that errors can be caused by strict
adherence to a standard so that the last line of data is ignored if it
does not end with a newline.

Emacs is a non-Unix text editor that was ported to Unix, but kept its
foreign text-file format that uses line separators, not terminators.
Emacs users pollute the Unix filesystem with non-text files that have
trailing bytes beyond the zero-or-more Unix text lines unless they
change Emacs's defaults. Unix programs are not at fault here, nor is
their `strict' adherence; why bother otherwise?

Post by RS
My approach would be to ignore the CR or to recognise three possible
end of record markers, but it seems that is not allowed.

It certainly is allowed. It would be the Richard Text File Format.
Programs could code to handle it. More precise specification would be
required. Is the first line-ending encountered from the start of the
file the one that must be used thence? Or may each line differ? RTTF
is incompatible with reading the POSIX `oneline' text file above so
conversion would be required, presumably by some means of escaping the
CR? And escaping the escape character? And converting files that were
already using the escape character to not mean escape.
https://xkcd.com/927/

Post by RS
I can exchange data in elaborate formats like DOCX, ODT or H.264 but I
can't as a text file.

They are file formats designed for an application, or a standard for
interchange. A text file is an operating-system file format, and they
show their lineage. CPM and DOS let the needs of the hardware leak into
the file format, wasting a byte each time. Before them, Unix, really
Multics, bettered that by having a logical representation and leaving
the hardware's needs to the device driver.

I understand macOS, the latest name for it, has moved to POSIX's format.

Post by RS
What is the answer? How can Windows and Linux exchange data in text
files without requiring the user to run them through external
conversion programs?

A program can code for one format everywhere, e.g. POSIX, but that then
depends on editors that stick with POSIX format on non-POSIX systems.
They exist and are becoming ever more common. Or it can do as Python as
done and try to adapt on the fly, but that corrupts some POSIX text
files on reading, which is why Python lets you disable it.

Because C has lines terminated by LF, like Unix, C libraries on Windows
convert CRLF to LF on reading a text file, and do the reverse on
writing. Perl implements that.

[RS]

Post by Ralph Corderoy

Post by RS
My approach would be to ignore the CR or to recognise three possible
end of record markers, but it seems that is not allowed.

It certainly is allowed. It would be the Richard Text File Format.
Programs could code to handle it. More precise specification would be
required. Is the first line-ending encountered from the start of the
file the one that must be used thence? Or may each line differ?

Hi Ralph

I suggested ignoring CR because they are there. Ideally they would not
be there. The files are internal to get_iplayer so they can be in any
format. I would have thought it sensible to use LF as a line terminator
for get_iplayer's internal files across all operating systems. The
question is how the CR are getting into the files in Windows.

Since the script uses
print \n
the answer must be that Perl is translating \n as CR LF in Windows. If
Perl can then handle the variants portably with chomp VARIABLE that's
fine, but it doesn't seem to be working properly at the moment.
Otherwise the solution may be to replace \n with \015.

Best wishes
Richard

[Ralph Corderoy]

Hi Richard,

Post by RS
I suggested ignoring CR because they are there. Ideally they would
not be there. The files are internal to get_iplayer so they can be in
any format.

Only if you don't want them to be native text files, editable on that
system with a text editor by any user. And I thought you do because you
have been editing them.

Post by RS
I would have thought it sensible to use LF as a line terminator for
get_iplayer's internal files across all operating systems. The
question is how the CR are getting into the files in Windows.

I pre-empted this in the email you're replying to.

Because C has lines terminated by LF, like Unix, C libraries on
Windows convert CRLF to LF on reading a text file, and do the
reverse on writing. Perl implements that.

Post by RS
If Perl can then handle the variants portably with chomp VARIABLE
that's fine, but it doesn't seem to be working properly at the moment.

That's not what chomp() does. It removes a single trailing LF if it's
present. The conversion of CRLF to LF on reading a text file is
PerlIO(3perl)'s `:crlf' layer, stacked on Windows systems.

[RS]

Post by Ralph Corderoy

Post by RS
I suggested ignoring CR because they are there. Ideally they would
not be there. The files are internal to get_iplayer so they can be in
any format.

Only if you don't want them to be native text files, editable on that
system with a text editor by any user. And I thought you do because you
have been editing them.

Hi Ralph

I have only ever edited the options file by running the script you gave
me to convert CR LF to LF. There is no need to edit the file directly
because it can be displayed with --prefs-show. Entries can be changed
and added with --prefs-add and removed with --prefs-del. The format of
those commands and the display are different from each other and from
the file format, so editing directly would be a recipe for confusion.

Post by Ralph Corderoy

Post by RS
I would have thought it sensible to use LF as a line terminator for
get_iplayer's internal files across all operating systems. The
question is how the CR are getting into the files in Windows.

I pre-empted this in the email you're replying to.
Because C has lines terminated by LF, like Unix, C libraries on
Windows convert CRLF to LF on reading a text file, and do the
reverse on writing. Perl implements that.

If the conversion only occurs in Windows, that's no good. I have cache,
history and options files created in Windows with CR LF. I want
get_iplayer to be able to read and update those files in Linux. Ideally
I want to be able to put them on a shared drive so that both
installations can update them. I thought from the article I read chomp
VARIABLE could do that, but clearly I have misunderstood.

Best wishes
Richard

[RS]

Post by Ralph Corderoy

Post by RS
I would have thought it sensible to use LF as a line terminator for
get_iplayer's internal files across all operating systems.  The
question is how the CR are getting into the files in Windows.

I pre-empted this in the email you're replying to.
     Because C has lines terminated by LF, like Unix, C libraries on
     Windows convert CRLF to LF on reading a text file, and do the
     reverse on writing.  Perl implements that.

If the conversion only occurs in Windows, that's no good.  I have cache,
history and options files created in Windows with CR LF.  I want
get_iplayer to be able to read and update those files in Linux.  Ideally
I want  to be able to put them on a shared drive so that both
installations can update them.  I thought from the article I read chomp
VARIABLE could do that, but clearly I have misunderstood.

It's worse than I thought. I had got the impression from the perlport
perldoc that if replaced \n with \012 in a print statement I would get a
LF on its own in Windows. I don't. If I insert \015 I can have a CR on
its own, but \012 is still replaced with CR LF.

This article
http://www.perlmonks.org/?node=binmode
says I can use binmode, so that may be an answer. It's quite old, so it
may no longer apply.

Best wishes
Richard

[David Cantrell]

Post by RS
It's worse than I thought. I had got the impression from the perlport
perldoc that if replaced \n with \012 in a print statement I would get a
LF on its own in Windows. I don't. If I insert \015 I can have a CR on
its own, but \012 is still replaced with CR LF.
This article
http://www.perlmonks.org/?node=binmode
says I can use binmode, so that may be an answer. It's quite old, so it
may no longer apply.

binmode does still work AFAIK, but a more modern and flexible method is
to use the crlf I/O layer, which is documented here:
https://perldoc.perl.org/PerlIO.html

Note however that an awful lot of perl code just doesn't bother. Windows
is very much a second-class citizen in the perl world.

[Ralph Corderoy]

Hi David,

Post by David Cantrell
binmode does still work AFAIK, but a more modern and flexible method
https://perldoc.perl.org/PerlIO.html
Note however that an awful lot of perl code just doesn't bother.

Windows stacks the `:crlf' layer by default. I *think* Richard's trying
to avoid that because he wants his get_iplayer to use POSIX text files
on Linux, as normal, and Windows.

[iz]

Post by Ralph Corderoy

Post by David Cantrell
binmode does still work AFAIK, but a more modern and flexible method
https://perldoc.perl.org/PerlIO.html
Note however that an awful lot of perl code just doesn't bother.

Windows stacks the `:crlf' layer by default. I *think* Richard's trying
to avoid that because he wants his get_iplayer to use POSIX text files
on Linux, as normal, and Windows.

PerlIO enables you to go either way, and no code changes are necessary. The simplest solution is to run the equivalent of 'PERLIO=crlf get_iplayer [...]' on Linux, perhaps via an alias in your shell. Probably best not to set PERLIO=crlf permanently if you run other Perl programs. You may not like having files with CRLF line endings, but they will still work on both platforms if you screw up editing and get mixed line endings. If you really can't abide CRLF, use PERLIO=perlio on Windows instead to produce LF line endings. Probably simplest to set it locally in the get_iplayer batch file. You just have to be careful not to inject CRLF line endings if you edit any files directly, though I would expect any decent text editor on both platforms can avoid that.

[r***@gmx.com]

I don't know the history of this thread, but a Windows perl will use the `:crlf` layer which is fine for both Windows (CR LF) and Linux (LF) line terminations. The CR will be removed, and every line read will end with LF ("\n")

The most common problem arises when a Linux perl attempts to read a Windows text file. The default PerlIO stack on Linux is an empty one, so the CR LF characters appear intact in every line read

There are a few solutions:

1/ Change the default PerlIO layers

use open qw/ :std :crlf /

will cause every file to be opened with the Windows `:crlf` layer, which removes any CR if it is there. It doesn't affect Linux files at all

2/ If you *always* `chomp` every line, then use

s/\R\z//

instead, which will remove any line termination character or character pair

3/ If you *always* `split` every line using the default of `split ' ', $_` then there is nothing to do: CR is considered to be a whitespace character and will be removed from the fields

There are clearly variations on this idea, but it shouldn't be hard to resolve from here

Can someone please tell me whether I have answered the original question, or if I'm miles off?

Rob Dixon
Norfolk
England


-----Original Message-----
From: Ralph Corderoy <***@inputplus.co.uk>
To: ***@lists.infradead.org
Sent: Tue, 06 Mar 2018 13:30
Subject: Re: Format of options file

Hi David,

Post by David Cantrell
binmode does still work AFAIK, but a more modern and flexible method
https://perldoc.perl.org/PerlIO.html
Note however that an awful lot of perl code just doesn't bother.

Windows stacks the `:crlf' layer by default. I *think* Richard's trying
to avoid that because he wants his get_iplayer to use POSIX text files
on Linux, as normal, and Windows.

[David Cantrell]

Post by RS
Essentially you seem to be saying I can't have a CR in a Linux text file
because it is non-standard. It seems a strange standard that prevents
data interchange rather than facilitates it.

Yes, it is strange that Microsoft chose a non-standard line ending
convention. In this case though we can't really blame them as they were
just copying CP/M. CP/M chose its line ending convention in 1974, a few
years after Unix chose its convention.

[MacFH - C E Macfarlane]

Please see below ...

Post by RS

Post by MacFH - C E Macfarlane

Post by RS

Post by MacFH - C E Macfarlane
But as the OP is changing over to Linux, perhaps the de facto
standard 'dos2unix' would be a more appropriate recommendation.

Thanks for the suggestion but I don't think I need to convert the
download_history file (although gedit complained about invalid UTF-8
characters) and thanks to Ralph's command I now have an options file
which consistently uses 0x0D as a line terminator.

You seem to be confused about which line terminators are used by each
     OS        Mnem    Hex code
     ---------------------------------
     Linux    LF           0x0A
     Mac     CR           0x0D (I believe)
     Win     CRLF       0x0D0x0A

You're right I am a bit confused, but not about the line termination
conventions for each OS, although I can't speak for the Mac.
What I said the download_history file was wrong.  I have now looked at
it with a hex editor which allows me to view the whole file.  Most of
it was written in Windows and has CRLF as a line terminator.  The most
recent records, appended to the end, written in Linux, have LF as a
line terminator.  The important thing is that the discrepancy does not
stop it working.  The entries are indexed by pid.  The script is able
to find the pid whether it is preceded by LF or CRLF, and identify the
programme as already in history.  The mixed file works both in Linux
and in Windows.
In the case of the options file the discrepancy in line termination
does prevent some options from being interpreted correctly.  To find
out why I need to compare the Perl code for parsing the
download_history file with that for parsing the options file.  I have
not yet managed to find it.

Most probably it is those options which are passed to an external
program that cause difficulty  -  my guess, I stress that it is a guess 
-  the program is getting confused by the unexpected CR 0x0D when it is
expecting only LF 0x0A, and may be trying to interpret the CR as part of
the parameter it is being given.

Post by RS
As for using win2unix to convert, if you look at the win2unix
documentation you will see it is not that simple.  It is possible to
use it just to change the line terminations.  You also need to think
about character sets.  Some of the comments in the code say UTF-8 is
used for Linux and cp850 for Windows.  If I remember from 30 odd years
ago that was the oldest MSDOS code page.  Should that be converted
too?  If I want to go backwards and forwards I don't want to keep
converting.

The Windows ANSI and UTF-8 character sets are sufficiently close that I
wouldn't worry about any need to convert, just do the line-termination. 
Given two directories, .get_iplayer.bak and .get_iplayer, beneath the
present working directory HOME aka ~, this command should do what you
want (I've doubled up the spaces for better clarity of the syntax):
    for  F  in  options  download_history  *.cache  ;  do dos2unix -n 
.get_iplayer.bak/$F  .get_iplayer/$F  ;  done

Regards Charles.

[MacFH - C E Macfarlane]

Please see correction below ...

Post by MacFH - C E Macfarlane
    for  F  in  options  download_history  *.cache  ;  do dos2unix -n 
.get_iplayer.bak/$F  .get_iplayer/$F  ;  done

I've just realised that this may no do the *.cache files because they
are in a different directory than the one where you are giving the
command, so safer would be:
    for  F  in  options  download_history  radio.cache  tv.cache  ; do
dos2unix -n  .get_iplayer.bak/$F  .get_iplayer/$F  ;  done

Apologies for the slight error.
Regards Charles.

[David Cantrell]

Post by RS
You're right I am a bit confused, but not about the line termination
conventions for each OS, although I can't speak for the Mac.

Macs use \n like normal Unix machines. They used to use \r in the bad
old days before they went Unixy.

[Ralph Corderoy]

Hi David,

Post by David Cantrell
Macs use \n like normal Unix machines. They used to use \r in the bad
old days before they went Unixy.

Yes, you're right. Earlier in the thread I said I thought they used CR,
but that was much earlier up to Mac OS 9. Mac OS X came along around
2000, was based on Unix, unlike 9, and switched to its LF line
terminator. macOS is still version 10, the X Roman numeral, but has
changed name to OS X, and now macOS along the way.

[Nick Payne]

Post by Bernard Peek

Post by James Scholes

The Windows download_history file seems to work correctly.  I hope I
don't need to edit it.

You've probably edited your options file by hand in the past,
resulting in a mixture of weird line endings from a Windows editor,
and appropriate ones when get_iplayer itself has updated it.  That's
just my guess after going through exactly the same thing recently. 
I'd bet you probably don't go editing your download_history file by
hand very often so it's most likely fine.

For what it's worth I noticed that under Windows the Wordpad program
seems to understand UNIX style terminations and retains them.

The free Notepad++ editor does also. In fact, you can set in the program
preferences whether you want line endings to be formatted as the Windows
CRLF, Unix LF, or Mac CR.

Nick