radio recordings using latest gip version

[Jim Lesurf]

Until yesterday I've been using an old version of gip to fetch radio
programmes. This is on my usual "Ain't broke" -> "Don't fix" basis. However
yesterday when I tried to get the latest set of R3 Proms files it failed
for the items that started with the pids that begin with 'm'.

As an aside, I know zip about perl. But I did experiment with seeing if
this was a filter problem I could fix. I found a couple of places in the
version I was using that seemed to be requiring a 'p' or 'b' and added 'm'
in the '[ ]' braces. But no cigar. Out of curiosity, if this should have
been possible I'd welcome someone explaining. However, on with the main
reason for writing this...

I got the current version of gip, and this did let me fetch the files.
However each time it began it popped up some comments to the effect

"Error response 500 Can't connect to aod-hls-uk-live ...

... certificate verify failed."

It then said to ignore this if the download worked. Which it did. But my
questions are: Can I tell it to skip this apparent attempt to 'verify' and
not prompt or have to report the error? Or am I doing something wrong?

FWIW I've been using a series of settings

--type=radio --mode-hafhigh --force --no-tag --pid (then the pid)

Doing the above, there is something else I can report in case someone can
comment or find it of interest.

I fetched a series of half a dozen 'Proms' files inc. some of the
'headphone mix' ones which I'm in two minds about. The first fetch ('m'
prefix pid) popped up an error partway though the process.

502 bad gateway

and said a segment was missing.

The others were all OK.

I refetched the first file and this time it came without any 502. Comparing
the two files, they have exactly the same length, but I've not yet had a
chance to compare them to spot any diff.

The 502 was curious. But I also noticed that the first series of fetches
all ran at about the same, moderately slow, rate. Whereas the re-fetch ran
much faster. Which strenghtens a feeling I have that items more than a few
days old are slower to transfer, but can then get into some kind of 'cache'
which means they then can be obtained more quickly until bumped out by
something else. Is this correct?

Cheers,

Jim

[RS]

Post by Jim Lesurf
Until yesterday I've been using an old version of gip to fetch radio
programmes. This is on my usual "Ain't broke" -> "Don't fix" basis.
However yesterday when I tried to get the latest set of R3 Proms files
it failed for the items that started with the pids that begin with 'm'.

The first prom with an m prefix seems to be Prom 52, m00006pc.  I have
just tried to download it with get_iplayer 3.17 using hafhigh and
dafhigh. There did not appear to be any problem, although I did not run
the downloads to completion.  The default modes were hafhigh1 and
dafhigh1, both of which used the Limelight CDN.

I was a bit too slow starting downloads of Proms.  I missed Prom 1
(although I did get the headphone version of The Planets).  I had
problems similar to the ones you describe with Proms 2 and 3 (with a b
prefix) with the Akamai CDN (hafhigh2).  These went away when I retried
with the Limelight CDN (hafhigh1) which was also a lot faster.  I have
not had any further problems since.

I generally prefer Akamai for television programmes, but it may not be a
good choice for radio.

Best wishes
Richard

[Jim web]

Post by Jim Lesurf
Until yesterday I've been using an old version of gip to fetch radio
programmes. This is on my usual "Ain't broke" -> "Don't fix" basis.
However yesterday when I tried to get the latest set of R3 Proms files
it failed for the items that started with the pids that begin with 'm'.

The first prom with an m prefix seems to be Prom 52, m00006pc. I have
just tried to download it with get_iplayer 3.17 using hafhigh and
dafhigh. There did not appear to be any problem, although I did not run
the downloads to completion. The default modes were hafhigh1 and
dafhigh1, both of which used the Limelight CDN.

I just tried hafhigh1 to get last night's Prom. Again this gave me the 500
error, but completed OK regardless. Given what you wrote about CDNs I
noticed that the progress line showed "(hafhigh1/ak) after the ETA. I don't
know what the "ak" means there but it made me wonder if I was still getting
contact with Akamai?

FWIW The error gets reported before the download commences.

I'll try other mode specs when I get a chance and see if I can vary the
behaviour.

Thanks,

Jim

[RS]

Post by Jim web
I just tried hafhigh1 to get last night's Prom. Again this gave me the 500
error, but completed OK regardless. Given what you wrote about CDNs I
noticed that the progress line showed "(hafhigh1/ak) after the ETA. I don't
know what the "ak" means there but it made me wonder if I was still getting
contact with Akamai?
FWIW The error gets reported before the download commences.
I'll try other mode specs when I get a chance and see if I can vary the
behaviour.

Your CDNs are in a different order from mine.  For television for most
programmes I have
hvfhd1 BIDI (bi)
hvfhd2 Limelight (ll)
hvfhd3 Akamai (ak)

BIDI is the BBC's own CDN.
http://www.bbc.co.uk/blogs/internet/entries/8c6c2414-df7a-4ad7-bd2e-dbe481da3633
It may only be available in some areas and for some modes.  If a CDN is
not present the other suffixes will be reduced, so for radio there may
only be hafhigh1 and hafhigh2.

Try --radiomode=hafhigh2 to see whether that gives you Limelight.

Best wishes
Richard

[Timothy]

RS wrote: Try --radiomode=hafhigh2 to see whether that gives you Limelight.

Uh, why not just be completely sure you're using it by using the exclude supplier option [1]? In other words, put "--exclude-supplier akamai,bidi and you should be good from there.

Timothy

[1]: https://github.com/get-iplayer/get_iplayer/wiki/options#recording-options

[Jim web]

I've now tried hafhigh2 and dafhigh1. They also give me the same error, and
then seem to go on to fetch without any actual problem.

Here a '1' gives me /ak and a '2' gives me /ll.

Jim

[Jim web]

To update, etc...

hafhigh3 also gives me /ak. I tried --exclude-supplier=akamai,limelight but
this failed to find anything from bidi. I've since realised that this
switch option should probably have had a space not an equals sign. So I'll
check that again later. However some other oddity has come up I want to
check to see if it is 'just me', or alert anyone else affected.

Here prom 56 (pid m0000736) contains a chunk of 'digital black' between
about 19m17s and 19m20s from the start. Looks like a missing segment filled
with nothing. I've tried fetching this twice, once from ak, once from ll,
and they are the same. I'll ask someone at the BBC in case the source is
faulty, but thought I'd mention it here in case it affects someone else -
or they can say that they do get the audio for this segment and it must be
something wrong here.

Jim

Post by Jim web
I've now tried hafhigh2 and dafhigh1. They also give me the same error,
and then seem to go on to fetch without any actual problem.
Here a '1' gives me /ak and a '2' gives me /ll.
Jim

[RS]

Post by Jim web
Here prom 56 (pid m0000736) contains a chunk of 'digital black' between
about 19m17s and 19m20s from the start.

I have just downloaded it from Limelight, --radiomode=hafhigh1. I have
only listened from 19min to 20min using VLC; I have not tried to display
the waveform. I agree there is 3s of silence between 19min17s and 19min20s.

--exclude-supplier=akamai,limelight and
--exclude-supplier akamai,limelight

are equally valid.

Best wishes
Richard

[mccarthy kevin]

It's the same on the BBC iPlayerversion itself.
Maybe complain to BBC?

Kevin

Post by RS

Post by Jim web
Here prom 56 (pid m0000736) contains a chunk of 'digital black' between
about 19m17s and 19m20s from the start.

I have just downloaded it from Limelight, --radiomode=hafhigh1. I have
only listened from 19min to 20min using VLC; I have not tried to display
the waveform. I agree there is 3s of silence between 19min17s and 19min20s.
--exclude-supplier=akamai,limelight and
--exclude-supplier akamai,limelight
are equally valid.
Best wishes
Richard
_______________________________________________
get_iplayer mailing list
http://lists.infradead.org/mailman/listinfo/get_iplayer

[CJB]

Just a thought - do other bit rates have the same gap? If not then it
would be easy enough to fill same using Audacity. If they DO have the
same gap then the entire recording is corrupt and the Beeb needs to
know. CJB

Post by RS

Post by Jim web
Here prom 56 (pid m0000736) contains a chunk of 'digital black' between
about 19m17s and 19m20s from the start.

I have just downloaded it from Limelight, --radiomode=hafhigh1. I have
only listened from 19min to 20min using VLC; I have not tried to display
the waveform. I agree there is 3s of silence between 19min17s and 19min20s.
--exclude-supplier=akamai,limelight and
--exclude-supplier akamai,limelight
are equally valid.
Best wishes
Richard
_______________________________________________
get_iplayer mailing list
http://lists.infradead.org/mailman/listinfo/get_iplayer

[RS]

Post by CJB
Just a thought - do other bit rates have the same gap? If not then it
would be easy enough to fill same using Audacity. If they DO have the
same gap then the entire recording is corrupt and the Beeb needs to
know. CJB

hafstd1, 128kbit/s, also from Limelight but a much slower download, has
the same gap.

Best wishes
Richard

[RS]

Post by RS
hafstd1, 128kbit/s, also from Limelight but a much slower download, has
the same gap.

As does dafhigh2 from Akamai.

It also had errors

INFO: Downloading radio: 'BBC Proms: 2018 - Prom 56: Mozart and Bruckner
(m0000736) [original]'
INFO: Downloaded: 313.14 MB (02:09:48) @ 3.82 Mb/s (dafhigh2/ak) [audio]

WARNING: Failed to download file segment [1218]
WARNING: Response: 502 Bad Gateway
WARNING: Retrying radio: 'BBC Proms: 2018 - Prom 56: Mozart and Bruckner
(m0000736) [original]'
INFO: Downloaded: 92.12 MB (00:38:11) @ 3.65 Mb/s (dafhigh2/ak) [audio]

but they are probably caused by the very slow download speed which at
times was a low as 2Mbit/s.

Best wishes
Richard

[Jim web]

Post by RS

Post by RS
hafstd1, 128kbit/s, also from Limelight but a much slower download,
has the same gap.

As does dafhigh2 from Akamai.
It also had errors
INFO: Downloading radio: 'BBC Proms: 2018 - Prom 56: Mozart and Bruckner
Mb/s (dafhigh2/ak) [audio]
WARNING: Failed to download file segment [1218] WARNING: Response: 502
Bad Gateway WARNING: Retrying radio: 'BBC Proms: 2018 - Prom 56: Mozart
and Bruckner (m0000736) [original]' INFO: Downloaded: 92.12 MB
but they are probably caused by the very slow download speed which at
times was a low as 2Mbit/s.

I got a similar 502 "Bad Gateway" during my first use of the current
version of gip for an earlier prom. (The first with an 'm' prefix.) But I
can't find any gap in the audio at or near the indicated time. I guess the
'retry' is successful.

FWIW I've been wondering about writing a simple 'digital black scanner' to
check fetched items for streams of 'nothing'. But as yet have been held up
by a shortage of round tuits. :-)

[Jim web]

Post by RS
I have just downloaded it from Limelight, --radiomode=hafhigh1. I have
only listened from 19min to 20min using VLC; I have not tried to display
the waveform. I agree there is 3s of silence between 19min17s and 19min20s.
--exclude-supplier=akamai,limelight and
--exclude-supplier akamai,limelight
are equally valid.

OK. That's curious. I'd assumed using it would give me a fetch from bidi.
But instead I was given a message to the effect that the item wasn't
available.

That seems to chime with hafhigh3 connecting to /ak just like hafhigh1. I
seem only to have contact with akamai or limelight.

I've already emailed someone at the BBC about the gap in the Prom. Looks
like a problem 'upstream'. Hopefully, someone will fix it now I've pointed
it out. If they do and I'm told, I'll let people know.

Jim

[Jim web]

Post by Jim web
I've already emailed someone at the BBC about the gap in the Prom. Looks
like a problem 'upstream'. Hopefully, someone will fix it now I've
pointed it out. If they do and I'm told, I'll let people know.

It's been confirmed to me that the source file is faulty. All being well, a
new version should replace it at some point.

Jim

[Jim web]

Just as info...

I fetch TV programmes before 9am in the morning to avoid my data 'cap'. At
that time such downloads run at just over 60 Megs. Which is the nominal
rate I've been told by my ISP, etc, that I should be able to get.

When fetching some more radio 3 proms this morning using hafhigh I noticed
they were fetching at rates around 7 Megs or less. This seems from '/ak'
reported to be from akamai. So I tried using hafhigh2 which gives '/ll'.
These fetches ran at more like 20 Megs or higher.

May be chance, but seems that here such files come much quicker from
limelight than akamai.

Jim

[RS]

Post by Jim web

Post by RS
--exclude-supplier=akamai,limelight and
--exclude-supplier akamai,limelight
are equally valid.

OK. That's curious. I'd assumed using it would give me a fetch from bidi.
But instead I was given a message to the effect that the item wasn't
available.
That seems to chime with hafhigh3 connecting to /ak just like hafhigh1. I
seem only to have contact with akamai or limelight.

I think I have only seen BIDI for television programmes.

Best wishes
Richard

[Alan Milewczyk]

Post by RS
(although I did get the headphone version of The Planets)

What is the "headphone version"?

Thanks

Alan

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[Clive]

Post by Alan Milewczyk

Post by RS
(although I did get the headphone version of The Planets)

What is the "headphone version"?
Thanks
Alan

https://www.bbc.co.uk/programmes/p06drb3s

[Alan Milewczyk]

Post by Clive

Post by Alan Milewczyk

Post by RS
(although I did get the headphone version of The Planets)

What is the "headphone version"?
Thanks
Alan

https://www.bbc.co.uk/programmes/p06drb3s

Ah, thank you, sounds interesting.

A

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[RS]

Post by Clive

Post by Alan Milewczyk

Post by RS
(although I did get the headphone version of The Planets)

What is the "headphone version"?
Thanks
Alan

https://www.bbc.co.uk/programmes/p06drb3s

Last year about half a dozen Proms were made available as binaural
stereo simulations or some such phrase.
http://www.bbc.co.uk/programmes/articles/2913JxRtQl3ZTvw0wz5C4D1/bbc-proms-in-binaural-sound
I wasn't that impressed with the ones I listened to.  On
https://www.bbc.co.uk/events/e8j3v2
The Planets in Prom 1 is said to be available as a headphone mix.  I
don't know if that is the same or something different.  Clive has given
the link to the recording.  I haven't checked whether it is still there.

Best wishes
Richard

[Alan Milewczyk]

Post by RS

Post by Clive

Post by Alan Milewczyk

Post by RS
(although I did get the headphone version of The Planets)

What is the "headphone version"?
Thanks
Alan

https://www.bbc.co.uk/programmes/p06drb3s

Last year about half a dozen Proms were made available as binaural
stereo simulations or some such phrase.
http://www.bbc.co.uk/programmes/articles/2913JxRtQl3ZTvw0wz5C4D1/bbc-proms-in-binaural-sound
I wasn't that impressed with the ones I listened to.  On
https://www.bbc.co.uk/events/e8j3v2
The Planets in Prom 1 is said to be available as a headphone mix. I
don't know if that is the same or something different.  Clive has
given the link to the recording.  I haven't checked whether it is
still there.

Yes, it's still there, I've just downloaded all the programmes on that
page and will listen to them tomorrow. I wasn't aware that this had been
done last year as well.

Regards

Alan

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[Jim web]

I wasn't that impressed with the ones I listened to. On
https://www.bbc.co.uk/events/e8j3v2 The Planets in Prom 1 is said to be
available as a headphone mix. I don't know if that is the same or
something different. Clive has given the link to the recording. I
haven't checked whether it is still there.

Going into detail is OT here, however I've also not been impressed by the
'headphone' and 'binaural' 'mix' versions I've had a chance thus far to
compare with the normal R3 versions. I fear the key term is 'mix'. i.e.
someone is doing a 're mix' of the mics outputs. i.e. *not* what I'd regard
as 'binaral' which requires a specific type of microphone system.

However others may prefer them, so they are worth a go if you use
headphones.

Jim