BBC's iPlayer works when get_iplayer doesn't

[Alan Milewczyk]

Hi there

I'm now back in the UK after 4 months in the Philippines so I'm into
catch-up mode while new programmes keep on coming relentlessly. Last
year it took me about 2 months to catch up on 4 months missed TV!

Hit a problem with Monday's episode of EastEnders (yeah, sad, I know!).

The command get_iplayer --pid b08l429m (albeit in a batch file) failed
to record anything, producing the following output:

"INFO Trying to download PID using type tv
INFO: pid found in cache
Matches:
2455: EastEnders - 27/03/2017, BBC One, b08l429m

INFO: 1 Matching Programmes
INFO: No versions of this programme were selected (available versions:
audiodescribed)"

Slightly surprising for a mainstream programme only to be available in
AD version, I thought. Indeed, doing a

get_iplayer --info --pid b08l429m

confirms the situation with the following output:

"INFO Trying to download PID using type tv
INFO: pid found in cache
Matches:
2455: EastEnders - 27/03/2017, BBC One, b08l429m
INFO: No versions of this programme were selected (available versions:
audiodescribed)
INFO: File name prefix = EastEnders_-_5465._2017-03-27_b08l429m_default

available: 2017-03-27T20:30:00+01:00
brand: EastEnders
categories: Drama,Soaps
category: Drama
channel: BBC One
desc: Michelle reaches breaking point. Whitney goes all out to
support Tina.
desclong: Michelle reaches breaking point. Whitney goes all out to
support Tina.
descmedium: Michelle reaches breaking point. Whitney goes all out to
support Tina.
descshort: Michelle reaches breaking point. Whitney goes all out to
support Tina.
dir: F:\iPlayer-TV
dldate: 2017-03-29
dltime: 20:24:04
duration: 1800
durations: audiodescribed: 1800
episode: 5465. 27/03/2017
episodenum: 5465
episodeshort: 27/03/2017
expires: in 28 days 0 hours (2017-04-26T19:30:00+00:00)
ext: EXT
filename: F:\iPlayer-TV\EastEnders_-_5465._2017-03-27_b08l429m_default.EXT
filepart:
F:\iPlayer-TV\EastEnders_-_5465._2017-03-27_b08l429m_default.partial.EXT
fileprefix: EastEnders_-_5465._2017-03-27_b08l429m_default
firstbcast: audiodescribed: 2017-03-27T20:30:00+01:00
firstbcastdate: audiodescribed: 2017-03-27
firstbcastrel: audiodescribed: 1 days 23 hours ago
index: 2455
lastbcast: audiodescribed: 2017-03-27T20:30:00+01:00
lastbcastdate: audiodescribed: 2017-03-27
lastbcastrel: audiodescribed: 1 days 23 hours ago
longname: EastEnders
modes: audiodescribed:
dvfxsd1,dvfxsd2,dvfxsd3,dvfxsd4,dvfxhigh1,dvfxhigh2,dvfxhigh3,dvfxhigh4,dvflow1,dvflow2,dvflow3,dvflow4,hvfxsd1,hvfxsd2,hvfhigh1,hvfhigh2,hvfxhigh1,hvfxhigh2,hvflow1,hvflow2
modesizes: audiodescribed:
dvfxsd1=319MiB,dvfxsd2=319MiB,dvfxsd3=319MiB,dvfxsd4=319MiB,dvfxhigh1=164MiB,dvfxhigh2=164MiB,dvfxhigh3=164MiB,dvfxhigh4=164MiB,dvflow1=87MiB,dvflow2=87MiB,dvflow3=87MiB,dvflow4=87MiB,hvfxsd1=387MiB,hvfxsd2=387MiB,hvfhigh1=386MiB,hvfhigh2=386MiB,hvfxhigh1=210MiB,hvfxhigh2=210MiB,hvflow1=121MiB,hvflow2=121MiB
name: EastEnders
nameshort: EastEnders
pid: b08l429m
player: http://www.bbc.co.uk/iplayer/episode/b08l429m
runtime: 30
senum: s01e5465
seriesnum: 1
thumbfile: F:\iPlayer-TV\EastEnders_-_5465._2017-03-27_b08l429m_default.jpg
thumbnail: Loading Image...
timeadded: 1 days 11 hours ago (2017-03-28T07:32:35+00:00)
title: EastEnders: 27/03/2017
type: tv
verpids: audiodescribed: p04y79vv
version: default
versions: audiodescribed
web: http://www.bbc.co.uk/programmes/b006m86d


INFO: 1 Matching Programmes"

At first I wondered if this was an error on the part of the BBC team
until I went to the BBC iPlayer website. It wasn't reported as an issue
on the BBC site and curiouser still, not only does it play from the BBC
website but it also downloads successfully using BBC's iPlayer. It plays
normally as if it were the "original" version, not a hint of AD!

I've never comes across this situation before, if it's available for
download on the BBC's iPlayer site then it's downloadable using
get_iplayer.

As I said earlier, due to me being in "catch-up mode", by the time I got
round to watching it, it would have long rolled off BBC's iPlayer. As it
happens, fortunately, I've managed to find the programme online
elsewhere but I'm wondering am I missing something here?

Any thoughts?

Alan

[tellyaddict]

Interesting find there Alan. You are right, GiP 2.99 only picks up the audiodescribed version. As you can see when you play the programme on iPlayer and right click, the vpid for the non AD version is b08l4283 but GiP misses it completely.

Version 2.94 can see it but anything after that for some reason can't. Also in 2.99 in the option to use alternative metadata streams --ybbcy which also sees the missing version.

Post by Alan Milewczyk
The command get_iplayer --pid b08l429m (albeit in a batch file) failed
"INFO Trying to download PID using type tv
INFO: pid found in cache
2455: EastEnders - 27/03/2017, BBC One, b08l429m
INFO: 1 Matching Programmes
audiodescribed)"
Slightly surprising for a mainstream programme only to be available in
AD version, I thought. Indeed, doing a
get_iplayer --info --pid b08l429m
"INFO Trying to download PID using type tv
INFO: pid found in cache
2455: EastEnders - 27/03/2017, BBC One, b08l429m
audiodescribed)
INFO: File name prefix = EastEnders_-_5465._2017-03-27_b08l429m_default
available: 2017-03-27T20:30:00+01:00
brand: EastEnders
categories: Drama,Soaps
category: Drama
channel: BBC One
desc: Michelle reaches breaking point. Whitney goes all out to
support Tina.
desclong: Michelle reaches breaking point. Whitney goes all out to
support Tina.
descmedium: Michelle reaches breaking point. Whitney goes all out to
support Tina.
descshort: Michelle reaches breaking point. Whitney goes all out to
support Tina.
dir: F:\iPlayer-TV
dldate: 2017-03-29
dltime: 20:24:04
duration: 1800
durations: audiodescribed: 1800
episode: 5465. 27/03/2017
episodenum: 5465
episodeshort: 27/03/2017
expires: in 28 days 0 hours (2017-04-26T19:30:00+00:00)
ext: EXT
filename: F:\iPlayer-TV\EastEnders_-_5465._2017-03-27_b08l429m_default.EXT
F:\iPlayer-TV\EastEnders_-_5465._2017-03-27_b08l429m_default.partial.EXT
fileprefix: EastEnders_-_5465._2017-03-27_b08l429m_default
firstbcast: audiodescribed: 2017-03-27T20:30:00+01:00
firstbcastdate: audiodescribed: 2017-03-27
firstbcastrel: audiodescribed: 1 days 23 hours ago
index: 2455
lastbcast: audiodescribed: 2017-03-27T20:30:00+01:00
lastbcastdate: audiodescribed: 2017-03-27
lastbcastrel: audiodescribed: 1 days 23 hours ago
longname: EastEnders
dvfxsd1,dvfxsd2,dvfxsd3,dvfxsd4,dvfxhigh1,dvfxhigh2,dvfxhigh3,dvfxhigh4,dvflow1,dvflow2,dvflow3,dvflow4,hvfxsd1,hvfxsd2,hvfhigh1,hvfhigh2,hvfxhigh1,hvfxhigh2,hvflow1,hvflow2
dvfxsd1=319MiB,dvfxsd2=319MiB,dvfxsd3=319MiB,dvfxsd4=319MiB,dvfxhigh1=164MiB,dvfxhigh2=164MiB,dvfxhigh3=164MiB,dvfxhigh4=164MiB,dvflow1=87MiB,dvflow2=87MiB,dvflow3=87MiB,dvflow4=87MiB,hvfxsd1=387MiB,hvfxsd2=387MiB,hvfhigh1=386MiB,hvfhigh2=386MiB,hvfxhigh1=210MiB,hvfxhigh2=210MiB,hvflow1=121MiB,hvflow2=121MiB
name: EastEnders
nameshort: EastEnders
pid: b08l429m
player: http://www.bbc.co.uk/iplayer/episode/b08l429m
runtime: 30
senum: s01e5465
seriesnum: 1
thumbfile: F:\iPlayer-TV\EastEnders_-_5465._2017-03-27_b08l429m_default.jpg
thumbnail: http://ichef.bbci.co.uk/images/ic/150x84/p01vg679.jpg
timeadded: 1 days 11 hours ago (2017-03-28T07:32:35+00:00)
title: EastEnders: 27/03/2017
type: tv
verpids: audiodescribed: p04y79vv
version: default
versions: audiodescribed
web: http://www.bbc.co.uk/programmes/b006m86d
INFO: 1 Matching Programmes"

[Vangelis forthnet]

Post by Alan Milewczyk
I'm now back in the UK after 4 months in the Philippines

Hi there, dear Alan :-)
Back in the nest, I guess, for you then...
I do hope things have stayed just as you left them...

Post by Alan Milewczyk
so I'm into catch-up mode while new programmes keep on coming
relentlessly.
Last year it took me about 2 months to catch up on 4 months missed TV!

...At least you do manage to properly listen to/watch your GiP downloads...
I think at some point I stopped fetching progs I realised I will never find
the time to properly listen to (am most a radio downloader); with cheap
storage
nowadays, I was dangerously bordering digital hoarding...

Post by Alan Milewczyk
get_iplayer --info --pid b08l429m
(snip)
INFO: No versions of this programme were selected
(available versions: audiodescribed)
At first I wondered if this was an error
on the part of the BBC team
(snip)
I'm wondering am I missing something here?
Any thoughts?

... Your reported issue intrigued me, hence one
of my (notoriously?) long-ish analyses follows...

TL;DR: The manifestation of your issue is caused by an
unfortunate coincidence of two things:
1. Incomplete metadata on the beeb's side for that
particular episode (YES, the BBC is to blame in part)
2. An otherwise very sane commit in the main GiP
perl script, that was pushed on Jan 6, 2016.

Those with traces of Hercule Poirot inside you, do read on!
======================================================

Post by Alan Milewczyk
You are right, GiP 2.99 only picks up the audiodescribed version
(snip)
Version 2.94 can see it but anything after that for some reason can't.
Also in 2.99 in the option to use alternative metadata
streams --ybbcy which also sees the missing version.

Many thanks tellyaddict for those additional clues!

Now, under normal conditions (XML feeds up and running),
GiP uses a prog's XML page
"http://www.bbc.co.uk/programmes/<pid>.xml"
to retrieve many metadata, including versions
availability and their vPIDs (used to point to stream data).
For said programme (pid=b08l429m), if you load the page
in a browser and focus on the <versions> element you get:

<versions>
<version canonical="0">
<pid>b08l4283</pid

Post by Alan Milewczyk
<duration>1800</duration>

<types>
<type>Store only</type>
</types>
</version>
<version canonical="0">
<pid>p04xtjsg</pid>
<duration>1800</duration>
<types>
<type>Store only</type>
</types>
</version>
<version canonical="0">
<pid>p04y79vv</pid>
<duration>1800</duration>
<types>
<type>Dubbed Audio Described</type>
<type>Store only</type>
</types>
</version>
</versions>

So, three (3) versions are present:
v1 = "Store only", vpid=b08l4283 [-> streams exist]
v2 = "Store only", vpid=p04xtjsg [-> no streams]
v3 = "Dubbed Audio Described/Store only", vpid=p04y79vv [-> streams exist]

With GiP 2.94, --info will produce (among other):

title: EastEnders: 27/03/2017
type: tv
verpids: audiodescribed: p04y79vv
verpids: default: b08l4283
verpids: store: b08l4283
version: default
versions: audiodescribed,default,store

while 2.95 will show:

title: EastEnders: 27/03/2017
type: tv
verpids: audiodescribed: p04y79vv
version: default
versions: audiodescribed

where one sees the "store" version not mentioned...
Closely inspecting
"https://github.com/get-iplayer/get_iplayer/compare/v2.94...v2.95"
one can see why; it's because of
"Skip store only version"
https://github.com/get-iplayer/get_iplayer/commit/a5153fc

99.999% of times, the "store only" version lacks any streams,
so rightfully the maintainer pushed the commit to disregard
it; but in this case with "broken" metadata, the first
"store only" version corresponds to the only non-AD, non-SL
version and GiP 2.95+ skips it, taking into account only
the AD flavour...

Workaround 1:
In your version of the main script
(on Windows it's get_iplayer.pl), locate line

next if $type =~ /store/;

(in GiP 2.99 it's line 4286)
and temporarily comment it out (prepend #).
Then you should be able to fetch the non-AD version.

As to why --ybbcy switch, as reported by tellyaddict,
also functions as a workaroud,
1. --ybbcy switch assumes outage of XML metadata pages,
2. commit "Remove fallback metadata method" (2.98+)
https://github.com/get-iplayer/get_iplayer/commit/1b5160b
assumes the same for (and does away with) RDF metadata pages, so
3. GiP only parses JSON playlist URIs
INFO: iPlayer metadata URL (JSON) =
http://www.bbc.co.uk/programmes/b08l429m/playlist.json
and scrapes the prog's iplayer webpage
INFO: iPlayer metadata URL (HTML) [default] =
http://www.bbc.co.uk/iplayer/episode/b08l429m/
so vpid=b08l4283 is successufuly harvested,
because the "skip store only version" rule
doesn't apply here...

Workaround 2: Issue
get_iplayer --pid=b08l429m --ybbcy

CASE CLOSED AND FILED!

As an exercise for the curious, load
http://www.bbc.co.uk/programmes/b08l42d7.xml
and try to understand why GiP 2.99 has no
problem fetching the non-AD version of
EastEnders: 28/03/2017

All the best,
Vangelis.

[Alan Milewczyk]

Post by Vangelis forthnet

Post by Alan Milewczyk
I'm now back in the UK after 4 months in the Philippines

Hi there, dear Alan :-)
Back in the nest, I guess, for you then...
I do hope things have stayed just as you left them...

Hi again dear Vangelis :-)

Yep back in the UK after a tortuous flight back. It was bad enough being
on an excruciatingly long 12.5 hour flight from Guangzhou to Amsterdam,
only to find that the last leg of my journey back to Manchester had been
cancelled due to weather problems, as had many other flights. In the
ensuing chaos in the baggage handing area, my checked in luggage was
"misplaced". I wasn't willing to travel without it, missing replacement
flights I'd been booked on meaning I ending up spending some 12 hours in
Schiphol when all I wanted to do was to sleep in my bed. Anyway, it all
ended well but now I have a leg infection and very painful cellulitis -
been on the antibiotics for 2 days now and no evidence yet of them
kicking in! Sigh... :-(

Post by Vangelis forthnet

Post by Alan Milewczyk
so I'm into catch-up mode while new programmes keep on coming
relentlessly.
Last year it took me about 2 months to catch up on 4 months missed TV!

...At least you do manage to properly listen to/watch your GiP
downloads...
I think at some point I stopped fetching progs I realised I will never find
the time to properly listen to (am most a radio downloader); with
cheap storage
nowadays, I was dangerously bordering digital hoarding...

I don't have a serious hoarding issue with the TV programmes although I
do with radio, but then I am a hoarder (or should it be collector?) :-(

Post by Vangelis forthnet

Post by Alan Milewczyk
get_iplayer --info --pid b08l429m
(snip)
INFO: No versions of this programme were selected
(available versions: audiodescribed)
At first I wondered if this was an error
on the part of the BBC team
(snip)
I'm wondering am I missing something here?
Any thoughts?

... Your reported issue intrigued me, hence one
of my (notoriously?) long-ish analyses follows...
TL;DR: The manifestation of your issue is caused by an
1. Incomplete metadata on the beeb's side for that
particular episode (YES, the BBC is to blame in part)
2. An otherwise very sane commit in the main GiP
perl script, that was pushed on Jan 6, 2016.
Those with traces of Hercule Poirot inside you, do read on!

I do admire your forensic abilities regarding software issues. Must
admit I'm a hardware man and time is a factor in stopping me pursuing
the nuts and bolts of a program what with my countless other projects
currently on the go.

Post by Vangelis forthnet
======================================================

Post by Alan Milewczyk
You are right, GiP 2.99 only picks up the audiodescribed version
(snip)
Version 2.94 can see it but anything after that for some reason can't.
Also in 2.99 in the option to use alternative metadata
streams --ybbcy which also sees the missing version.

Many thanks tellyaddict for those additional clues!

Yes belated thanks too to tellyaddict!

Post by Vangelis forthnet
Now, under normal conditions (XML feeds up and running),
GiP uses a prog's XML page
"http://www.bbc.co.uk/programmes/<pid>.xml"
to retrieve many metadata, including versions
availability and their vPIDs (used to point to stream data).
For said programme (pid=b08l429m), if you load the page
<versions>
<version canonical="0">
<pid>b08l4283</pid

Post by Alan Milewczyk
<duration>1800</duration>

<types>
<type>Store only</type>
</types>
</version>
<version canonical="0">
<pid>p04xtjsg</pid>
<duration>1800</duration>
<types>
<type>Store only</type>
</types>
</version>
<version canonical="0">
<pid>p04y79vv</pid>
<duration>1800</duration>
<types>
<type>Dubbed Audio Described</type>
<type>Store only</type>
</types>
</version>
</versions>
v1 = "Store only", vpid=b08l4283 [-> streams exist]
v2 = "Store only", vpid=p04xtjsg [-> no streams]
v3 = "Dubbed Audio Described/Store only", vpid=p04y79vv [-> streams exist]
title: EastEnders: 27/03/2017
type: tv
verpids: audiodescribed: p04y79vv
verpids: default: b08l4283
verpids: store: b08l4283
version: default
versions: audiodescribed,default,store
title: EastEnders: 27/03/2017
type: tv
verpids: audiodescribed: p04y79vv
version: default
versions: audiodescribed
where one sees the "store" version not mentioned...
Closely inspecting
"https://github.com/get-iplayer/get_iplayer/compare/v2.94...v2.95"
one can see why; it's because of
"Skip store only version"
https://github.com/get-iplayer/get_iplayer/commit/a5153fc
99.999% of times, the "store only" version lacks any streams,
so rightfully the maintainer pushed the commit to disregard
it; but in this case with "broken" metadata, the first
"store only" version corresponds to the only non-AD, non-SL
version and GiP 2.95+ skips it, taking into account only
the AD flavour...

Ah, I get it

Post by Vangelis forthnet
In your version of the main script
(on Windows it's get_iplayer.pl), locate line
next if $type =~ /store/;
(in GiP 2.99 it's line 4286)
and temporarily comment it out (prepend #).
Then you should be able to fetch the non-AD version.
As to why --ybbcy switch, as reported by tellyaddict,
also functions as a workaroud,
1. --ybbcy switch assumes outage of XML metadata pages,
2. commit "Remove fallback metadata method" (2.98+)
https://github.com/get-iplayer/get_iplayer/commit/1b5160b
assumes the same for (and does away with) RDF metadata pages, so
3. GiP only parses JSON playlist URIs
INFO: iPlayer metadata URL (JSON) =
http://www.bbc.co.uk/programmes/b08l429m/playlist.json
and scrapes the prog's iplayer webpage
INFO: iPlayer metadata URL (HTML) [default] =
http://www.bbc.co.uk/iplayer/episode/b08l429m/
so vpid=b08l4283 is successufuly harvested,
because the "skip store only version" rule
doesn't apply here...
Workaround 2: Issue
get_iplayer --pid=b08l429m --ybbcy
CASE CLOSED AND FILED!

Brilliant analysis, Vangelis, thank you.

Post by Vangelis forthnet
As an exercise for the curious, load
http://www.bbc.co.uk/programmes/b08l42d7.xml
and try to understand why GiP 2.99 has no
problem fetching the non-AD version of
EastEnders: 28/03/2017

Ok, let's try.... for my failing programme the relevant section in the
xml was:
<versions><version
canonical="0"><pid>b08l4283</pid><duration>1800</duration><types><type>Store
only</type></types></version><version
canonical="0"><pid>p04xtjsg</pid><duration>1800</duration><types><type>Store
only</type></types></version>

whereas for your working version it was:
<versions><version
canonical="1"><pid>b08l42bq</pid><duration>1800</duration><types><type>Original
version</type></types></version><version
canonical="0"><pid>p04xtj64</pid><duration>1620</duration><types><type>Store
only</type></types></version><version
canonical="0"><pid>p04ybhb3</pid><duration>1800</duration><types><type>Dubbed
Audio Described</type></types></version></versions>

Sorry I've lost the formatting in the cut and paste but basically my
programme had only two versions, "Store only" and "Dubbed Audio
Described" whereas the working programme had three versions, "Original
version", "Store only" and "Dubbed Audio Described".

Not sure of the significance of <version canonical="1"> in yours and
<version canonical="0"> in mine, though!

I must admit my eyes have glazed over in the past when looking at xml
files but now I don't feel as daunted!

Appreciate the analysis as always. Having seen the intolerant attitude
to queries in "the other place", I'm glad I'm here.

Post by Vangelis forthnet
All the best,
Vangelis.

And to you my friend.... I'm in pain so I'm off back to bed with my
painkillers.


Alan

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus