With 3.07 .dash.m4v left behind

[Nick Payne]

I've had three instances since installing 3.07 where the download of a
programme finishes without any problems or errors indicated in the
console output, but as well as the mp4 file there is also a .dash.m4v
file of slightly smaller size. I assume this is meant to be deleted once
the conversion to mp4 has been completed, but this isn't happening
consistently.

OS is Windows 10 x64.

Nick

[Ralph Corderoy]

Hi Nick,

I assume this is meant to be deleted once the conversion to mp4 has
been completed, but this isn't happening consistently.
OS is Windows 10 x64.

Does Windows allow a file to be deleted if a process still has it open?
Unix does. Perhaps this is another manifestation of my guess in
http://lists.infradead.org/pipermail/get_iplayer/2017-November/011371.html
that got no replies, e.g. by people that know Windows and can track what
files are open when.

[Nick Payne]

Post by Ralph Corderoy
Hi Nick,

I assume this is meant to be deleted once the conversion to mp4 has
been completed, but this isn't happening consistently.
OS is Windows 10 x64.

Does Windows allow a file to be deleted if a process still has it open?
Unix does. Perhaps this is another manifestation of my guess in
http://lists.infradead.org/pipermail/get_iplayer/2017-November/011371.html
that got no replies, e.g. by people that know Windows and can track what
files are open when.

In that previous case (temp file not able to be renamed) a permissions
error appears on the console. In this case no problem is indicated at all.

[Ralph Corderoy]

Hi Nick,

Post by Nick Payne

Post by Ralph Corderoy

I assume this is meant to be deleted once the conversion to mp4
has been completed, but this isn't happening consistently.

Does Windows allow a file to be deleted if a process still has it
open? Unix does. Perhaps this is another manifestation of my guess
in
http://lists.infradead.org/pipermail/get_iplayer/2017-November/011371.html

In that previous case (temp file not able to be renamed) a permissions
error appears on the console. In this case no problem is indicated at all.

That just means get_iplayer is ignoring the failure to remove the file
rather that report it to you. A quick skim of all the `unlink' function
calls show none have their return value checked.

You might want to adjust `unlink $foo;' to

unlink $foo or
main::logger "WARNING: unlink $foo failed: $!\n";

`$!' will hopefully give a Windows-specific error indicating why the
removal failed, e.g. `file open'.

Post by Nick Payne

Post by Ralph Corderoy
by people that know Windows and can track what files are open when.

Google suggests FileMon did this in the past, and it's been superseded
by Process Monitor.
https://docs.microsoft.com/en-us/sysinternals/downloads/procmon

[RS]

From: Ralph Corderoy
Sent: Wednesday, December 6, 2017 2:08 PM

Post by Ralph Corderoy
That just means get_iplayer is ignoring the failure to remove the file
rather that report it to you. A quick skim of all the `unlink' function
calls show none have their return value checked.

Hi Ralph and Nick

What level of error reporting is being used? The default (since v3.06)

--ffmpeg-loglevel=fatal

does not even report running out of disk space for the output file. If you
use

--ffmpeg-loglevel=info or --ffmpeg-loglevel=verbose

it may tell you what the error is.

Running get_iplayer with --verbose may also help to narrow down where the
failure is occurring.

Best wishes
Richard

[Nick Payne]

Post by RS
From: Ralph Corderoy
Sent: Wednesday, December 6, 2017 2:08 PM

Post by Ralph Corderoy
That just means get_iplayer is ignoring the failure to remove the file
rather that report it to you.  A quick skim of all the `unlink' function
calls show none have their return value checked.

Hi Ralph and Nick
What level of error reporting is being used?  The default (since v3.06)
--ffmpeg-loglevel=fatal
does not even report running out of disk space for the output file. 
If you use
--ffmpeg-loglevel=info or --ffmpeg-loglevel=verbose
it may tell you what the error is.
Running get_iplayer with --verbose may also help to narrow down where
the failure is occurring.

Doesn't look as though using --verbose gives any information to help
with the problem. Here's the last part of the console output from
downloading the latest Peaky Blinders episode with --verbose. The output
folder at the end of the download still contains the .dash.m4v and
-temp-32694.mp4 files as well as the expected .mp4 file.

===============================================================
INFO: Converting to MP4
INFO: Command: "ffmpeg" "-loglevel" "fatal" "-stats" "-y" "-i"
"D:\Users\Nick\Videos\Peaky_Blinders-s04e04-Dangerous.dash.m4a" "-i"
"D:\Users\Nick\Videos\Peaky_Blinders-s04e04-Dangerous.dash.m4v" "-c:v"
"copy" "-c:a" "copy" "-map" "0:a:0" "-map" "1:v:0"
"D:\Users\Nick\Videos\Peaky_Blinders-s04e04-Dangerous.partial.mp4"
frame=172416 fps=34275 q=-1.0 Lsize= 1741151kB time=00:57:28.29
bitrate=4136.4kbits/s speed= 685x
INFO: Command exit code 0 (raw code = 0)
INFO: Loading recordings history
INFO: Downloading thumbnail
INFO: Getting URL: Loading Image...
INFO: Tagging MP4
INFO: Command: "AtomicParsley"
"D:\Users\Nick\Videos\Peaky_Blinders-s04e04-Dangerous.mp4" "--freefree"
"--overWrite" "--stik" "TV Show" "--advisory" "remove" "--copyright"
"2017 British Broadcasting Corporation, all rights reserved" "--title"
"Dangerous" "--artist" "BBC Two" "--albumArtist" "BBC TV" "--album"
"Peaky Blinders: Series 4" "--grouping" "Drama,Crime,Classic & Period"
"--composer" "BBC iPlayer" "--genre" "Drama" "--comment" "Tommy's
enemies edge nearer as an old friend makes a welcomed return." "--year"
"2017-12-06T21:00:00Z" "--tracknum" "4" "--disk" "4" "--lyrics" "The
Peaky Blinders are lured by the Italians into a cat-and-mouse chase on
the streets of Birmingham, where it becomes clear that Tommy has met his
match. Trapped in Small Heath, Tommy tries to console himself with a
visit from an old flame but it soon becomes clear that he can't always
get what he wants.

As his factory lies idle, Tommy confronts the possibility that the
Communists might win and he will be deemed a traitor to his class.
Meanwhile, Changretta prepares to spring another trap.

EPISODE
http://www.bbc.co.uk/iplayer/episode/b09j0hxs

SERIES
http://www.bbc.co.uk/programmes/p05hgs13" "--description" "Tommy's
enemies edge nearer as an old friend makes a welcomed return."
"--longdesc" "The Peaky Blinders are lured by the Italians into a
cat-and-mouse chase on the streets of Birmingham, where it becomes clear
that Tommy has met his match. Trapped in Small Heath, Tommy tries to
console himself with a visit from an old flame but it soon becomes clear
that he can't always get what he wants.

As his factory lies idle, Tommy confronts the possibility that the
Communists might win and he will be deemed a traitor to his class.
Meanwhile, Changretta prepares to spring another trap." "--hdvideo"
"true" "--TVShowName" "Peaky Blinders: Series 4" "--TVEpisode" "s04e04"
"--TVSeasonNum" "4" "--TVEpisodeNum" "4" "--TVNetwork" "BBC Two"
"--artwork" "D:\Users\Nick\Videos\Peaky_Blinders-s04e04-Dangerous.jpg"

 Started writing to temp file.
 Progress: ======================================================> 98% -|
 Finished writing to temp file.
INFO: Command exit code 0 (raw code = 0)

D:\Users\Nick>
===============================================================

[Ralph Corderoy]

Hi Nick,

Post by Nick Payne

A quick skim of all the `unlink' function calls show none have
their return value checked.

...

Post by Nick Payne
Doesn't look as though using --verbose gives any information to help
with the problem.

Well, it wasn't going to include anything about the unlinks returning
errors and what those errors were. :-)

[RS]

From: Ralph Corderoy
Sent: Thursday, December 7, 2017 8:11 AM

Post by Ralph Corderoy

Post by Nick Payne
Doesn't look as though using --verbose gives any information to help
with the problem.

Well, it wasn't going to include anything about the unlinks returning
errors and what those errors were. :-)

I was hoping to see what was happening in ffmpeg. For that you need to set

--ffmpeg-loglevel=info

The new default --ffmpeg-loglevel=fatal tells you nothing.

I have never used any of the DVF modes. Has anyone else used them and did
they work correctly? I have used --hls-hq-audio with HLS which also
downloads video and audio separately and combines them with the ffmpeg -map
option. That works correctly.

The only occurrence of the -map option is at line 5955, to build the ffmpeg
command at 5970, so it must be the same code for both.

I don't know the usage for unlink. At 5988

{unlink( $audio_file, $video_file );
}

acts on two files together. You could try separating them as

{unlink $audio_file;
unlink $video_file;
}

Best wishes
Richard

[RS]

Post by RS
I don't know the usage for unlink.

The documentation is here.

https://perldoc.perl.org/functions/unlink.html

I have just downloaded b09hm0v5 at DVFxsd. It worked fine and deleted the
intermediate files correctly. The only problem was that it gave me HE-AAC
audio, which I did not want.

Best wishes
Richard

[RS]

Post by RS
I have just downloaded b09hm0v5 at DVFxsd. It worked fine and deleted the
intermediate files correctly. The only problem was that it gave me HE-AAC
audio, which I did not want.

I have also downloaded the first two minutes of Peaky Blinders, b09j0hxs, at
DVFhd. That also worked correctly.

The clue may be in the speed of your machine.

D:\Users\Nick\Videos\Peaky_Blinders-s04e04-Dangerous.partial.mp4"
frame=172416 fps=34275 q=-1.0 Lsize= 1741151kB time=00:57:28.29
bitrate=4136.4kbits/s speed= 685x

As well as separating the two files in the unlink statement at 5988 you
could try inserting a delay between them as Ralph suggested.

Best wishes
Richard

[Ralph Corderoy]

Hi Richard,

Post by RS
As well as separating the two files in the unlink statement at 5988 you
could try inserting a delay between them as Ralph suggested.

No, I didn't. :-) I did suggest adding a delay *before* attempting to
unlink to allow all the bits of ffmpeg hanging around after the main
thread has exited to themselves exit. If that's done, and the problem
doesn't reoccur after enough attempts compared to its previous
frequency, then that would suggest I'm right about the cause.

Best of all would be to collect `$!' when one of the unlinks returns
false as I suspect that will be a specific `file is open' error.

[Ralph Corderoy]

Hi Richard,

{ unlink( $audio_file, $video_file ); }
acts on two files together.

That's fine. If removing either of those fails then unlink returns
false, setting `$!' to an error. You only need to do them separately if
you want to determine which had the error, and collect possibly
different reasons for the errors.

This problem and the other rename one look to be race conditions inside
ffmpeg; I'm suggesting it exits allowing get_iplayer to continue before
all of ffmpeg's threads have finished so some of the files it had open
are still open. As a race condition, it isn't likely to be trivially
repeatable, e.g. same PID.

Examination of ffmpeg might show this is possible. There may be ffmpeg
bugs already reported on it. And I suspect Process Monitor or something
similar could track the file and thread activity across all the
processes so post-inspection can confirm; but that's Windows and I
don't know it.

[Alan Milewczyk]

Post by Ralph Corderoy
Hi Nick,

I assume this is meant to be deleted once the conversion to mp4 has
been completed, but this isn't happening consistently.
OS is Windows 10 x64.

Does Windows allow a file to be deleted if a process still has it open?
Unix does.

Not unless you either unlock the file or kill the process. There are
plenty of Third Party apps that will do this, such as IObitlocker and
Unlocker.

Post by Ralph Corderoy
Perhaps this is another manifestation of my guess in
http://lists.infradead.org/pipermail/get_iplayer/2017-November/011371.html
that got no replies, e.g. by people that know Windows and can track what
files are open when.

A


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus