Repository navigation
Unexpected behavior from detect-content #96
Description
Activity
Hi @rsomani95;
Thanks for sharing this with me, I'll look into this when I have some time. I haven't analyzed the source video you're using in the notebook yet, but will do so as soon as possible.
Just so you know, the way detect-content works is that any frame delta exceeding the threshold triggers a new scene cut, but after doing so, any new scene cuts are ignored for N frames (default for N is 10). This is to prevent noise/flashes from triggering false positives, while ensuring that the new scenes are at least captured within N frames. It seems this might explain some (but not all) of the behaviour you're describing.
One thing I'm looking into doing for the future to solve this issue is, let's say we have 3 frames, A, B, and C, and the delta between A and B is very large (indicating a new scene). It may also be worth taking the delta of A and C, in this case, to see if frames A and C are related/part of the same scene, in which case this would be considered a false negative.
This is part of flash detection, which seems to be causing some issues in your analysis (very thorough by the way, well done). See #35 for some more discussion on the matter.
Thanks for the submission, and the detailed analysis - this will definitely be useful for improving the detect-content algorithm.
I've done some work to make your life easier. Clone this directory and follow along with the
tdk-bankrobbery/PySceneDetect_Analysis.ipynbnotebook to see what exactly the issue is. To give you a glimpse -- this is what the notebook offers:
I had to strip the output of most of the cells to manage filesizes.
Just so you know, the way detect-content works is that any frame delta exceeding the threshold triggers a new scene cut, but after doing so, any new scene cuts are ignored for N frames (default for N is 10). This is to prevent noise/flashes from triggering false positives, while ensuring that the new scenes are at least captured within N frames. It seems this might explain some (but not all) of the behaviour you're describing.
This explains the undetected cuts. Frame #3052 in the notebook might also offer some insight into this as well.
One thing I'm looking into doing for the future to solve this issue is, let's say we have 3 frames, A, B, and C, and the delta between A and B is very large (indicating a new scene). It may also be worth taking the delta of A and C, in this case, to see if frames A and C are related/part of the same scene, in which case this would be considered a false negative.
I don't think this will work out too well -- A and C could have a delta of ~0 but still be completely different scenes -- this is well documented in the notebook.
Thanks for the submission, and the detailed analysis - this will definitely be useful for improving the detect-content algorithm.
Happy to contribute :)
Reacted by Brandon CastellanoUpdate --> I've experimented with another peak-detection algorithm that worked perfectly! I've added this to a new section in the notebook.
Thanks for your contributions @rsomani95. I'll definitely be looking into this when I have some more time on my hands, very nice analysis.
Reacted by Rahul Somani- Repository owner locked and limited conversation to collaborators
on Jul 4, 2021
Hello. Firstly, great project!
When using the
detect-contentoption, I observe some unexpected behavior. While the cuts are detected accurate, the exact timing of the cuts is off. This is very odd, since the frame at which the cut is detected doesn't have much of acontent_valdifferential as compared to the frame where the cut actually happened.I've documented this here after taking a closer look at the stats file.
This notebook is quite long, but well organised, so head over to
Exploring & Analysing the Data →
Looking At Detected Cuts And Their Surrounding Frames →
Correctly Detected (But How?)