Neat video creating Freeze frames on Render in FCP 10.6.3

resolve technical issues related to use of Neat Video
joshua2577
Posts: 1
Joined: Mon May 23, 2022 6:45 pm

Neat video creating Freeze frames on Render in FCP 10.6.3

Post by joshua2577 »

Hi all,

Don't know if you've come across this but I'm using Neat Video on ProRes 422 footage on an M1 Studio Ultra and it's creating random freeze frames on almost every shot after I render it on the timeline in Final Cut 10.6.3. I did just update, but I was experiencing it on 10.6.2 also. I've removed all the other filters and effects and it's the only plugin I have left on at this point and it's still creating it often for 2 to 3 freeze frames at a time. Wondering if anyone else has encountered this issue and if there might be some settings in the Neat video interface to try.

Thank you,

Josh Meyers
NVTeam
Posts: 2745
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by NVTeam »

We haven't seem such a problem yet, as far as I know. Please prepare a small test project allowing to reproduce this problem. Then please export the test project to XML and upload together with the source clip to https://wetransfer.com/ for us (support at neatvideo.com). We will run it on our machines and see if it is reproducible. Once we can reproduce it, we can identify the cause and possibly find a solution.

Thank you,
Vlad
pau)_
Posts: 2
Joined: Thu Jun 02, 2022 12:55 am

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by pau)_ »

joshua2577 wrote: Mon May 23, 2022 8:37 pm Hi all,

Don't know if you've come across this but I'm using Neat Video on ProRes 422 footage on an M1 Studio Ultra and it's creating random freeze frames on almost every shot after I render it on the timeline in Final Cut 10.6.3. I did just update, but I was experiencing it on 10.6.2 also. I've removed all the other filters and effects and it's the only plugin I have left on at this point and it's still creating it often for 2 to 3 freeze frames at a time. Wondering if anyone else has encountered this issue and if there might be some settings in the Neat video interface to try.

Thank you,

Josh Meyers
HI Joshua,

I am having the same problem. Render/export ProRes 422 on a M1 Studio (Mac OS X 12.4, FCP X 10.6.3, NeatVideo5 (SR) v5.5.2 for FCPX) I've tried deleting all the render files and re-rendering with only NeatVideo applied as a plugin and the same issue occurs. Kinda dead in the water at this point. Have you had any success with fixes or work arounds?
Thanks,
Paul
NVTeam
Posts: 2745
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by NVTeam »

We have reproduced something that may be the same thing that you observe -- a stutter in the render output of FCPX when Neat Video is also applied and this seems to be caused by one of the recent updates of FCP or related Apple software. The problem is not reproducible in an older version (10.6.1) but is reproducible in 10.6.2 and 10.6.3. We are working with Apple to investigate this in more detail and find a solution.

Vlad
pau)_
Posts: 2
Joined: Thu Jun 02, 2022 12:55 am

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by pau)_ »

Thank you very much Vlad!
joema
Posts: 11
Joined: Tue Sep 13, 2016 1:17 pm

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by joema »

I am investigating this issue for my video team. Is there any updated info? Here is what I've observed so far, with most testing done on a Mac Studio M1 Ultra, MacOS 12.4, FCP 10.6.3 and Neat Video 5.5.2 (SR):

- It is not restricted to Neat Video, but can happen on retimed clips, especially if involving temporal effects or transitions, however it's very elusive.
- There is a spectrum of manifestations from very mild to severe.
- In mild cases it can be very difficult and tedious to identify since it seems to require retimed clips and that in turn produces an uneven frame cadence which makes a single anomalous dropped or duplicated frame hard to spot.
- It seems more common if exporting to H264 but it can happen with an all-ProRes workflow
- It involves more than incorrectly-ordered frames. I've sometimes observed torn frames -- see attachment.
- It's not related to hard drive or a specific M1 Ultra machine -- it's been observed on multiple machines.
- It's not specific to M1 Ultra; I've seen it on M1 Max.
- I have not seen it when using temporal noise reduction in Resolve Studio 17.4.6 on M1 Ultra, but I've done less testing on that.
- If it is isolated to M1 Ultra and M1 Max (not regular M1), those are the only CPUs with multiple hardware encode/decode accelerators. It is conceivable in that case the problem could involve a thread safety issue across the multiple parallel accelerators, however that is speculation. It's interesting that FCP 10.6.2 was the version which allegedly better harnessed those accelerators. From studying the Video ToolBox framework and Grand Central Dispatch, it's unclear to me how a programmer would manage multiple threads making async calls to VTCompressionSession.
- In the Video ToolBox framework, hardware acceleration is used by default. The programmer can disable acceleration in the VTCompressionSession call by setting EnableHardwareAcceleratedVideoDecoder to False. However this is not exposed in FCP. If there was a trace flag or debugging menu for that it would help in cases like this.
Attachments
M1UltraFCP10_6_3_TornFrameRetimePlusCubeTrans.jpg
M1UltraFCP10_6_3_TornFrameRetimePlusCubeTrans.jpg (119.61 KiB) Viewed 1908 times
NVTeam
Posts: 2745
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by NVTeam »

We have received reports of the temporal stutter happening on M1 machines and also on one Intel Xeon machine. So it is not limited to M1 architecture. We have been able to directly reproduce this on a M1 Max machine but not on other types of M1 or Intel machines in our labs.

We have been able to reproduce the issue even without using Neat Video, in a test project with only built-in effects of FCPX.

We have added this as a known issue to the official list of known issues and are continuing working with Apple to help them diagnose and fix the problem.

We have not seen the problem shown in your screenshot. It may be related or not related to the problem of stutter.

Thank you,
Vlad
josch
Posts: 4
Joined: Wed Apr 29, 2015 8:26 am

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by josch »

To add my observations:
Because, I now own a Mac Studio, I'm going over some semi finished projects with older versions of NeatVideo, cleaning things up with the recent NV version.
While I see the stutter with ProRes originals, I shot with an original BMPCC, footage (AVC1) I shot with an Olympus E-M10 Mk I has no stutter at all.
My guess would be that formats, the M1 Max can handle with its Media Engine, will be prone to the stutter, while other formats are perfect.
joema
Posts: 11
Joined: Tue Sep 13, 2016 1:17 pm

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by joema »

M1 Ultra has two H264/HEVC decode engines, four H264/HEVC encode engines and four ProRes encode/decode engines. M1 Max has half of those.

On current versions of MacOS, hardware acceleration is enabled by default for apps. It can be disabled programmatically by apps using the Video Toolbox framework when calling VTDecompressionSession by setting the EnableHardwareAcceleratedVideoDecoder option to false. I wish FCP had a trace flag or debugging option for this.

Be advised there is a superficially similar but separate problem with the FCP built-in video noise reduction (NR) effect. This apparently has existed since at least 10.5.2 and exists today on 10.6.3 on Intel and Apple Silicon. In this case a retimed or rate conformed clip containing FCP NR may appear to periodically drop and duplicate frames. An easy way to reproduce this is by putting a 29.97 fps clip in a 59.94 timeline, selecting Modify>Retime>Automatic Speed which retimes the clip to 200%, making playback smooth. However if FCP built-in video NR is added, the retime operation is disregarded and every other frame is duplicated. Many variations of this exist for various rate conforming paths. I will likely file a bug with Apple on this today, plus will discuss with Pro Apps escalation support.

Due to the above FCP bug IMO no user should use FCP built-in NR. If any users are concerned about the current issues with Neat Video on FCP, using the built-in NR is not the answer.

When I get the bug filed I'll post the details here.
Ivorydom
Posts: 1
Joined: Fri Jul 08, 2022 8:30 am

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by Ivorydom »

Hello,

I am experiencing the same bug over here on both M1 Max Macbook Pros and on Intel based ones.

I thought I'd post here before I open a new thread.

The pattern is: if you use Neat Video on a compound clip, then you get freeze frames and the syncing is lost as well.
Massive problem when you do a long export and realize everything is wrong.

The only workaround is to break apart clips and use Neat video on each clip individually.

I hope this helps some users out there but it is certainly not ideal.

Any news about this issue?
joema
Posts: 11
Joined: Tue Sep 13, 2016 1:17 pm

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by joema »

Thanks for the comment about compound clips, I will test that. If breaking those apart works for you, OK but in my testing the problem is not isolated to Neat Video (compound clips or not), therefore it's not caused by a bug in Neat Video. It can happen when using other multi-frame effects such as the FCP built-in "Trails" effect. The behavior is highly intermittent and difficult to reproduce. This leads to cases where you change something, that appears to cause it or prevent it, yet that change was not really related.

The problem did not happen on FCP 10.6.1 but began with 10.6.2, which had some performance improvements. This was mentioned in the 10.6.2 release notes: "Optimized playback and graphics performance for M1 Max and M1 Ultra on the new Mac Studio". While that was done for M1 Max/Ultra the underlying source code change may have affected Intel as well: https://support.apple.com/en-us/HT201237

The behavior implies a timing or race condition, whereby under some conditions threads do not properly synchronize their work and inadvertently use a buffer which should be locked. This can happen if multiple threads are making asynchronous calls to the underlying Video Toolbox framework. Note this statement from the WWDC20-10090 talk "Decode ProRes with AVFoundation and VideoToolbox":

"When a CVPixelBuffer allocated from a pool is released and no longer in use, the IOSurface will go back into the CVPixelBufferPool so that the next CVPixelBuffer allocated from the pool can reuse that memory." One little mistake and under narrow conditions a thread can grab the wrong buffer.

Such problems are among the most difficult to fix due to the fleeting, intermittent nature. Fortunately there are a lot of people working on this. I had a long discussion with Apple yesterday; they remotely connected to my machine to observe the problem, and I will be uploading a package to them to help define and reproduce this. Neat Video engineers are also working with Apple on this, even though it's not Neat Video's fault.

The best current solution is revert to FCP 10.6.1, but that may be difficult since there were library changes and 10.6.1 cannot open a 10.6.3 library. You should be able to export a project or library XML from 10.6.3 and import that into 10.6.1, assuming you have 10.6.1. Unfortunately Apple does not have a site for back-level versions of FCP, so it's good practice before each update to zip the current .app bundle and save that.

FCP has a built-in video noise reduction effect which is very inferior to Neat Video, but I recommend against using that, even as a contingency. It has its own problems which may cause cadence errors on retimed clips. Those are not new to 10.6.2 but existed since at least 10.5.2.
joema
Posts: 11
Joined: Tue Sep 13, 2016 1:17 pm

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by joema »

Vlad/Team: what is the latest confidence this can be reproduced on Intel? I have a semi-reliable small replication that works on M1 Ultra and M1 Max MacBook Pro 16 but I cannot reproduce it on a top-spec 2019 i9 MacBook Pro 16. All are running FCP 10.6.3 and Monterey 12.4. All have the same 4k/23.98 ProRes 422 media file and the same project loaded via XML. I don't have a regular M1 or M1 Pro to test.

It definitely does not happen on FCP 10.6.1 (on Monterey 12.4). As you stated it began with 10.6.2 and continues in 10.6.3.

I'll be writing up a report on this for my video team tonight or tomorrow, also filing a bug with Apple. If we knew the scope of the problem, that would help assess the risk. E.g, if it's isolated to M1 Max/Ultra (or that plus Xeon-powered Mac Pros) that is less severe than if it's any Intel Mac running FCP >= 10.6.2.

If it can happen on any machine running 10.6.2 or later, that implies any rendered media (whether using Neat Video or not) that was produced since 10.6.2 was released on April 12 has a small but finite chance of having misordered frames. In some cases a QC search might be needed through all that media, including archives. If it is isolated to just M1 Max & Ultra, that narrows the possible scope of that search.

Thanks again to Neat Video for working so hard on this, even though it's not a problem with your code.
NVTeam
Posts: 2745
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by NVTeam »

I believe we received one report where a Xeon-based system was showing the same symptoms but we have not been able to reproduce that on any of our Intel-based machines. I will double-check with my colleagues tomorrow.

Vlad
joema
Posts: 11
Joined: Tue Sep 13, 2016 1:17 pm

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by joema »

I list below the most reliable simple scenario which reproduces the problem. This works using only a short retimed 1080p ProRes 422 clip and the built-in FCP "Trail" effect (thanks to Neat Video for that tip). It works for 23.98 in a 23.98 timeline, 29.97 in a 29.97 timeline and 59.94 in a 59.94 timeline. On my M1 Ultra and M1 Max it quickly and reliably reproduces the problem.

1 - Use an approx. 4 sec 1080p ProRes 422 clip which was shot using a smooth steady horizontal panning rate. A good panning rate is about 2 sec per frame width. That is relatively fast (by cinematography standards) but it helps visualize the problem. Any of the above frame rates should work. I have not tested 24.0, 25.0 or 50.0 fps.

2 - In FCP 10.6.3 create a new library for testing. In my testing I had FCP preferences>import set to "leave files in place", but I don't think it matters. I suggest having FCP preferences>playback>background render turned off for better control.

3- In FCP 10.6.3, create a project (aka timeline) using auto settings. The first clip added will determine the project frame rate.

4 - Add the clip to the timeline.

5 - Add the built-in FCP "Trail" effect to the clip. Select the effect option "Trail On" to "dark", which makes any frame problem easier to see.

6 - Retime the clip to 25% normal speed. Other settings work but I had good results with this one. The keyboard shortcut for retiming is CTRL+OPT+R.

7 - Duplicate that clip and drag it on top of the 1st one. The FCP command is click and drag up on the clip while holding the OPT key. If any difficulty doing that, select the clip, copy with CTRL+C, then do Edit>Paste as Connected Clip. It will retain the 25% retiming and Trail effect.

8 - Select top clip, then in Inspector>Transform, set Scale (All) to 60%. Adjust X and Y position so the clip borders are in the upper left of the frame

9 - Repeat for bottom clip but position its borders in the bottom right, so there is a little overlap between the two clips.

10 - Do a CMD+A in the timeline to select both clips. Render them to cache with CTRL+R. The render dots will go away when completed.

Tomorrow I will provide a file package including project XML and test files to make recreating this easier.

11 - Use JKL commands to scrub through the timeline at about 4x speed (hit the L key twice). If the timeline is not rendered to cache the problem doesn't manifest as much.

12 - Problem replication seems quite reliable (maybe 50% or better) but if it doesn't happen, delete render cache and re-render the timeline. To delete: select library in left sidebar, then do File>Delete Generated Library Files>Delete Render Files>All. Then select both timeline clips with CMD+A and re-render to cache with CTRL+R. Then scrub through the timeline using JKL commands.

When the problem happens it often (but not always) will persist into an exported ProRes 422 file. In theory there should be a 1:1 mapping of the render cache to an exported ProRes file because as an export optimization FCP concatenates the render segments and writes the output file. However there are some cases where it must re-read the source file, during which the problem may not happen.
joema
Posts: 11
Joined: Tue Sep 13, 2016 1:17 pm

Re: Neat video creating Freeze frames on Render in FCP 10.6.3

Post by joema »

Here is a DropBox link to a simple problem replication, along with a .txt file describing how to run it. The package is very small, only about 62 megabytes. It includes an FCP project XML file which recreates the needed timeline. Initial setup is very simple: just load the XML, then relink to the single ProRes file.

https://www.dropbox.com/s/tpkp977lc4qf4 ... 1.zip?dl=0

On an M1 Max or M1 Ultra running FCP 10.6.2 or 10.6.3 on Monterey 12.4, this will reliably reproduce the problem, often with the 1st attempt. Neat Video is not used, only the FCP built-in "Trails" effect on a duplicated short 29.97 fps clip retimed to 25% speed and scaled so they are both visible.

It is definitely more likely with multiple streams vs single stream. That might imply a race condition introduced by the changes in FCP 10.6.2 which attempted to improve playback performance on M1 Max and Ultra.

Edit/Add: As of right now, it cannot be reproduced on regular M1 or Intel, only M1 Max and Ultra. Some other users are now trying to reproduce it on a 2019 Mac Pro using the above scenario. I will report those results when available.
Post Reply