NeatVideo doesn't work with Premiere Pro tone mapping

[Trixter]

This year, Premiere Pro added colorspace tone mapping, which is a feature that (primarily) allows users to drop any footage into a timeline without converting it to that timeline's color space. However, NeatVideo doesn't work when you use that feature: Adding NeatVideo to such a clip in such a timeline alters the luminance of the clip, crushes blacks, and changes the saturation of the clip. NeatVideo is not supposed to do that, and has never done that. Are the developers aware of this issue, and are they working on a fix?

Background information:

I shoot 10-bit 4:2:2 V-LOG footage. For many years prior to the Premiere Pro 2023 update, my workflow for delivering Rec.709 content was:
- Import footage
- Drop footage onto sequence timeline
- Apply NeatVideo to a clip (NeatVideo works best when it is the very first effect applied to a clip)
- Use a LUT on that clip to transform the footage from V-LOG to Rec.709.

With the Premiere Pro 2023 update, it is now possible to work with the footage directly using automatic tone mapping, but NeatVideo transforms the color and luminance values incorrectly when that happens. For example, this is how the modern workflow is supposed to go:
- Import footage, then use Modify -> Interpret footage -> color space override and pick v-log
- In sequence settings, ensure "auto tone-map media" is checked
- Drop clip onto Rec.709 timeline; it looks normal, thanks to automatic tonemapping, no LUT is needed
- Apply NeatVideo to a clip

As soon as I perform that last step, NeatVideo changes the overall luminance of the clip, crushes the blacks, and changes color saturation slightly. Proceeding with making a noise profile does not help; the clip is de-noised, but still has the wrong luminance and color saturation.

Is there an ETA for a fix? If not, is there a workaround?

[NVTeam]

This looks like that old design problem / bug of Premiere itself showing up in yet one more scenario. The problem is that Premiere processes video data differently when working using GPU vs using CPU alone.

Lets demonstrate that directly. Please remove Neat Video from the clip, then go to Premiere's menu File > Project Settings > General..., switch Video Rendering and Playback Renderer to "Mercury Playback Engine Software Only" and observe how the image in preview of Premiere immediately changes (even without Neat Video). This clearly demonstrates that it is Premiere who is changing the image, not Neat Video. It is a problem of Premiere: it processes image differently in GPU vs CPU modes. This problem shows up when you apply some effects because with certain effects Premiere decides to switch its processing from GPU to CPU, which leads to that problem of Premiere's CPU processing path producing different results.

We reported this problem to Adobe on multiple occasions but so far it has not been fixed yet.

Perhaps a possible workaround is to use the old way of applying a LUT after Neat Video, instead of using these new tools of Premiere 2023.

Vlad

[Trixter]

Unfortunately, this appears to be a different problem; hardware or software, the behavior still persists. I have prepared a video demonstrating what happens in both instances:

https://youtu.be/8SZviQ2wliE

Even in software-only mode, it is still significantly changing the picture.

Premiere 2024 is evolving their color management even more, in a new "settings" tab in the Lumetri panel, so I hope you're able to figure out what's happening. (Even if it cannot be fixed, it would be good to know what is causing the problem, and this is going to affect many more people than myself in the future as cell phones move towards shooting HLG by default.)

[NVTeam]

Please kindly prepare a small test project with one short clip and Neat Video applied to it. Please verify that this test project allows to reproduce the problem on your machine and then please upload the test project and source clip to https://wetransfer.com/ for us (support [at] neatvideo.com). We will run in our machines and determine what exactly is not working correctly in that scenario. The issue can be related to the project settings as well as to specific clip format, so a reproducible test case is important.

Thank you,
Vlad

[Trixter]

I have prepared a small test project (In Adobe Premiere Pro 2024, which was just released, which behaves in exactly the same way) that illustrates the problem and sent it to you via wetransfer as you requested. It uses the exact same footage as in the screencast I posted. The file is "Neatvideo Changing levels Test.zip", let me know if you didn't receive it.

I tested a few more scenarios: Project set to GPU, Project set to CPU, Neatvideo set to GPU only, Netvideo set to CPU only, sequence set to Maximum Bit Depth on and off. In all eight combinations, the behavior was present.

To recap: Adding a clip with a colorspace override to a timeline that has auto-tone-mapping enabled, then adding neatvideo to that clip, shifts the luminance and colors of that clip. Building a noise profile does not affect the incorrect behavior.

[NVTeam]

Thank you for sending the test project. It will take us some time to update our machines for this new Premiere, and then we will run the tests using the test project.

Premiere's incorrect behavior is triggered by adding a video effect, yes, that is correct. The effect doesn't even have to do anything to the video. It is just a catalyst of the problem. We will see what exactly is triggered in this specific case.

Thank you,
Vlad

[Trixter]

Premiere's incorrect behavior is triggered by adding a video effect, yes, that is correct. The effect doesn't even have to do anything to the video.

I was unable to replicate this behavior you're describing; I tried some modern (Levels, Proc Amp) and "Obsolete" (Fast color corrector) and they didn't change anything when applied. Are there any particular effects you've tested that does this?

[NVTeam]

Please excuse the delay on our side. We have spent some time on studying this issue, which is on one hand looks somewhat new in 2023 and 2024 versions, but on the other hand it is still another manifestation of the same old problem of Premiere that I mentioned above: Premiere processes video differently producing different results in different situation, for example, when working with a temporal effect vs non-temporal one, when using the GPU vs CPU, when working in high bitdepth vs low bitdepth. The difference is directly visible with naked eye and that is not normal. Different built-in effects can in certain cases used to directly see these differences, for example Echo and Posterize Time. However it is not easy to demonstrate that in this particular case using these effects alone. It is possible to do with Neat Video but it is less obvious.

On the other hand, now that we know about the cause of the problem we can suggest a workaround. For example in your test project that you sent us earlier, if you add the Color Balance effect (with all 0s, meaning no actual adjustments are applied) just before/above Reduce Noise v5, then the problem of changed image should go away, because Color Balance will force Premiere to work in one specific mode rather than to switch between different modes. This could also be a practical workaround in your situation. There is a drawback though: the render speed may be somewhat lower.

A proper solution would be to fix Premiere to make it produce the same results in different modes. If Adobe engineers would like to do that, we invite them to contact us to receive additional technical details.

Thank you,
Vlad

[Trixter]

When I add Color Balance with all 0s, the image immediately shifts in the same manner that Neat Video does. As this is an Adobe Premiere built-in effect, this proves the problem is with Premiere, and not NeatVideo. Thank you for analyzing the issue.

While your workaround did not work for me, I found one that did: On the same panel where you can force a working colorspace override ("Override Media Color Space"), you can instead use the selection immediately above that and choose an "Input LUT" and pick a LUT right there. This works consistently and does not cause an issue with neatvideo (or other effects) changing the luminance, as it is effectively the same thing as manually adding a LUT in the Lumetri panel, but has the advantage of the LUT staying with the footage even if the Lumetri effect is removed and/or the footage is cut into clips.

Adobe had better fix this at some point, or they're going to get a lot of bug reports and angry customers. The iphone 15 can now shoot to prores directly in a log colorspace, so even casual people may start using HLG and log footage in Premiere.

[NVTeam]

Thank you for sharing your finding regarding Input LUT. It may indeed be a good option for such situation. We will test it from our side too as a possible general recommendation.

I agree with you that this type of issues will become more common now with more people using these news color spaces offered by the new hardware. Hopefully this will be addressed in Premiere too. We keep watching this and doing what we can to convince Adobe to make it work.

Thank you,
Vlad