Help...Building noise profiles...NEVER WORKS! :(

questions about practical use of Neat Video, examples of use
NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Help...Building noise profiles...NEVER WORKS! :(

Post by NV_Fan » Tue Mar 15, 2016 1:20 am

Hi guys, how are you all doing?

I'm going to try and keep this concise but I have downloaded and carefully printed the calibration targets onto a nice white thick paper and proceeded many times with my camera capturing a few seconds of footage at each ISO setting.

I am throwing 500W of light in the scene so I can get it very well and evenly exposed, shooting at f1.4 so I can get it slightly out of focus (giving the coloured squares a smooth look and theoretically would read them better) and using an ND filter to expose each shot correctly (exposing for highlights then expose 1-stop). Of course every time a change was made (in therms of ISO setting and exposure) the white balance was corrected.

The end result has the exposure looking consistent throughout ISO 160 to ISO 12800, with the exception that it was getting grainier.

This was the second time I have taken a lot of time trying to set this up and build the profiles but every time I take it to Neat Video to manually fine-tune them it NEVER picks up all the frequencies, many of the squares in the calibration target just read the same measured data in Neat Video whilst some others just never get recognized.

I am at my wits-end! I am trying to build the perfect profile for my camera and in a controlled environment which should lead me to getting good profiling, but it always fails and never works.

Am I doing something wrong? Please can someone probably explain why could this be happening?? I would really appreciate it guys!

NVTeam
Posts: 2286
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Post by NVTeam » Tue Mar 15, 2016 5:47 pm

Do you shoot the target in a controlled studio environment or in the actual conditions of the real shooting process? It may be better to do that latter - to include the target into the actual shot in each scene (if shooting/lighting conditions of the scenes change). Otherwise you may be building good profiles for the studio conditions but they would not match the conditions of the actual working scene, for example because lighting would be different, and then the noise profile will simply not match the properties of the actual noise in the working shot. Which is why it is usually better to build a noise profile from the working shot itself, even if the target is not included in it. That is the preferred way of building profiles with Neat Video.

Preparing a reusable set of profiles is fine too but then when you load one of them to process a working shot, you should at least run Auto Fine-Tune on the shot to adapt the profile to that specific clip.

Hope this helps,
Vlad
Image Image Neat Video team
noise reduction for video and photos

NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Post by NV_Fan » Sun Mar 20, 2016 11:20 pm

Hi Vlad,

Thanks for the reply and apologizes for my late response.

Even as what you've said, there has been a time were I have shot the calibration target during a scene and once again it still does not pick up all the squares on the card :(

Why is this being the case? And what does one have to do inorder to have them all registered perhaps??

I'm just a bit concerned because from my understanding the target is meant to work in this manner, and it just seems a bit pointless if I or others can get it to work :(

NVTeam
Posts: 2286
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Post by NVTeam » Sun Mar 20, 2016 11:44 pm

NV_Fan wrote:Even as what you've said, there has been a time were I have shot the calibration target during a scene and once again it still does not pick up all the squares on the card :(
I wouldn't worry about that much because it is not strictly necessary to explicitly cover all points. That probably happens because the target is not exposed in such a way as to make the black and white areas exactly black and white. Again, that is no big problem.

The target is often better than no target because it provides multiple uniform areas of different brightness that contain only noise, which can be measured by Neat Video. A regular image may not contain so many areas suitable for profiling. So even if not all points are covered, it is still a significant help from the standpoint of noise analysis.

Vlad
Image Image Neat Video team
noise reduction for video and photos

NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Post by NV_Fan » Mon Mar 21, 2016 6:27 pm

I can assure you that the blacks and white in the scene or chart were of black and white, every time an exposure changes (ISO) and with the ND filter to compensate the amount of light, a white balance gets corrected in camera. I also would kind of worry as this tends to fall into slight inaccurate results.

What tends to happen is sometimes the same squares in the software are matched more than once when calibrating from the square targets and they either have higher or lower values when selected (hope this makes sense?).

Also shadows are in any situation where noise/grain would reside, and what I often get is shadows being the less noisiest than those approaching the highlights. The end result looks somewhat wrong and this is something I am trying to rectify.

Thus may @I ask, what does one have to do in order to get all the suares to match accordingly?

NVTeam
Posts: 2286
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Post by NVTeam » Mon Mar 21, 2016 7:09 pm

NV_Fan wrote:What tends to happen is sometimes the same squares in the software are matched more than once when calibrating from the square targets and they either have higher or lower values when selected (hope this makes sense?).
I am not quite sure what you mean here.

How do you use the shot of the target? Do you open it in Neat Video and just click Auto Profile? Or you do something more? If you do, what exactly?

Thank you,
Vlad
Image Image Neat Video team
noise reduction for video and photos

NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Post by NV_Fan » Mon Mar 21, 2016 8:05 pm

I of course open it in Neat Video and I do Auto Profiling on the center/large square, which I then go and do Manual Profiling on each square.

When I do this, it some or many times selects the same square in the 'noise profile equalizer' and readjusts it higher or lower.

In addition, many other squares don't get selected or detected at all. I am asking, since this is an ongoing problem and tried different configurations, what does one have to do to get all (if not, most) squares in the equilizer profiled???

NVTeam
Posts: 2286
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Post by NVTeam » Mon Mar 21, 2016 8:23 pm

NV_Fan wrote:I of course open it in Neat Video and I do Auto Profiling on the center/large square, which I then go and do Manual Profiling on each square.
I hope you remember that Auto Profile has just done that already: it initially measures the visible primary selection and then it automatically fine-tunes that primary analysis by measuring the noise level in all those numerous squares around the center. So doing manual profiling is for the most part a duplication of efforts in case of the Calibration Target.
NV_Fan wrote:When I do this, it some or many times selects the same square in the 'noise profile equalizer' and readjusts it higher or lower.
That is normal too. The noise levels slightly change from place to place, that is to be expected.
NV_Fan wrote:In addition, many other squares don't get selected or detected at all. I am asking, since this is an ongoing problem and tried different configurations, what does one have to do to get all (if not, most) squares in the equilizer profiled???
You need the image itself to contain the areas that:
(1) correspond to those unmeasured nodes of the graph;
(2) contain noise;
(3) contain no details.

When you use the Calibration Target and you shoot it as described in the instructions supplied with the target, the resulting test shot usually satisfy those conditions for the most part. Normally, such a shot offers the sufficient number of different areas to cover most of the graph nodes using Auto Profile alone. If that doesn't happen and it covers says 90%, that is just as fine too. There is no absolutely no need to cover them all, that is not really necessary. The software is designed to work in a situation of imperfect and incomplete information, so trying to make the measurements 100% complete just takes more time but brings few additional benefits.

Vlad
Image Image Neat Video team
noise reduction for video and photos

NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Post by NV_Fan » Tue Mar 22, 2016 5:14 am

I find this a bit strange then, because when I do Auto Profile and then Manual profiling it gives me different results from the automatic process. But nevertheless I've done both combinations.

I find your comments very informative and useful so I am much appreciative Vlad! However I am still liking to know, what does one have to do to cover all the nodes in the equilizer?

I say this also with a bit of concern since the black square gives me the least ranking in the nodes when it should be the highest since it represents shadow areas.

NVTeam
Posts: 2286
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Post by NVTeam » Tue Mar 22, 2016 11:58 am

The black patch in the table should be exposed in such a way as to contain near-black pixel values (0-15), the white patch should be exposed in such a way as to contain near-white pixel values (240-255). Then, fine-tuning using those areas from patches will affect the leftmost rightmost nodes of the equalizer graph.

The actual noise levels (vertical positions of the nodes) in those brightness ranges can be both lower and higher than in other ranges, that is not in itself abnormal. I would pay more attention to the overall shape of the curve in the equalizer. It should normally be smooth and not reaching extreme values (high or low).

Vlad
Image Image Neat Video team
noise reduction for video and photos

NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Post by NV_Fan » Sat Mar 26, 2016 3:19 am

I can completely assure you, whenever I'm shooting the calibration targets, the blacks are almost always back and the whites are white, this is attributed to the white balancing and exposure, of course I overexpose by 1-stop just so the blacks aren't crushed and so it would be picked up but that hasn't been the case always! :(

I understand what you're saying about the vertical position of the nodes, and I always pay attention to the curves and that is actually one of the main problems, the curve's shape is abnormal and almost always has an inverted result, and this is attributed to some of the nodes not being picked up when profiling the calibration chart :(

I can probably get the black and white squares exposed correctly, it's just a matter of getting all the other squares to give results individually to different nodes.

UPDATE: I was just doing more tests again with my test shots of the chart, I get best results everytime I Auto Profile on the black square itself. I compared the graph chart when I Auto Profile on the large middle square and with nothing selected and it always pick up more noise in the shadows/blacks better when profiling on the black square, is this the proper way of doing it? However other nodes towards the highlights are left untouched!

NVTeam
Posts: 2286
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Post by NVTeam » Sat Mar 26, 2016 11:44 am

It is usually best to select the area showing the strongest noise and used it for Auto Profile. The latest versions of Neat Video 4 show the noise level measured in the selection, so it should be easy to find the area with strongest noise when you do that manually. Auto Profile may not always find the best area, so if you want to help it then try different areas and use one with strongest noise.

Vlad
Image Image Neat Video team
noise reduction for video and photos

NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Post by NV_Fan » Sun Apr 24, 2016 2:20 am

Sorry for bringing this up again, but I am still having trouble in obtain very good profiles with a nice curve in the noise profile equalizer.

From your previous comment black patch in the table should be exposed in such a way as to contain near-black pixel values (0-15), the white patch should be exposed in such a way as to contain near-white pixel values (240-255). This is very hard to achieve especially when using flat/low-contrast picture profiles on a camera, the same goes as well for film stock where the image is very logarithmic i.e. very low contrast.

Because of this, achieving those pixel values is near impossible (they will almost always come out looking a bit grey).

Would you ever reconsider in creating a new calibration target specifically aiming to target all the nodes in the noise profile equalizer??

Here is an old example of my GH2 footage at ISO1600 at 24p.

Image

NVTeam
Posts: 2286
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Post by NVTeam » Sun Apr 24, 2016 12:22 pm

Do regular video clips that you need to denoise also show the same levels of black and white as the shot of the target?

Vlad
Image Image Neat Video team
noise reduction for video and photos

NV_Fan
Posts: 10
Joined: Wed Feb 26, 2014 4:53 pm

Post by NV_Fan » Sun Apr 24, 2016 5:32 pm

Theoretically yes, anything that is black and white at a properly exposed shot on ISO1600 should have the same levels.

Post Reply