VR- Noise reduction

suggest a way to improve Neat Video
Post Reply
Jan Raiber
Posts: 4
Joined: Fri Mar 02, 2018 11:18 pm

VR- Noise reduction

Post by Jan Raiber »

Hi i would be very happy, if neat video support equirectengular video in future.
Cause of the stretched projection, the noise of this shots have different sizes and can not be sampled right. To denoise them i have to do a lot of work: denoise all shots before stitching, or reproject the final edit to cubemaps, where the noise have a little more even size and denoise it than. but all this workflows introduce a huge amount of work, rendertime and data to accomplish. I know a lot of studios which are in the same situation. Here Neat Video can fill a huge gap.
Thank you, and all the best,
Jan
NVTeam
Posts: 2745
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Re: VR- Noise reduction

Post by NVTeam »

Hi Jan,

Pre-denoising the original rectangular shots before stitching seems to be the most natural solution.
Anything done at later stages in a workflow would likely have to deal with modified noise, which in
many cases can make accurate noise reduction more difficult.

I recommend to look into direction of making that initial pre-denoising more automated (many video editing
applications can in some way help with that) so that you would simply batch the originals through such
a standalone tool. For example, think of a combination of VirtualDub, NV for VirtualDub, and a script for VirtualDub
that takes any input video clip, applies NV with specified noise profile and filter preset, and then saves the output
to a new clean video file. You could then use this script to process the set of clips any time you need to pre-denoise
the footage before stitching it in your further workflow.

Thank you,
Vlad
Atmosphaeres
Posts: 3
Joined: Sat Nov 21, 2020 2:29 pm

Re: VR- Noise reduction

Post by Atmosphaeres »

Hi Vlad,

the two main problems with de-noising VR footage in the original rectangular format is two-fold. On the one hand side, we as creators would need to create a near-lossless export version of each and every lens/sensor, which would result in a HUGE storage increase. Let's say that the original file is a 4:26 minutes H264 MP4 with 3.9GB file size. In our case, with an Insta360 Pro 2 we record 6 of those. So that comes to about 23GB for all files. If we have to denoise each one of these files into say ProRes, the file size is about 10x that. So for a single shot, we would get about 200-250 GB of a near-lossless file. On our last shoot, we filmed 50 shots, so that would amount to about 10TB of de-noised original footage. Also, these 6 de-noised ProRes files (all 3840x2880) would need to be loaded into the stitching software (e.g. Mistika VR) at the same time. Even a Synology RAID with 10GBe connection is going to struggle with that amount of data.
On the other hand, we never know how bad the noise is in each shot because the conditions vary and we change settings all the time. In a daylight shot with 'standard' recording settings, the noise will be much less than in 'ilog' mode during sunset. So a batch encoding process that uses the same preset is simply not possible. Instead, we have to optimise the de-noise setting for each shot. Also, we may only need 1 minute of those 4:26 minutes or only 30 seconds are usable. We can only see that in the final stitched file. So there's a massive overhead that eats up storage space and time for de-noising footage that is not going to be used in the end.

Having a, algorithm for an equirectangular format would help big time as the industry is growing and users get more aware of issues like noise.

Thank you for your excellent work,

Eric
NVTeam
Posts: 2745
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Re: VR- Noise reduction

Post by NVTeam »

A big problem with VR shots is the size of the frame that is several times larger than the size of each individual shot produced by an individual camera. That large total size creates a bottleneck during processing in the host application and in the memory of the computer as a whole, leading to considerably slower processing speed. So what you win on not having to use a large temporary storage you then lose on that slower processing of large VR shots. Perhaps it is no big problem if you only need to take a small part of the long shot, but then there are also other issues related to stretched nature of the image and noise that is modified unevenly by the VR projection. That is a big problem for noise reduction algorithms. I seriously doubt there is a good solution for it other than preventing it by denoising before doing the projection.

Perhaps you can shorten the process and limit the required disk space by using the following workflow:
1) take the VR video or individual shots into your editing project and select the required segments of the video that you are going to actually use;
2) export those selected parts to intermediate files;
3) denoise those intermediate files in a separate project;
4) take the denoised intermediates into your editing project and replace the originals with new shorter and cleaner versions.

I am not sure if the project and editing tools allow you do that easily but that is what I would try to implement if I had to organize the workflow in a more efficient way than either (1) just apply NR to the whole large VR video or (2) apply NR to the whole individual shot for every camera and then join those into a VR video and use the required sections.

Hope this helps,
Vlad
Atmosphaeres
Posts: 3
Joined: Sat Nov 21, 2020 2:29 pm

Re: VR- Noise reduction

Post by Atmosphaeres »

Our current workflow is the following:

- Have 6 original, individual shots on 10GBe RAID
- Stitch into one, full-length euqirectangular video with Mistika VR. Export format = ProRes or Cineform
- Clean equirect video in After Effects, including Noise Reduction. Export format for intermediate file = Cineform. Export time per intermediate video, about 5 hours.
- Colour, sound and shortening in Adobe Premiere.

This is the only practical workflow for us because the file size increase and time investments of adjusting the preset and denoising the 6 original, individual shots is not feasible at all.

We know plenty of fellow 360 videographers who use this workflow and for who de-noising individual sources before stitching is not an option and there will likely be coming more and more who will follow in our footsteps as 360 video/VR is gaining momentum. So it would be great to have the option of de-noising strechted, distorted equirect videos. Or maybe there's an option to create a plugin for Mistika that denoises the 6 individual shots during export?

Thanks for your always helpful answers, Vlad.

Cheers,

Eric
NVTeam
Posts: 2745
Joined: Thu Sep 01, 2005 4:12 pm
Contact:

Re: VR- Noise reduction

Post by NVTeam »

I have double-checked with Mistika specialists and they say that it should be possible to do the following:

1) Import those 6 shots into Mistika Boutique;
2) Apply individual instance of Neat Video to each of those 6 shots;
3) Send the output of those 6 instances of Neat Video to VR Stitch node to join them into one euqirectangular shot;
4) Export the video from Mistika Boutique to an intermediate file;
5) Take that file for further editing to Premiere or any other application.

Hope this helps,
Vlad
Atmosphaeres
Posts: 3
Joined: Sat Nov 21, 2020 2:29 pm

Re: VR- Noise reduction

Post by Atmosphaeres »

Thanks a lot, Vlad, that is actually a feasible option. Mistika Boutique & VR are subscription services that we only subscribe to every few months when we have a batch of videos that we're working on, so it's not something that I'd pay €100+ for just to try this out. But I will try it as soon as we have another batch of videos to de-noise and stitch. Thanks a lot for your help :)

Eric
Post Reply