limit on number of files in batch list?

[FransK]

I am currently processing 1 hour of video, converted with VirtualDub to sequencially numbered image files (about 100,000 frames/files) with home edition of Neat Image (v4.25).
In a .bat file with a "for %%F in I:\unfiltered\*.bmp do NeatImage %%F...." command, NeatImage is called to process each image file.

This takes days of processing on my 1.7 GHz P4 Win-Xp system (and 200 GB disk space if I did it in one batch).
In NeatImage batch mode, it would be 15-20% faster.

Can the Pro edition handle this amount of files (100,000) in its batch? Is the batch window stil reasonably workable then?
(reading directories containing these numbers of files does take 10-s of seconds in de Windows Explorer)

[taob]

I don't know if this is documented anywhere, but I just tried it with 4.25 Pro+, and the batch queue limit appears to be 3000 images. You'll have to break up your 100000-frame job into about 34 batches, but at least that's still better than doing them one at a time.

[FransK]

Thanks for the info.

I have filtered half the movie now, after putting my PC to work for several nights.
I'll stay with to the command shell batch for now. Moving a bunch files to the input subdirectory and starting the command procedure are the only things necessary to create and start a job.
I need to shuffle around the frame image files in sets of 2 to 5.000 anyhow, because working with 50.000 files in one direcory under Windows XP (NTFS) is rather timeconsuming.
And continuing a "job" after a power failure is not difficult either.

[moldmichael]

If you could say a few words on why you embarked on this task I would be interested.

Was the result worth the effort?

Mike

[FransK]

Was the result worth the effort?
Mike

Probably not, as this was a home video for family-use only.
I wonder if anyone else except me and my wife (who noticed the PC was running for days and nights, appearently doing nothing) will notice it. (I will put a piece of the original video in, to make them notice the difference).

But is was fun to find out how I could apply the superior noise reduction that I use with digital photography, in my other hobby: video editing.

If you could say a few words on why you embarked on this task I would be interested.

I did complete the filter process for the whole film sequence. Over a 2 weeks period it took about 20 job (re-)starts and about 1 to 1.5 week of CPU time.

The input video was quite noisy because it was shot in low-light conditions. NeatImage produces filtered images that are far out superior to any other video filters I tried, ever though I coult calibrate NI only on existing images (when I shot the video, I did not know yet about NI and its calibration chart).

I found it easier to use the DOS batch method (invoke NeatImage from the command line, one image at a time) rather then the built-in NeatImage batch.

Apart from the time it took, there were two problems with the (DOS batch) procedure:

A) The PC could not be used for anything else.
The problem was that NeatImage is started about every 5 seconds and NI's program window took focus. If you were doing something in another program, (explorer, typing text in an editor), you suddenly were typing text into the NI window!
I tried to solve this by using the START command available in Windows XP batch language, there you can specify that the program is to run minimized (no window visible, so it does not take focus) and set a lower priority. But I couldn't get NeatImage to recognise the (3) file names on the command line in this way.

Advantage of the DOS batch is, that there is no limit on the number of files that are to be processed (apart from available disk space). I could let it run for a complete weekend without needing to touch the PC.

B) It is easy to loose images (with these large numbers of separate images), which results in audio/video to be out of sync when the image sequence is recombined with the original audio track.
And is very timeconsuming and almost impossible to correct it later on. For this video, the sound is very important.

I have tried to create a plug-in filter for the video editor I use (VirtualDub) but I didnot have the necessary C++ compiler.


My conclusions:

- Above all, I would like to use NeatImage directly from inside my video editing program as a kind of plug-in filter.

This way there is no chance that audio and video get out-of-sync.
Also a lot of intermediate steps kan be skipped.
And there are no huge quantities of free disk space needed to store the individual (uncompressed) video frames twice (the unfiltered and the filtered frame).

- It would be even easier to program, less suspect to runtime problems (users hitting a wrong key that influences of even kills NI) and a little faster if NeatImage could be invoked as a subroutine (a Windows DLL or ActiveX control) rather then started a separate process.

- I prefer the DOS batch method above the NI built-in queue because of the limited number of files in the latter.

- Next time I'll try to create a frame processing filter in my video editor (VirtualDub) that runs NI.

[moldmichael]

Well it's been a while but thank you for that. Its the sort of thing I would try but I need a more potent machine than this laptop!

Mike Lewis

[FransK]

My pleasure.
It's kind of fun to publish the results of your own experiment.

[DaveQ]

After several weeks of intense coding, I am finally ready to present v.1 to the world!

The reason for all this work, was the need to process very large numbers of video frames with Neat Image (119,000 +). I soon realized that I could add features to the barebones code and make the tools useful to others, who had also expressed an interest in this usage.

http://www.isochroma.com/Testfiles/NB/neatbatch.htm

Partial list of features:

> Open Source
> AVI or Image File input
> Very large batches (up to 2,147,483,646 images)
> Priority setting
> Input/Output sequence name & number remapping
> Input & Output image format conversion
> Automatic AVS generation for video output
> DNP, NFP, all switches supported
> Automated cleanup of input, output files

All done on-the-fly... with only 1 frame overhead.