WinXP Pro+ 32bit batch mode disk efficiency

resolve technical issues related to use of Neat Image
Post Reply
Nadrek
Posts: 2
Joined: Thu Apr 16, 2009 2:58 am

WinXP Pro+ 32bit batch mode disk efficiency

Post by Nadrek »

While running a batch of 11GB worth of 16-bit TIFs (about 230 of them, some over 80MB in size; Canon 50D RAW images converted by Bridge CS4), I noticed the following:

After adding the directory to the batch, and before it actually started doing the processing (i.e. the filtration queue didn't have the names yet), Sysinternals Process Monitor showed that NeatImage appeared to be executing sequential reads of every single file in chunks on the order of 1/3900th of the file's size (12KB to 19KB).

During filtration, again there are sequential reads in the tiny little chunks, followed by sequential writes in the tiny little chunks.

So, my questions are:
A) Can I please avoid the first pass of reading everything? It's increasing the batch cycle time quite significantly, and I can think of no real benefit to my "start batch, wait for the batch to finish" workflow.

B) Could you please update the code to execute reads and writes in signficantly larger chunks (64KB, 256KB, 4MB)? My Microsoft sqlio tests show that even at the end of a single standard hard drive (WD 1TB FALS) without partition alignment used for the OS (while other software was wrunning), 8KB sequential writes went at 53MB/s, while 32KB and up writes went at 72MB/s. Reads showed the same pattern. On an otherwise empty, fresh, partition aligned 7200.12, sequential writes at 8KB went at 82MB/s, while at 64KB they went at 122MB/s. Reads again showed the same results.
B1) Also importantly, for those using SSDs, larger, sector multiple write sizes in particular can be important for performance and drive lifespan both, to write to as few sectors as possible.


Thank you for your time!


Before the actual filtration:
2:53:34.9434756 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4653.tif SUCCESS Offset: 20,643,732, Length: 17,220
2:53:34.9439354 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4653.tif SUCCESS Offset: 20,660,952, Length: 17,220
2:53:34.9443631 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4653.tif SUCCESS Offset: 20,678,172, Length: 17,220
2:53:34.9447875 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4653.tif SUCCESS Offset: 20,695,392, Length: 17,220
2:53:34.9452579 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4653.tif SUCCESS Offset: 20,712,612, Length: 17,220
2:53:34.9456789 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4653.tif SUCCESS Offset: 20,729,832, Length: 17,220
2:53:34.9461097 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4653.tif SUCCESS Offset: 20,747,052, Length: 17,220



Filtration reads (at the end of a file):
3:03:38.5846393 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4578.tif SUCCESS Offset: 70,097,430, Length: 19,008
3:03:38.5851242 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4578.tif SUCCESS Offset: 70,116,438, Length: 19,008
3:03:38.5855989 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4578.tif SUCCESS Offset: 70,135,446, Length: 19,008
3:03:38.5863420 AM NeatImage.exe 3712 ReadFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4578.tif SUCCESS Offset: 70,154,454, Length: 19,008
3:03:38.5868306 AM NeatImage.exe 3712 CloseFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\2009BringKidsToWork_MG_4578.tif SUCCESS


Filtration writes:
3:03:46.5417858 AM NeatImage.exe 3712 CreateFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif SUCCESS Desired Access: Generic Read/Write, Disposition: OverwriteIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, AllocationSize: 0, OpenResult: Created
3:03:46.5423152 AM NeatImage.exe 3712 WriteFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif SUCCESS Offset: 0, Length: 8
3:03:46.5428258 AM NeatImage.exe 3712 QueryStandardInformationFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif SUCCESS AllocationSize: 8, EndOfFile: 8, NumberOfLinks: 1, DeletePending: False, Directory: False
3:03:46.5429543 AM NeatImage.exe 3712 WriteFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif FAST IO DISALLOWED Offset: 8, Length: 19,008
3:03:46.5430988 AM NeatImage.exe 3712 WriteFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif SUCCESS Offset: 8, Length: 19,008
3:03:46.5433667 AM NeatImage.exe 3712 WriteFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif FAST IO DISALLOWED Offset: 19,016, Length: 19,008
3:03:46.5434865 AM NeatImage.exe 3712 WriteFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif SUCCESS Offset: 19,016, Length: 19,008
3:03:46.5437732 AM NeatImage.exe 3712 WriteFile C:\Backup\7200.12-FullDDrive\0PhotosWORKINGBackupOnNetwork\50D\20090409DeltaBringKidsToWorkDay\TIFFsForWeb\Flt\2009BringKidsToWork_MG_4578_flt.tif SUCCESS Offset: 38,024, Length: 19,008
NITeam
Posts: 3173
Joined: Sat Feb 01, 2003 4:43 pm
Contact:

Post by NITeam »

Thank you for your suggestions. I will add them to the wish list.

Kind regards,
Vlad
Nadrek
Posts: 2
Joined: Thu Apr 16, 2009 2:58 am

Post by Nadrek »

You're welcome; it's amazing what looking at detailed information can show (Bridge likes to do 3KB sequential reads at times, for example).

I continue to use Neat Image both because it works, and because the NI Team is willing to listen to their users.
Post Reply