View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000266||TeraCopy||Engine||public||2017-09-29 06:46||2017-10-27 05:45|
|Target Version||Fixed in Version|
|Summary||0000266: check checksums immediately after copy/move|
|Description||currently copying/moving and checking checksums are done serially, i.e. one after another. But wouldn't it be possible to check the checksum of a file directly after copying/moving is finished (on another processor?) and the next file is copied/moved in parallel to the test of the checksum?|
|Tags||No tags attached.|
||In most cases (except RAID), you have only one I/O channel, so you'll get half write speed + half read speed = same speed in result.|
That "one I/O channel only" argument applies for writing data to the same spindle that you are reading from, so I'd like to second the request for a similar option.
I primarily use teracopy between to copy large amounts of data(300GB to 15TB) between slower devices (different drives/devices/networks). If I watch Resource Explorer during a verify, I often see teracopy scream through the local SSD read(200MB/s high disk activity), after which the disk activity drops to zero, and then network activity jumps from zero to 50MB/s, and then the whole cycle repeats for another file. One bus is always idle, yet I have 4 cores that could be doing checksums on different files. Even if I'm writing network to network, the GBe bandwidth is more than enough to handle 2-3 queued streams. In addition, the latency drag that comes in when you copy/ verify 100,000 small files could be greatly reduced. But even with big files the network is the bottleneck, and we are wasting bandwidth if it sits idle for the read of the local drive. On a multi-terabyte copy/verify , this wasted time makes the difference between having an overnight copy job "done" in the morning, and having to wait until the next day.