View Issue Details

IDProjectCategoryView StatusLast Update
0000266TeraCopyEnginepublic2017-10-27 05:45
ReporterMo-Gul 
PrioritynormalSeverityfeatureReproducibilityN/A
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0000266: check checksums immediately after copy/move
Descriptioncurrently copying/moving and checking checksums are done serially, i.e. one after another. But wouldn't it be possible to check the checksum of a file directly after copying/moving is finished (on another processor?) and the next file is copied/moved in parallel to the test of the checksum?
TagsNo tags attached.
OS
OS Version

Relationships

Activities

CodeSector

2017-10-07 23:34

administrator   ~0000183

In most cases (except RAID), you have only one I/O channel, so you'll get half write speed + half read speed = same speed in result.

DonkeyDon

2017-10-27 05:45

reporter   ~0000190

That "one I/O channel only" argument applies for writing data to the same spindle that you are reading from, so I'd like to second the request for a similar option.

I primarily use teracopy between to copy large amounts of data(300GB to 15TB) between slower devices (different drives/devices/networks). If I watch Resource Explorer during a verify, I often see teracopy scream through the local SSD read(200MB/s high disk activity), after which the disk activity drops to zero, and then network activity jumps from zero to 50MB/s, and then the whole cycle repeats for another file. One bus is always idle, yet I have 4 cores that could be doing checksums on different files. Even if I'm writing network to network, the GBe bandwidth is more than enough to handle 2-3 queued streams. In addition, the latency drag that comes in when you copy/ verify 100,000 small files could be greatly reduced. But even with big files the network is the bottleneck, and we are wasting bandwidth if it sits idle for the read of the local drive. On a multi-terabyte copy/verify , this wasted time makes the difference between having an overnight copy job "done" in the morning, and having to wait until the next day.

Issue History

Date Modified Username Field Change
2017-09-29 06:46 Mo-Gul New Issue
2017-10-07 23:34 CodeSector Note Added: 0000183
2017-10-27 05:45 DonkeyDon Note Added: 0000190