1 Topic by shagbark (edited by shagbark 2022-01-12 17:52:20)

  • shagbark
  • Member
  • Offline
  • Registered: 2016-01-14
  • Posts: 11

Topic: Error-correction: GParted cloning vs. copying files

I need to copy a 10TB NTFS drive that's over 99% full.
I prefer to clone NTFS drives, because NTFS has a special hell of endless access-control-list bugs for those who copy it file-by-file.
But it seems it will take days to clone this drive, and copying by file may actually be faster.

My biggest concern, though, is bit errors.  The specs for the Western Digital Red drives indicate a non-recoverable error rate of <1 in 10^14.  This drive contains 10^14 bits.  So getting one or two bit errors is not unlikely.  I don't want that to happen.

I find via web search that NTFS has no bit-level error correction.  Does GParted?
(If it doesn't, it really should, now that hard drives are too big for WD's hardware checksums to cope with them.)

If GParted also has no bit-level checksums or verification, how do GParted and NTFS file-copying compare at detecting errors at other levels, such as filesystem errors?

2 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Error-correction: GParted cloning vs. copying files

Some rhetorical questions:
1. Why aren't you using ZFS which checksums all data so that it can confirm all data is read and written without bit errors?
2. Why aren't you using ZFS mirrored volumes so that ZFS can automatically provide the correct data in the face of one drive returning bit-corrupted data?
3. (With ZFS ECC RAM is strongly recommended, and seems pointless without it).  If you didn't have ECC RAM how do you know your data wasn't affected by bit errors before being written to the drive?

None of this is advice to use ZFS, just that it is reportedly the safest way to store your data if you care about it that much.

As NTFS doesn't have checksum data, then unless you have done that yourself, there is no way to be certain that there aren't already bit errors in your data.  This is before you even think about using GParted (or not) to copy it.

GParted can't add checksums to file systems that don't have them, as that would make them incompatible and be unreadable by anything else.
GParted can't add checksums to the host to drive communication.

If a bit flips on a drive platter and the drive says error instead of return the data from a particular sector then there is nothing GParted can do other than report that error and stop when copying the data.  (Hard disk drives do checksum every sector of data however those checksums are *NEVER* available to the computer or user.  The drive either returns the data, because it matches the checksum, or it returns a read error and you'll never get that sector's data again).  In such cases specialist data recovery tools are needed.  Ddrescue is a tool that can try to read as much data as possible from a drive which has failed sectors.  It will try to re-read the failed sectors again, though the drive will have already tried for 8 to 120 seconds and each subsequent attempt to read a failed sector will incur the same lengthy retry cycle.  Ddrescue is available on the GParted Live CD/USB along with other rescue systems and many distributions.

As a practical but technical and lengthy suggestion try this:
1. Prepare the destination hard drive with the same size partition.
2. Use ddrescue to copy your NTFS from the source to new destination partition.
3. Run md5sum on the source and destination partitions.
If the md5sums match then the copy didn't introduce any bit errors.  Each md5sum run will take approximately as long as the data copy.  3 time longer in total.

3 Reply by shagbark (edited by shagbark 2022-01-13 22:58:15)

  • shagbark
  • Member
  • Offline
  • Registered: 2016-01-14
  • Posts: 11

Re: Error-correction: GParted cloning vs. copying files

mfleetwo wrote:

Some rhetorical questions:
1. Why aren't you using ZFS which checksums all data so that it can confirm all data is read and written without bit errors?
2. Why aren't you using ZFS mirrored volumes so that ZFS can automatically provide the correct data in the face of one drive returning bit-corrupted data?
3. (With ZFS ECC RAM is strongly recommended, and seems pointless without it).  If you didn't have ECC RAM how do you know your data wasn't affected by bit errors before being written to the drive?

1. I'm not using ZFS because I'd never heard of ZFS.
2. I won't use ZFS because it doesn't work with Windows, and I must use Windows.
3. I couldn't find an acceptable and affordable motherboard that used ECC RAM.

I wasn't asking for GParted to add checksums to filesystems, nor access the hard drive's checksums.  I wanted to know if it did a checksum on the data it read in, and then, after writing that data somewhere else, read it back and compared checksums.

I can compute the checksums myself, as you suggest; but I was hoping GParted computed that checksum itself, and would let me save it somewhere, perhaps to a flash drive.  That would be a nice feature that would save me time.

4 Reply by mfleetwo (edited by mfleetwo 2022-03-02 11:21:46)

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Error-correction: GParted cloning vs. copying files

Note that for EXT2/3/4, NTFS and XFS file systems GParted uses file system specific commands to copy just the used data.  So GParted isn't even reading the data itself, plus a partition level checksum won't match because the not copied unused blocks in the file system between the source and destination partitions will be different.