1 (edited by bradpitt 2014-01-08 07:45:42)

Topic: GParted crashed while moving NTFS partition to the left

Hi.  So I had 3 partitions:

+-----------+-----------+-------------------+-----------+
| /dev/sda1 | /dev/sda2 | unallocated space | /dev/sda3 |
|   NTFS    |    EXT4   | 35632170 sectors  |   NTFS    |
|           |           |     16.99 GiB     |           |
+-----------+-----------+-------------------+-----------+

And decided to move /dev/sda3 partition to the left (into whole unallocated space).  Unfortunately, while doing this operation at some point linux live-cd system (SystemRescueCd 3.8.1) froze (Scroll Lock and Caps Lock lights began blinking).  I wasn't at the computer at that moment, so I don't know how much data have been already moved.  Upon later reboot to the linux live-cd, I examined partition table, and the /dev/sda3 partition was set to:

- start: first sector of previously unallocated space (seems reasonable)
- size: old size of /dev/sda3 + 35632170 sectors (seems reasonable)

So what I'm thinking right now is: I need to find out how many sectors were copied (so I can move them back and restore filesystem as it was bit-by-bit).  If I assume that GParted started copying from the start of /dev/sda3 to the start of unallocated space, so there somewhere on the /dev/sda3 should be 2 identical chunks of data (of the size 35632170 sectors) in sequence, i.e. 2 identical chunks one after another.  Am I right?

                    /dev/sda3
+-----+------------------+------------------+-----+
| ... | identical blocks | identical blocks | ... |
| ... | 35632170 sectors | 35632170 sectors | ... |
+-----+------------------+------------------+-----+

If my logic is correct, I wrote a little python script to help me out to find those identical chunks:

SECTOR_SIZE = 512 # bytes
SHIFT = 35632170 * SECTOR_SIZE # bytes
OLD_PARTITION_SIZE = 440421975 # sectors

equal_sectors = 0
disk = open("/dev/sda3", "rb")
for sector_num in xrange(OLD_PARTITION_SIZE):
    disk.seek(sector_num * SECTOR_SIZE)
    read_wo_shift = disk.read(SECTOR_SIZE)
    disk.seek(sector_num * SECTOR_SIZE + SHIFT)
    read_with_shift = disk.read(SECTOR_SIZE)
    if read_wo_shift == read_with_shift:
        if equal_sectors == 0:
            print("First match found at sector: " + str(sector_num))
        equal_sectors += 1
    else:
        if equal_sectors > 0:
            print("Total identical sectors in sequence found: " + str(equal_sectors))
            equal_sectors = 0
if equal_sectors_num > 0:
    print("Total identical sectors in sequence found: " + str(equal_sectors_num))
disk.close()

This script is looking for identical sectors spaced apart by 35632170 sectors and the number of identical sectors in sequence should be also 35632170 sectors.  Well, at least I'm hoping that it does that.  While I'm waiting for the script to finish the work, any comments, ideas?

Thank you

2 (edited by bradpitt 2014-01-08 06:35:04)

Re: GParted crashed while moving NTFS partition to the left

Hi once again.  So my hypothesis proved to be correct, I eventually found almost identical two large continuous chunks of data (of size 35632170 sectors) following each other on /dev/sda3.  Like this:

                    /dev/sda3
+-----+------------------+------------------+-----+
| ... | identical blocks | identical blocks | ... |
| ... | 35632170 sectors | 35632170 sectors | ... |
+-----+------------------+------------------+-----+


Almost identical.  Here's the breakdown of the comparison between those 2 large chunks of data:

Sequence of identical bytes: 166528704  (158.81 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 147860415  (141.01 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 143779775  (137.12 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 39235775   (37.42 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 561131295  (535.14 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 773283007  (737.46 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 387368095  (369.42 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 14527295   (13.85 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 2312125567 (2205.01 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 431078847  (411.11 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 5722775103 (5457.66 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 231653439  (220.92 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 255        (0.00 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 967009407  (922.21 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 548029119  (522.64 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 498781183  (475.67 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 103722367  (98.92 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 95120191   (90.71 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 92161215   (87.89 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 2981113151 (2843.01 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 1385919551 (1321.72 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 517902591  (493.91 MiB)
Sequence of differing bytes: 1
Sequence of identical bytes: 122564671  (116.89 MiB)
---------------------------------------
                Total bytes: 18243671040 (35632170 sectors) (correct!)
      Total different bytes: 22

Can someone explain what is happening here?  Why do I have different bytes interspersed between correct continuous data?  I checked the drive for read errors (badblocks read only test) and it does not reported any read errors.  I thought that GParted, when moves the partition, copies data at block level (not filesystem level) sector by sector, so there should not be any different bytes.  Do GParted does something else under the hood?

3

Re: GParted crashed while moving NTFS partition to the left

bradpitt wrote:

Can someone explain what is happening here?  Why do I have different bytes interspersed between correct continuous data?  I checked the drive for read errors (badblocks read only test) and it does not reported any read errors.  I thought that GParted, when moves the partition, copies data at block level (not filesystem level) sector by sector, so there should not be any different bytes.  Do GParted does something else under the hood?

As you discovered, your theory about how GParted moves data is correct.  :-)

When moving a file system to the left, GParted copies sector by sector starting from the front of the partition and ending at the tail of the partition.  This order is reversed when moving a file system to the right. 

If GParted crashed or hung in the middle of the move operation, then I would expect all of the sectors to be identical up until the location where the crash occurred.

If; however, GParted finished the move then the partition size would have been changed to the new partition location.  Since this is not the case we must be dealing with the former situation.

If the file system was mounted then that could affect some of the sectors in the file system since things like last access time are updated, even if a file is only viewed and not written to.

Other than that the copies should be identical....

If your move operation did not include overlapping regions, you might consider restoring the original partition boundaries, and then trying the move again.

Please note that I highly recommend having a backup of your data whenever editing partitions in case something goes wrong such as a software bug, hardware failure, or power outage.

4 (edited by bradpitt 2014-01-08 15:11:36)

Re: GParted crashed while moving NTFS partition to the left

gedakc, thank you for your reply.

Yeah, that's a mystery why there are differing bytes.  If I would had space for backups I would have moved whole partition with dd (dd to some other partition, then back to the new place) :)

As an aside, can I move partition to the left manually with dd without involving some other spare filesystem/partition?  Let's say:

dd if=/dev/sda3 bs=1M skip=XXXX of=/dev/sda3 seek=0

5

Re: GParted crashed while moving NTFS partition to the left

bradpitt wrote:

As an aside, can I move partition to the left manually with dd without involving some other spare filesystem/partition?

Yes tools such as dd can certainly be used to move the partition, though if there are any overlapping regions you must be extra careful to ensure that the tool will copy in the correct order (e.g. start to end, or end to start).

Since NTFS contains the partition offset from the start of the drive in the NTFS Partition Boot Record, you will need to either use ntfsfix to adjust this value, or manually edit the value with a hex editor.

Also you will still need to update the partition table after the move is completed.

6

Re: GParted crashed while moving NTFS partition to the left

bradpitt wrote:

[…]
If my logic is correct, I wrote a little python script to help me out to find those identical chunks:
[…]

Wow, didn't know Brad Pitt was also a coder! 

7 (edited by Dan Harkless 2022-04-09 23:55:38)

Re: GParted crashed while moving NTFS partition to the left

[Argh, this forum software silently truncates posts if you put in an emoji smiley instead of the forum-markup ones.  Here's my full post as intended.]

bradpitt wrote:

[…]
If my logic is correct, I wrote a little python script to help me out to find those identical chunks:
[…]

Wow, didn't know Brad Pitt was also a coder!  tongue  Seriously, though, I wanted to thank you very much for your script.  I had an out-of-memory crash of my Cygwin X server during the move part of a shrink-and-move-right on an NTFS partition on one of my Linux servers, which of course killed Gparted.  Lesson learned on that — always run Gparted operations on the actual console, even when it's more painful to keep track of the progress that way.

Your script beautifully found where the move left off, and then, since I didn't want to mess with manually fixing the partition offset — thanks for the info on that, gedakc! — I decided to move the relocated section back to the left using dd_rescue, then reboot into Windows for a chkdsk (the version of fsck.ntfs I have seems only partially implemented). 

I hadn't manually resized the partition to match the ntfsresize-shrunk volume, and chkdsk actually hung indefinitely at the point where the partition should have ended, so I had to reboot back into Linux, check for the proper cutoff point with ntfsresize -f --info, fix it to be that with fdisk, then reboot back into Windows for another chkdsk, which didn't hang this time, and found no errors (yay!).  Then it was back to Linux for a Gparted move of the partition to the right as originally intended.

Anyhow, thanks again, bradpitt — I could have implemented the equivalent of your script in Perl or C, but you saved me a lot of thinking and coding time.

Gparted should really write a log to disk of its progress (deleted on a manual Quit, unless gparted is run with a --keep-log option or something), to make it easier to manually recover from a crashed operation.  Down the road, that could also enable someone to implement automated resumption of interrupted operations.  That said, thanks for Gparted in its current state too; it's been quite helpful to me.  smile