1

Topic: How to shrink a partition without data loss...

Is there any way to know how much you can shrink a given partition, without deleting any data in the process?

I backup anyway, but at the moment, shrinking the size of a partition is guesswork...

Either with gparted, or the command line, is there a way to know how much contiguous unused space resides at the end of a partition please?

Thank you.

2

Re: How to shrink a partition without data loss...

Hi,

gr4nt wrote:

Is there any way to know how much you can shrink a given partition, without deleting any data in the process?

In theory you can shrink any partition so far that the data stored there (together with the required file system metadata) will consume all the available space - and no space will be left.
In practice, this is typically not useful - and many resizing tools are not able to handle the task of leavong no free space on a device, but shrinking any partition so far that onyl 10% free space remain should be no problem.

Either with gparted, or the command line, is there a way to know how much contiguous unused space resides at the end of a partition please?

For FAT32 or NTFS partitions, most Windows defragmenting tools should be able to tell you this. However, it is not necessary to have empty space at the end of your partition, since GParted will relocate your data from the part of the partition that is to be "cut away" to the part that will remain.
However, if you have your partition (mainly NTFS) cleanly defragmented before using GParted, you'll reduce the risk of anything going wrong since GParted then will have to perform much less relocations and metadata updates.

3

Re: How to shrink a partition without data loss...

Thanks for you help... I will shrink partitions with confidence smile

4

Re: How to shrink a partition without data loss...

Hello All,

Have been searching Ubuntu and many other forums for a reply to a related question briefly touched on by Stormhead's reply:  metadata and other hidden item relocations by Gparted.    Please give me link to another reply if this question is answered.

Does Gparted:
1) do checksums on partitions that are being moved or copied?
2) most defrag programs do not touch metadata, boot records, and the like that are written (and invisible) in apparently "free" space.  During a partition size reduction does Gparted relocate these items that could be in the way (i.e. in the middle of the free space)?

Here's an attempt of including an image of what Perfect Disk defrag program had to show in the available space:

screen shot 5

5

Re: How to shrink a partition without data loss...

Hi!

Gingerale wrote:

1) do checksums on partitions that are being moved or copied?

No.

2) most defrag programs do not touch metadata,

Metadata contains - among many other things - a list of sectors (counted from the beginning of the partition) that make up any given file, along with the order in which these sectors must be read. As a result, when moving data from one sector to another one (which is the basic building block of any defragmenting), metadata updates must take place - or you'll end up with 100% data loss. As a result, every defrag tool touches and changes your file system metadata!

boot records,

That's correct - boot records reside outside the file system and are thus not touched by the defragging tools, since they are only interested in the space occupied by the file system.

and the like that are written (and invisible) in apparently "free" space.  During a partition size reduction does Gparted relocate these items that could be in the way (i.e. in the middle of the free space)?

Anything that is recorded as present within the metadata is relocated, and the according metadata entries are updated. Anything not recorded will be left alone, overwritten, or simply not relocated before chopping off that sector from the partition - but this does nothing bad, since the metadata record are the only place where the contents of the file system are stored. Anything not listed in the metadata records is thus unknown to the file system - and will eventually be overwritten while the file system is allocating new blocks for a file, since nobody knows that stuff is there.

Here's an attempt of including an image of what Perfect Disk defrag program had to show in the available space:

Anything listed in this screen shot as "occupied" (that is, non-white block) is recorded in the file system metadata area and will thus be relocated during resizing. However, these blocks might me marked as e.g. "excluded" by your tool since they are
- either defective (the file system metadata records them as unusable then, and GParted can take care not to use them while resizing. I don't know if it actually does this - maybe one of the developers can help you here!);
- or occupied by files that Windows currently holds open (paging file, registry, ...). This won't be a problem either, since there will be no running Windows while GParted is being used.

I noticed that this screenshot also lists some "metadata" blocks, but also holds several "MFT" blocks. This means that this tool makes a difference between the MFT on a NTFS partition (this data structure records which file is stored where, which access rights apply to any given file and so on) and "other" metadata.
In contrast to this, I called anything that is not your user data or a directory as "metadata" - as a result of this, I'd simply include the MFT in the "metadata" area.
Unfortunately, this term is not so clearly defined...

6

Re: How to shrink a partition without data loss...

Hi Stormhead!

Thx for the tremendous answer.  It's your last paragraph that addresses the issue of concern causing me to gobble antacids for worry.  I am not concerned about Gparted properly handling the data or boot records.  I'm worried about the MFT blocks, and blocks labeled "metadata" itself.  If Gparted works as you say, which makes sense, by moving data, and updating the metadata, how does it handle moving the metadata itself and where does it update the information about the location of the metadata?  (i.e. metadata for the metadata).

I wrote to one of the team, François Dupoux I believe (can't find my e-mail copy) but got no reply.  Who would you write to get this information?

Thx

7

Re: How to shrink a partition without data loss...

Hi!

Gingerale wrote:

If Gparted works as you say, which makes sense, by moving data, and updating the metadata, how does it handle moving the metadata itself and where does it update the information about the location of the metadata?  (i.e. metadata for the metadata).

The file system driver (no matter if you're running Windows, Linux or whatever) expects the metadata at a given position, counted in sectors, starting from the beginning of the partition. To switch to your working, the "metadata of the metadata" is stored inside the MBR partition table (and all your EBR tables, if you have an extended partition); this information in fact is updated by GParted (without updating this information, any partition operation apart from formatting an already existing partition would be impossible).

Who would you write to get this information?

Simply post it on the forum - the developers read (and answer!) here.

8

Re: How to shrink a partition without data loss...

stormhead wrote:

Anything listed in this screen shot as "occupied" (that is, non-white block) is recorded in the file system metadata area and will thus be relocated during resizing. However, these blocks might me marked as e.g. "excluded" by your tool since they are
- either defective (the file system metadata records them as unusable then, and GParted can take care not to use them while resizing. I don't know if it actually does this - maybe one of the developers can help you here!);

GParted uses external tools to perform resize operations.  For instance, to resize FAT file systems, GParted uses the library libparted from the parted project.  To resize NTFS file systems, GParted uses command line tools from the ntfsprogs project.  Hope this helps :-)

Gingerale, if you are planning to edit disk partitions then I highly recommend that you create a backup of your data first.  The gparted application is designed to enable you to edit partitions while reducing the risk of data loss. The application is carefully tested and is used by the GParted project team. However, loss of data might occur due to software bugs, hardware problems, or power failure.

9

Re: How to shrink a partition without data loss...

stormhead wrote:
Gingerale wrote:

Who would you write to get this information?

Simply post it on the forum - the developers read (and answer!) here.

The answer's by stormhead above, and including this one are correct.

The forum is the best place to search and post questions.  That way others can benefit from the answers and contribute to the conversation.

10

Re: How to shrink a partition without data loss...

Hi,

Stormhead:  thx for the complimentary information,this basically puts my concerns about metadata management by Gparted to rest.

Gedakc:  thx for the confirmation.  On the subject of backing up, the data goes without saying, absolutely, but the system is another matter.  Rebuilding a Linux machine with all the software for a given business environment can take anywhere between half a day to a couple of days, and rebuilding a Windows machine can be longer than that. 

I've now come to the conclusion that I want all workstations to have minimal software, use application servers where possible and all data on our file server, and that each machine should have a compressed image of the system and application programs on the file server.  After reviewing four forums and a google search on the matter I've come to the conclusions:

  • that the best choice, as reported but I have not tried, is x4L (previously G4L) vs Norton Ghost or Acronis True Image.

  • that partition images are restored fine but that the boot loader must be re-installed for some reason

  • so this means that, other than the compression, Gparted could be used for the same purpose

Now in view of the mention by Stormhead that some of the operations for partition resizing is done with externally sourced library modules would it not be possible to add to the partition copy function a "tarball" option so that the partition copy is compressed?


Thx for your patience.

11

Re: How to shrink a partition without data loss...

Gingerale wrote:

Rebuilding a Linux machine with all the software for a given business environment can take anywhere between half a day to a couple of days, and rebuilding a Windows machine can be longer than that.

Yes, rebuilding with only the software install disks and backup of the data can take a long time.  For quick rebuilding, I use a image copy of the partition.  My experience with partimage has been very good.

If you are looking for a solution similar to Norton Ghost then take a look at Clonezilla.

Another promising backup product under active development is fsarchiver.

Gingerale wrote:

Now in view of the mention by Stormhead that some of the operations for partition resizing is done with externally sourced library modules would it not be possible to add to the partition copy function a "tarball" option so that the partition copy is compressed?

Such a feature has been requested and is planned.  At the moment one of the most promising solutions would use fsarchiver as the third party backup tool.

12

Re: How to shrink a partition without data loss...

Hi!

Gingerale wrote:

that partition images are restored fine but that the boot loader must be re-installed for some reason

Not for some reason, but simply because parts of the boot loader are located outside of any partition. The boot loader starts with a small section contained in the boot drive's MBR, then it might include some program code stored in the slack space between the MBR and the beginning of the first partition (using the msdos disklabel format, each partition should start and end at a cylinder boundary. Since the MBR is the first sector on the first track and head of the hard drive, the first partition can not start before the first sector of track 1 / head 0 - leaving the largest part of track 0 otherwise unused. This area is often called "slack space"), and then only come the parts of the boot loader that might be located inside the boot partition itself.
To be more precise, the MBR contains the basic code used to initiate the booting process; the slack space is a typical point where a small file system driver would be located, enabling the boot loader to read the boot partition. The boot partition itself then contains the boot loaders menu system, configuration files, background images etc.
Since anything outside a partition is not in the scope of a partition imaging tool, these areas will not be backed up - and thus not restored.

13

Re: How to shrink a partition without data loss...

Hello Gedak,

Thx for the info on the other partition imaging programs.  Read what I could find following your links.  Here's a summary of my understanding of these:

  • fsarchiver, the cadillac by far of the group, and I understand why Gparted would go with this option, is essentially implemented, per their target goals, and is essentially stable, but testing is not completed to their liking.  Of course if this was MS it would already be released lol

  • Clonezilla reported stable and has a power/efficiency suitable for system deployment

  • fsarchiver is about to change their archive file format, and so for a newbe, waiting for the next stable version (no date?) would be worth while

  • partimage reported stable and would be a defacto choice

  • Both Clonezilla and partimage do not purport to be sufficiently tested (to their respective team's liking) for NTFS partitions, but reports do not seem to indicate any significant problem

The most attractive choice from my point of view is fsarchive with two levels of checksums and capacity to recover damaged archive.  My understanding is that Clonezilla and partimage do not use checksums or can recover the remainder of an archive after hitting a damaged sector.  Is this so?

A general question ensues about making use of an image generated by one of these programs should your computer die.  So the working assumptions here are that you have an image that is good and your computer is fried such that the new image must be installed on a new machine.  The questions that come to mind, which also apply to deployment of an image on multiple computers:   what happens when the low level drivers (video, network, etc.) are not the same as in the image?

So 3 little questions.  Thx for your time and help.  Looking forward to your replies.

Cheers,


Gingerale

14

Re: How to shrink a partition without data loss...

Hello Stormhead,

Thx again to you too for the reply and detailed information.  So for some silly reason I'm drawing a blank on how to build the 1st MBR (assuming there's a backup also on the disk).  We have a nice slate of choices for imaging (and in most cases) compressing the partitions.  What about the MBR with the bootstrap and bootloader materials?

I also much appreciate your time and looking forward to your answer.


cheers,


gingerale

15

Re: How to shrink a partition without data loss...

Gingerale wrote:
  • fsarchiver is about to change their archive file format, and so for a newbe, waiting for the next stable version (no date?) would be worth while

The author of the tool would be in the best position to provide a date.  Since it will take some time and testing to integrate the fsarchiver functionality into GParted, I suspect that fsarchiver will be ready before GParted is ready.

Gingerale wrote:

The most attractive choice from my point of view is fsarchive with two levels of checksums and capacity to recover damaged archive.  My understanding is that Clonezilla and partimage do not use checksums or can recover the remainder of an archive after hitting a damaged sector.  Is this so?

I do not know if checksums are used in the above programs.  You could check with the respective authors, or if you are keen, you could peruse the source code.  cool

Gingerale wrote:

A general question ensues about making use of an image generated by one of these programs should your computer die.  So the working assumptions here are that you have an image that is good and your computer is fried such that the new image must be installed on a new machine.  The questions that come to mind, which also apply to deployment of an image on multiple computers:   what happens when the low level drivers (video, network, etc.) are not the same as in the image?

When an image is installed on new hardware that is not the same, it is very likely that some problems will be encountered with the device drivers that are different from the original computer.  This is as it should be.  For instance, why would you install an NVidia video driver on a computer that only has an ATI video card.

If you wish to use a single image to build multiple different hardware computers, you will have to take this into account and adjust for the differences.

16

Re: How to shrink a partition without data loss...

Hi gedakc,

fsarchiver:  hmmm, thx for the advice.  Think I'll use fsarchiver as is and store in the same directory as the image a copy of that version of fsarchiver to manage the image format update issue that they mention on their web site.

With respect of diving into code I've lost comfort and thus keeness of doing such things.

For the driver issue mentioned, one strategy for deployment, could be to create a basic deployement image with generic drivers?

However for the primary use being discussed here, use of an image for recovering from a disaster like a fried computer, to take up the same example as yourself, is it possible to configure "fallback drivers" in Linux such that if a driver stops working due to change of video card, or computer ?   


Maybe the two residual questions should be: 

1)  In preparation for a partition operation such as a reduction using a tool like Gparted, and an image (ghost) of all partitions is made, and an unrelated hardware failure occurs and this image is needed but the hardware repair has caused a material change requiring a different driver (for example video card change) is it possible to restore the image and then start Linux in "safe mode" with all generic drivers, to be able to resolve the driver issue?

2) Does Gparted include functionality, on say a live CD version, to prepare a virgin hard disk with an MBR such that with imaged partitions created for backup can provide a complete recovery solution?


Thx,


xian

(short for Christian as in xmas)

17

Re: How to shrink a partition without data loss...

Hi!

Gingerale wrote:

Thx again to you too for the reply and detailed information.  So for some silly reason I'm drawing a blank on how to build the 1st MBR (assuming there's a backup also on the disk).  We have a nice slate of choices for imaging (and in most cases) compressing the partitions.  What about the MBR with the bootstrap and bootloader materials?

To restore a partition image to a formerly unpartitioned hard drive, the appropriate entries to the partition table must of course be made (otherwise, the system won't see any partitions even after the restore, and thus the restored data would be inaccessible). As a result, you can assume that the available partition imaging tools do create the appropriate partition table entries for the partitions they are currently restoring (if the restore is performed to some unallocated area), or they adapt a pre-existing partition table entry to the needs of the partition they are restoring (if restoring to a pre-existing partition).
These entries are stored inside the MBR - but they are created on the fly while restore is taking place; these partition imaging tools no NOT store a backup of the whole MBR including boot code and everything (this would only be useful if the restore takes place to a hard drive of the exact same size and geometry, and is the partition layout is not changed during restore - so even restoring your backup to a new hard drive would not work with a backed up MBR).
So you can say that the partiton imaging tools in fact do create a valid MBR for the hard drive they are operating on, but this MBR does only contain what is necessary to describe the partition layout - it does not contain the MBR parts of the boot loader (which is not a big deal, sionce the GRUB boot loader - which is currently the most widely used Linux boot loader - can easily be restored off nearly any Linux live CD).

18

Re: How to shrink a partition without data loss...

Hi Stormhead,

Ok so here's the procedure I understood for rebuilding a fried computer:

a) Using a Linux Live CD one installs the Grub (or other) boot loader
b) using Gparted one creates the required partitions
c) using the partition imaging software one restores to one of the partitions just created the desired partion image
d) return to Gparted to indicate which partiton has the boot

This is going to come in handy as my file server a RHEL ver 4, working well since 2004 (2005?) stopped suddenly; config is a RAID5 with four 280GB hard disk.  Think the power supply is dying.  The RAID uses the LVM.   From what I gather I can't use the imaging software that has previously been discussed, I can only use the LVM "snapshot" which is what?

Thx

xian

19 (edited by stormhead 2009-08-20 09:19:27)

Re: How to shrink a partition without data loss...

Hi!

Gingerale wrote:

Ok so here's the procedure I understood for rebuilding a fried computer:

a) Using a Linux Live CD one installs the Grub (or other) boot loader
b) using Gparted one creates the required partitions
c) using the partition imaging software one restores to one of the partitions just created the desired partion image
d) return to Gparted to indicate which partiton has the boot

That's not entirely correct, sinc you can't install GRUB without having a partition that contains all the GRUB components that are not stored in the MBR or slack space. Instead, the procedure would be
1. Check if your imaging software can restore to unallocated space (that is, create the required partitions on the fly during the restore operation). If it can not, create the required partitions in GParted.
2. Restore your partitions using the imaging software.
3. Boot into GParted, open a console window, and re-install the GRUB boot loader (yes, GParted brings along everything you need for this):

grub           # this starts the GRUB shell
root (hd0,1)   # this indicates the partition where GRUB expects its menu system,
               # configuration file, and all the other components that are stored
               # inside a partition. hd0 represents the first hard drive by BIOS
               # enumeration order; the second number is the partition number
               # where the GRUB files reside. Note that both numbers are
               # zero-based - that is, the second primary partition on your only
               # SATA hard drive would be /dev/sda2 in Linux, but (hd0,1) in GRUB!
               # This command also causes GRUB to analyse the file system used
               # on this partition and decide which driver needs to be embedded in
               # the slack space, along with the information which partition is the
               # root device.
setup (hd0)    # this installs the GRUB boot loader to the hard MBR
               # of your first hart drive and embeds a pointer to the root device
               # and the required file system driver into the slack space. It is
               # also possible to install GRUB into a partition's boot sector
               # (using a "setup (hd0,1)" syntax), but if you are running only
               # Linux on your machine, this is hardly necessary.
quit           # this exits the GRUB shell

4. Mount your restored partition, and check that the fstab and kernel command lines (in the GRUB menu.lst) reflect the partition layout of this machibe after restore.
5. reboot the machine and boot into your freshly-restored system. If you want to be sure the GRUB version used by GParted matches the GRUB components contained in your imaged system, you might want to re-execute step 3 from within your restored system, once you've booted it.

From what I gather I can't use the imaging software that has previously been discussed

You might be able to use it, if it
1. brings along a driver for your RAID controller (or the software RAID system used on your installation, if you use a kernel-based software-only RAID) on its live media;
2. can handle LVM volumes. You'll have to ask the guys wh write this imaging tool if their live media can do this.

I can only use the LVM "snapshot" which is what?

I'm not deep enough into LVM to give any good answer here - but while setting up the LVM2 configuration on my "external hard drive" (an old dual-Pentium III machine, running at 450 MHz on 256 MBytes of RAM... but still fast enough to deliver up to 40 MBytes/sec. across the network), I figured out that the LVM man pages are really good; you'll find anything you need to know in there. However, they are written for users who are willing to dig into some quite complex stuff - so they might not be easily understandable if you don't have a somewhat technical background or have never worked with volume managers before.