1 Topic by rahrah

  • rahrah
  • New member
  • Offline
  • Registered: 2009-12-11
  • Posts: 7

Topic: Another NTFS volume size issue with possible resolution

Hi,

I, too, have encountered the problem where the ntfs partition is
the larger than the partition.  And have made some small
investigations in limited time.

System.

Slackware 12.2.

Linux 2.6.31.5 kernel.  Linus' main tree, with only this security
patch to fs/pipe.c

http://git.kernel.org/?p=linux/kernel/g … 0a8cc4466c

Custom kernel config.

Gparted 0.5.0 - no patches or updates.

Parted 1.9.0 - different patch sets (see below).

Running with parted 1.9.0 and only the patch you suggest making here:

http://git.debian.org/?p=parted/parted. … 0b1e7390ce

ad25892bb995f61b0ddf801ed1f74e0b1e7390ce

I got the error where the ntfs file system is bigger than the
partition.  Restoring the partition to its old size (with a saved
sfdisk output file) made the ntfs data accessible again.

Experimenting now, after shrinking a disk from 100GB to around
14GB, then resizing to 100MB, I'm left with a 100GB partition but
a 14GB ntfs file system.

# ntfsresize -i  -f -n /dev/sdd1
ntfsresize v2.0.0 (libntfs 10:0:0)
Device name        : /dev/sdd1
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 14608060928 bytes (14609 MB)
Current device size: 100027597824 bytes (100028 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 14606 MB (100.0%)
Collecting resizing constraints ...
You might resize at 14605754368 bytes or 14606 MB (freeing 3 MB).
Please make a test run using both the -n and -s options before real resizing!

Using a version of parted 1.9.0 with some fedora patches
applied seems to fix all these problems.

I downloaded it here (from a link in the forums):

http://kojipkgs.fedoraproject.org/packa … 12.src.rpm

Under Slackware, I extracted the patches and applied them in this
order as per the rpm spec file:

parted-1.9.0-appletv-support.patch
parted-1.9.0-extended-mbr.patch
parted-1.9.0-noheaders.patch
parted-1.9.0-pop-push-error.patch
parted-1.9.0-no-cylinder-align.patch
parted-1.9.0-remove-struct-elem.patch
parted-1.9.0-move-function-declarations.patch
parted-1.9.0-dasd-duplicate.patch
parted-1.9.0-new-duplicate.patch
parted-1.9.0-handle-dup-error.patch
parted-1.9.0-swap-flag.patch
parted-1.9.0-volkeysize.patch
parted-1.9.0-no-BLKPG.patch
parted-1.9.0-commit-without-close.patch
parted-1.9.0-dont-touch-part-nodes.patch

I haven't done a bisection to see which patch fixes the problem.

I've made many tests, and cannot get an error with the fedora
patchset version of parted-1.9.0, but get regular errors with just
the patch (commit: ad25892bb995f61b0ddf801ed1f74e0b1e7390ce) you
recommend.  In order to get the errors, on the disk I have, I do
something like this:

###
Start with about 12GB of data on the disk.
:loop
Resize the partition to max.
Test if all OK with gparted and ntfsresize -i  -f -n /dev/sdd1
If yes, continue, else investigate.
Mount disk with ntfs-3g and write data to the drive, perhaps using
a simple script like this:

wget http://www.kernel.org/pub/linux/kernel/ … .7.tar.bz2
wget http://www.kernel.org/pub/linux/kernel/ … 32.tar.bz2

rm -rf dir_*

while : ; do
  echo "mkdir dir_$CNT"
  mkdir dir_$CNT
  echo "cp *.bz2 dir_$CNT/"
  cp *.bz2 dir_$CNT/
  if [ $? -gt 0 ]; then
    break
  fi
  CNT=$((CNT+1))
done

Interrupt above script when about 10GBs of disk space has been
copied.

Delete all dirs bar the last dir made, so I possibly have some
fragmentation, and have written to clusters/sectors sitting
outside the smaller ntfs file system.

Resize too just +1 GB of the smallest size I can resize to.
Test if all OK with gparted and ntfsresize -i  -f -n /dev/sdd1
If yes, loop, else investigate.

###

I don't really know which patch set you are applying to
parted-1.9.0. I cannot seem to find the source code to your live
CD, and assume you don't actually release it (tut tut ;).  It
might be good, though, to indicate what patches you are using on
what packages, and perhaps give instructions (or scripts) on how
to recreate the CD.  At the moment, it seems that there are just FAR
too many parted-1.9.0 variants around to have a good idea which
patch set you recommend.

Further, it might be useful if you kept your own patched version
of parted on your own servers so we might all try to use the same
version.  Nobody seems to be using the official, unpatched one:

gzip -cd parted-1.9.0.tar.gz | sha1sum -
af1ec6ef4bcb672f83c91cdc5bbfadd6f3f7e34d  -

which is the same as the unpatched parted-1.9.0.tar.xz in the
fedora parted-1.9.0-14.fc12.src.rpm package.

Thanks for the fabulously useful gparted, and for making it
available under the GPL.  I hope these problems get resolved
quickly.

All the best,

===Rich

2 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Another NTFS volume size issue with possible resolution

Thank you Rich for your detailed post.

As you have discovered, this problem is a tricky one to solve.  We have recently learned that even the fedora patches to parted-1.9.0 do not fix the problem in all cases.  We still don't know the cause of this latest problem, but it occurs with GParted Live 0.5.0-3.  See
Bug 604298 -  Problems resizing file systems with gparted-live-0.5.0-3

We don't try to hide anything here at the GParted project.  In fact we try to keep everything as open as possible so that more people can contribute.  :-)

The patches we use, and instructions on how these patches are applied can be found in the following post:
How to apply fedora patches to parted-1.9.0

If you wish to build your own customized version of GParted Live, see this link:
Create GParted live from scratch

Hope that helps to answer your questions.

3 Reply by rahrah

  • rahrah
  • New member
  • Offline
  • Registered: 2009-12-11
  • Posts: 7

Re: Another NTFS volume size issue with possible resolution

Hi Curtis,

Thanks for getting back to me.

We don't try to hide anything here at the GParted project. 
In fact we try to keep everything as open as possible so
that more people can contribute.  :-)

I'm sorry to have sounded critical, and I'm very grateful that you
develop gparted in the open.  I don't mean to imply that you HIDE
stuff, but it is difficult to find all the info necessary to build
a working parted/gparted combination.

Might I suggest the following:

a) Have a local copy of parted, with ALL patches you recommend in
one place on your Web site, rather than having this info in many
places in forums.  There really should be a canonical version of
parted that we can all EASILY reference.  This way, we can all
start from the same place.  At the moment, it's easy to assume, as
I did, from your News section that there was only one patch to
make to parted.  We have lots of versions of parted to choose from
at the moment: the 1.9.0 src release, the git version, the version
patched with the only patch you say we need on your News page, the
Fedora patched version, the Fedora patched version with the
parted-1.9.0-mftres.patch.

b) Give a full list of all the patches you apply to all the
standard software on your live CD, in an easily accessible place
on your web site.  The generic Debian Live instructions you have
for creating one's own CD have nothing about the patches you make
to parted or, if any, to gparted. Having all these versions here,
there and everywhere with only devoted followers of your forums
being able to tell them apart is frustrating.  I've been building
a version for Slax.  With patch info easily to hand, I would have
saved hours building and testing all these versions of parted.

c) Note in your News section that there are still currently
problems with the software on certain platforms.  I would think
the majority of users would use gparted only on their own systems. 
If it fails, screwing their data up, they won't readily come back. 
It's a shame that this great piece of software should be tainted
in people's minds like that. It's great that you give people 1 to
1 advice in rescuing their data, but it might be better if they
hadn't lost it in the first place.

d) Devote one thread in your forum to things people can do to help
track this resize bug down, if you think they can.

Forgive me if this sounds critical or negative, it's not my
intention.  I am very grateful for all the hard work you put in
to make gparted the great utility it is.

Finally, which patches are you currently applying to parted-1.9.0?

Thanks again,

===Rich

4 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Another NTFS volume size issue with possible resolution

rahrah wrote:

a) Have a local copy of parted, with ALL patches you recommend in
one place on your Web site, rather than having this info in many
places in forums.  There really should be a canonical version of
parted that we can all EASILY reference.  This way, we can all
start from the same place.  At the moment, it's easy to assume, as
I did, from your News section that there was only one patch to
make to parted.  We have lots of versions of parted to choose from
at the moment: the 1.9.0 src release, the git version, the version
patched with the only patch you say we need on your News page, the
Fedora patched version, the Fedora patched version with the
parted-1.9.0-mftres.patch.

Until recently we did not have any patches, other than the normal Debian ones, in GParted Live for parted.  The discovery of this problem is a recent phenomenon.  The problem appears to be caused by some combinations of newer Linux kernels, parted, and some other unknown difference that continues to plaque this problem.

Unfortunately it is difficult to track down this problem.   We cannot reproduce the problem reliably either.  I have little to no time to solve the problem because I am busy solving fixing broken NTFS file sytems.  ;(

rahrah wrote:

b) Give a full list of all the patches you apply to all the
standard software on your live CD, in an easily accessible place
on your web site.  The generic Debian Live instructions you have
for creating one's own CD have nothing about the patches you make
to parted or, if any, to gparted. Having all these versions here,
there and everywhere with only devoted followers of your forums
being able to tell them apart is frustrating.  I've been building
a version for Slax.  With patch info easily to hand, I would have
saved hours building and testing all these versions of parted.

Ideally none of these patches would be required.  It might help if the parted project were able to make more frequent releases.  But like me they are probably very busy too.

If this resizing problem had never occurred it would have saved me TONS of time.  Instead I have had to spend most of my free time trying to figure out what the heck was going wrong.  If the root cause was within GParted, then I could fix the code directly.  Unfortunately this does not appear to be the case.

Still when I have a chance I can see that better error handling within GParted could catch the error sooner and abort before the NTFS volume grows larger than the partition size.  Of course being open source anyone can tackle this problem an propose a patch.

rahrah wrote:

c) Note in your News section that there are still currently
problems with the software on certain platforms.  I would think
the majority of users would use gparted only on their own systems. 
If it fails, screwing their data up, they won't readily come back. 
It's a shame that this great piece of software should be tainted
in people's minds like that. It's great that you give people 1 to
1 advice in rescuing their data, but it might be better if they
hadn't lost it in the first place.

News has been updated.

rahrah wrote:

d) Devote one thread in your forum to things people can do to help
track this resize bug down, if you think they can.

Unfortunately I do not know where the problem originates, and hence do not know what to ask others to help with.  The bug report is the place to track the problem.

rahrah wrote:

Finally, which patches are you currently applying to parted-1.9.0?

The last set of patches used in GParted Live 0.5.0-3 for parted are those from the fedora 12 "parted-1.9.0-14.fc12.src.rpm".

5 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Another NTFS volume size issue with possible resolution

rahrah wrote:

I'm sorry to have sounded critical, and I'm very grateful that you
develop gparted in the open.  I don't mean to imply that you HIDE
stuff, but it is difficult to find all the info necessary to build
a working parted/gparted combination.

No worries.  We are also grateful to people like you that make GParted available to a wider audience.

This partition resizing problem is particularly frustrating because the root cause appears to reside outside of the GParted source code.

6 Reply by rahrah

  • rahrah
  • New member
  • Offline
  • Registered: 2009-12-11
  • Posts: 7

Re: Another NTFS volume size issue with possible resolution

Hi Curtis,

Thanks for getting back to me, thanks for updating the
News section, and thanks for letting me know which patches
you are using!

Just wondered, are all these resize errors only with ntfs?
Also, is it only the final ntfsresize command, the one of this
form:

ntfsresize -P --force /dev/sdXn

that finally screws up the file system size?

Would it not be possible to check for errors in the output of
the penultimate ntfsresize command of the form:

ntfsresize -P --force /dev/sdXn --no-action

to see if the value of the output field:

Current device size:

is (much) more than was made in the last partition resize?

If it is ntfsresize that's getting the volume size wrong, is not
the problem somewhere in
ntfsprogs-2.0.0/libntfs/device.c:ntfs_device_partition_start_sector_get
?

Sorry if this is just noise, I've got a little bit too interested
in this problem.

All the best,

===Rich

7 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Another NTFS volume size issue with possible resolution

rahrah wrote:

Just wondered, are all these resize errors only with ntfs?

All file systems are affected.  NTFS just happens to store the size of itself within the file system for comparison to the partition size.

rahrah wrote:

Also, is it only the final ntfsresize command, the one of this
form:

ntfsresize -P --force /dev/sdXn

that finally screws up the file system size?

In this specific instance of the problem, yes it is the final ntfsresize command that finally "screws" up the file system size.

Unfortunately the problem is much bigger than this.  The crux of the problem is that parted sees a new updated copy of the partition table, whereas the Linux kernel still sees an older copy.  Hence this can affect partition deletion and creation as well.  Innocent commands such as formatting a partition with a file system will use this kernel version of the partition table (outdated copy).

rahrah wrote:

Would it not be possible to check for errors in the output of
the penultimate ntfsresize command of the form:

ntfsresize -P --force /dev/sdXn --no-action

to see if the value of the output field:

Current device size:

This could help catch this one instance of the problem.  We really need a solution that catches all instances of the problem.  Otherwise all system commands will see an outdated version of the partition table.

rahrah wrote:

is (much) more than was made in the last partition resize?

If it is ntfsresize that's getting the volume size wrong, is not
the problem somewhere in
ntfsprogs-2.0.0/libntfs/device.c:ntfs_device_partition_start_sector_get
?

Again the ntfsresize command is like many other commands.  These retrieve the kernel version of the partition table.  The crux of the problem is that the copy of the partition table that the kernel sees is not updated.

8 Reply by rahrah

  • rahrah
  • New member
  • Offline
  • Registered: 2009-12-11
  • Posts: 7

Re: Another NTFS volume size issue with possible resolution

The crux of the problem is that parted sees a new updated copy of the partition table, whereas the Linux kernel still sees an older copy.

Does not:

/sbin/blockdev --getsize64 /dev/sdXn

always give the kernel's view of the partition?  If so, could not the return val of this be compared with
parted's version as a sanity test before any at risk operation?

===Rich

9 Reply by stevenshiau

  • stevenshiau
  • Member
  • Offline
  • Registered: 2008-04-02
  • Posts: 293

Re: Another NTFS volume size issue with possible resolution

Finally, which patches are you currently applying to parted-1.9.0?

The patches applied in GParted live 0.5.0-3 are:
# Debian specific patches
doc-package
reiserfs-libname

#
01-appletv-support.dpatch
02-extended-mbr.dpatch
03-noheaders.patch.dpatch
04-pop-push-error.dpatch
05-no-cylinder-align.dpatch
06-remove-struct-elem.dpatch
07-move-function-declarations.dpatch
08-dasd-duplicate.dpatch
09-new-duplicate.dpatch
10-handle-dup-error.dpatch
11-swap-flag.dpatch
12-volkeysize.dpatch
13-no-BLKPG.dpatch
14-commit-without-close.dpatch
15-dont-touch-part-nodes.dpatch

You can find all of them in
http://free.nchc.org.tw/drbl-core/pool/ … /p/parted/
(version parted 1.9.0-4drbl).

Steven.

10 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Another NTFS volume size issue with possible resolution

rahrah wrote:

Does not:

/sbin/blockdev --getsize64 /dev/sdXn

always give the kernel's view of the partition?  If so, could not the return val of this be compared with
parted's version as a sanity test before any at risk operation?

That is a possibility.  Still the problem should not occur in the first place.  I have recently discovered that some updates to Ubuntu 9.10 have solved this problem of the failure to update the kernel of partition changes.  Next is to figure out what needs to be done to fix the GParted Live CD.

11 Reply by rahrah

  • rahrah
  • New member
  • Offline
  • Registered: 2009-12-11
  • Posts: 7

Re: Another NTFS volume size issue with possible resolution

stevenshiau wrote:

The patches applied in GParted live 0.5.0-3 are:

...

Steven.

Hi Steven,

Thanks for the info.  I hadn't seen DRBL before, so thanks for that, too.

All the best,

===Rich

12 Reply by rahrah

  • rahrah
  • New member
  • Offline
  • Registered: 2009-12-11
  • Posts: 7

Re: Another NTFS volume size issue with possible resolution

gedakc wrote:
rahrah wrote:

Does not:

/sbin/blockdev --getsize64 /dev/sdXn

always give the kernel's view of the partition?  If so, could not the return val of this be compared with
parted's version as a sanity test before any at risk operation?

That is a possibility.  Still the problem should not occur in the first place.  I have recently discovered that some updates to Ubuntu 9.10 have solved this problem of the failure to update the kernel of partition changes.  Next is to figure out what needs to be done to fix the GParted Live CD.

I hope the patches worked.  If so, could you post links to them, please.

Is it not a good idea to have some sanity tests in gparted to check for future errors like these?  Checking for
discrepancies, and perhaps trying to fix them instead of assuming that all's well?  If errors like this CAN occur, then
surely it would be best to be prepared for future instances.

===Rich

13 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Another NTFS volume size issue with possible resolution

Better error handling would be a good thing.  The error is first identified by libparted in a C language exception routine.

Perhaps you would consider writing a patch?

14 Reply by rahrah

  • rahrah
  • New member
  • Offline
  • Registered: 2009-12-11
  • Posts: 7

Re: Another NTFS volume size issue with possible resolution

gedakc wrote:

Perhaps you would consider writing a patch?

I take your point! smile

Thanks for listening.

15 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Another NTFS volume size issue with possible resolution

And thank you for your ideas.