1 Topic by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Topic: Idea to improved SSD handling by using TRIM

Hi,

I have an idea to improve how GParted handles SSDs (and thin provisioned enterprise class storage) by using TRIM.

  • Make GParted use TRIM when creating and/or deleting partitions.

    • After creating a partition TRIM the sectors within it, before doing anything else.
      (Recent versions of mkfs.ext[234] and mkfs.xfs already TRIM the partition before creating the FS).

    • Immediately before deleting the partition TRIM the sectors within it.

  • Optionally make it configurable in a menu item, defaulting to on.

    • Perhaps Device --> Enable/Disable TRIM.

Implementation ideas:

  • Use hdparm to query a disk for TRIM support.

  • Alternative, just always try TRIM and just ignore errors.

# hdparm -I /dev/sda | grep TRIM
       *    Data Set Management TRIM supported (limit 8 blocks)

  • Use hdparm to perform the TRIM.

    • May need to pass multiple spans to work around underlying limit of the maximum number of sectors the disk can accept in one TRIM command.

  • Alternatively, open the disk device and issue the relevant ioctl() calls directly.

    • Look at the code in mkfs.ext[234/xfs] to see how they do it.

# hdparm --trim-sector-ranges START_SECTOR:SECTOR_COUNT --please-destroy-my-drive /dev/DISK

Documenting here for discussion / review and to allow someone else to implement it.

Thanks,
Mike

2 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

What are the advantages and potential disadvantages to using TRIM?

3 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

Advantages:

  • Maintains maximum write performance of SSDs.

Disadvantages:

  • Irrevocably tells the SSD to erase all TRIMmed sectors.

  • Not necessarilarly fast.  I think some SSDs may take up to many minutes to TRIM for example 100 GB.

  • Works on sectors of whole block device, not partitions.  No partitio n boundary safety net.

(No disadvantages from using TRIM, just from incorrectly using TRIM)

Also see http://en.wikipedia.org/wiki/TRIM.

4 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

Your idea to detect and/or call trim on CREATE or DELETE sounds like a good one.

I had thought to do the detection on device refresh, but this will slow down the refresh of each and every device, even those that do not support TRIM.

5 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

I didn't intent to suggest when detection of TRIM would occur.  Only when to call TRIM, during partition create and/or delete.

Some stats using "hdparm -I /dev/sda" to detect whether TRIM is supported.

  • My desktop with fast CPU and slow disk:

    • Cold cache: 0.20 seconds

    • Hot cache: 0.01 seconds

  • My netbook with slow CPU and fast SSD:

    • Cold cache: 0.02 seconds

    • Hot cache: 0.01 seconds

I think that adding ~0.25 seconds (worst case) to the time of the disk refresh would be acceptable.  Alternatively this price could also just be paid in every partition create and/or delete.

Perhaps even better would be lazy detection.
1) On disk refresh record trim support as unknown.
2) On first use, detect and record supported as yes or no.
3) On subsequent uses its known.

Timing commands:

# Empty linux FS cache, specifically hdparm executable.
sync; echo 3 > /proc/sys/vm/drop_caches

# Cold cache query of disk capabilities
time hdparm -I /dev/sda > /dev/null

# Hot cache query of disk capabilities
time hdparm -I /dev/sda > /dev/null

6 Reply by mfleetwo (edited by mfleetwo 2012-05-20 09:52:49)

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

(This message is not here.  Sorry.  Testing forum).

7 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

Regarding the performance hit, it can get to be quite lengthy with more disk devices.  I have four drives connected to my development PC and occasionally add more.   The time for testing for TRIM is not necessarily the biggest consumer of time.

Caching even more of the results from the device refresh would be a good idea.  That way we could read most if not all of the data once at start-up.  Then only when changes are made to a device would we mark the cache for that device as dirty.  Then on a subsequent refreshes we could skip refreshing devices that have a non-dirty cache.

Of course that is a new request in and of itself.  :-)

8 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

Just though of a further design improvement:

  Use TRIM during partition (or file system) creation and NOT in partition deletion.

This is to maximise data safety and user forgiveness even after a user has confirmed and then performed partition deletion.  Deletion of the partition does not overwrite any of the file system so it is still recoverable using any of the methods documented in the GParted Manual.  Using TRIM at partition deletion makes it unrecoverable.  Therefore only used TRIM during creation when it overwrites any existing data and there is no reasonable expectation of being able to recover.

9 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

Very good point Mike.  It is much better to have a chance to recover data should anything go wrong.

10 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

mfleetwo wrote:

Just though of a further design improvement:

  Use TRIM during partition (or file system) creation and NOT in partition deletion.

This is to maximise data safety and user forgiveness even after a user has confirmed and then performed partition deletion.  Deletion of the partition does not overwrite any of the file system so it is still recoverable using any of the methods documented in the GParted Manual.  Using TRIM at partition deletion makes it unrecoverable.  Therefore only used TRIM during creation when it overwrites any existing data and there is no reasonable expectation of being able to recover.

Further to this point, I think we should consider using wipefs just prior to formatting a partition to clear the previous file system signatures from the partition.  For example:

wipefs -a /dev/sda1

to wipe all file signatures from the first partition on sda1.

I think that is what the user was trying to do in Bug 688523 - Can't format 'reiser4' by formatting over an ext3 file system with reiser4.

11 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

In short yes we should definitely use wipefs (or link with libblkid to achieve the same).

Research, mostly from a while ago:

When GParted is compiled with libparted < 3.0 it calls GParted_Core::erase_filesystem_signatures() when creating a partition and when formatting a partition with a file system.  This then calls libparted function ped_file_system_clobber().  For all recognised file system signatures in the partition it call a file system specific clobber to remove that signature.  Looking at the libparted 2.4 source code I think the list of file systems it can clobber is: ext2, ext3, (not ext4), fat16, fat32, hfs, hfs+, jfs, linux_swap, ntfs, reiserfs, xfs.  Certainly not all the file system types GParted supports.  In libparted 3.0, ped_file_system_clobber() doesn't exist and in libparted 3.1 it's a non-export symbol.

Can't create reiser4 file systems at the minute, but I have found this case which proves file system signature erasure is needed.  Create an unformatted partition then format as fat32 and finally format as nilfs2.  Blkid and GParted detect this as fat32, but corrupted.  Wipefs lists both fat32 and nilfs2 signatures.

I had though that a read after trim would always return zeros, but a quick search suggests that some devices return the original data!  Very strange.

Conclusions:

1) Yes we need to make GParted wipe signatures before creating new file systems, regardless of the version of libparted it is compiled with.  Yes using "wipefs -a /dev/DEVICE" seems like the best way as libblkid in util-linux package always seems to support the widest range of file systems.  Could use ped_file_system_clobber() only as a fall back when wipefs command is not available.  Alternatively we could implement our own signature wiping.  The only other alternative I can think of is to link with libblkid and call suitable functions.  Using wipefs command will be simpler.

2) Ordering when creating new file system should be:

  1. Erase all old signatures

  2. TRIM partition (if supported)

  3. Create file system

3) Currently create_partition() also calls erase_filesystem_signatures() as well as format() new file system.  This means that users can't re-create an unformatted partition to recover from accidental file system and partition deletion.  However a user just creating an unformatted partition probably otherwise expects it to be blank rather than containing previous contents.
Possible solutions:
3.1) Have create_partition() not erase signatures.  Add an extra option to create new partition which formats as "wiped".
3.2) Have create_partition() not erase signatures.  Add Partition > [W]ipe to allow users to optionally erase signatures.  To queue the operation or not?  Probably should make it a queued operation as it is destructive so users will be prompted for confirmation to apply.

12 Reply by gedakc (edited by gedakc 2012-11-19 21:14:02)

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

Thank you for the detailed response.

With the parted project, the focus is now on partitioning only, and to downplay file system functionality.  As you discovered, parted-3.0 removed much file system functionality (all resize ability).  Fortunately with parted-3.1 the project added back basic API resize ability, mainly because there are no free software alternatives for resizing FAT16/32 and HFS/+.

Responses to numbered points follow:
1)  My preference would be to use the command "wipefs -a /path-to-partition" instead of linking to the libblkid library and calling the appropriate API.  By using a command, a user can more easily follow along with the gparted_details.htm log file if they need to manually replicate gparted operations.  Also by using a command we do not need to add a hard dependency for an additional library.  However, in the case of libblkid, we are increasingly relying on it to provide the file system recognition needed by GParted.  As such there may come a time when it makes more sense to link to libblkid for improved error handling and more precise calls directly to the API, rather than being limited to what is available on the command line.

2)  Agreed.  The order you indicated seems best.  :-)

3)  Good catch on create_partition() calling erase_filesystem_signatures().  My thoughts are that erasing file system signatures belongs immediately before formatting a partition.  If GParted is refactored to work this way, then it will permit users to re-create partition boundaries only.  This could be useful if a user deletes a partition and then wants to immediately recreate the lost partition.  Currently a user would have to resort to fdisk or gdisk to perform a similar task.

3.1)  Agreed that create_partition() should not erase signatures.  Instead of adding an extra option to create new partition which formats as "wiped", I think that the "wiped" or "erase file system signatures" should be moved to just before the format operation.
I think we should refactor the code.
Since the remove_filesystem() method already exists, and is called immediately before formatting a partition, we could consider the following logic:

a)  Remove the call to erase_filesystem_signatures() from create_partition(), and format().
By removing the call from create_partition, we increase the ability of a user to recover data.

b)  Remove the call to remove_file_system from the OPERATION_DELETE in the apply_operation_to_disk() method.  This increases the ability of a user to recover data.
EDIT:
I'm not positive about this one.  If a user says he wants to delete a partition, then it might be best to delete the filesystem signatures to avoid problems in the future.

c)  Enhance remove_filesystem() method logic:

  If there is a filesystem specific remove command (fs.remove)
     Call the fs.remove
  Else if "wipe" exists
     Call "wipe"
  Else (if ped_clobber_filesystem() available)
     Call erase_filesystem_signatures()

3.2)  Regarding adding "Partition > [W]ipe" menu, I think that by using the above mentioned changes, the problem with file system signatures should be sufficiently handled.
There probably is a need for a menu command to completely overwrite a partition.  This has been requested in Bug 619348 - GParted should have option to shred.

Is this idea something that you would like to work on?

If so I think we should create a bug report for this enhancement.  In fact, it might be beneficial to create multiple reports:
a)  Improve SSD handling by using TRIM
b)  Improve filesystem signature erasure by using WIPE

13 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

1) Agreed.  Use wipefs command.

3) I think you have miss understood what I was trying to say with (3.1) and (3.2).  They were two different suggested ways that GParted could present the capability to create a partition with and without wiping all file systems signatures.  For (3.2) I wasn't suggesting adding shreding.  For (3.1) can you have a read of the following two commits.  They explain my view of what remove_filesystem() and erase_filesystem_signatures() are about.  I am only talking about erase_filesystem_signatures() here.

  Add file system specific remove() methods (#670171)
  http://git.gnome.org/browse/gparted/com … dd4c27ca1c

  Implement LVM2 PV remove() method (#670171)
  http://git.gnome.org/browse/gparted/com … e928a0d6f3

4) How to issue TRIM?  Use hdparm command if available, or call ioctl() directly.  More details in comment #1 in this thread.

I initially put this TRIM idea out there just in case someone else wanted to work on it but I'm happy to code it now it's more complicated, just as soon as we reach a common understanding.

14 Reply by gedakc

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

Thank you for the clarification Mike, the links to the commit messages, and for offering to code this enhancement.

3)  I had misunderstood your intentions.  I missed the usage of the remove_filesystems() method (had forgotten, and not read the comment under FS::EXTERNAL, my bad) .  The comment in the commit message certainly clears up the use of this method.

3.1)  I think I understand this now.  The suggestion is to have a filesystem type of "Wipe FS" that would erase filesystem signatures when the user choose to create an "Wipe FS" partition.  So "Unformatted" would not call erase_filesystem_signatures, and "Wipe FS" would call erase_filesystem_signatures.

Is this the proper interpretation?

This would allow a person to recreate the partition boundaries for an accidentally deleted partition by choosing "unformatted".  For users wishing to format a partition outside of GParted, they would choose the "Wipe FS" filesystem.

Are there better names we could choose for these "Unformatted" and "Wipe FS" file systems to help our users better distinguish between the two?


3.2)  In this case a new menu option would be added for "Partition --> Wipe File System", so that the user could manually invoke this feature.  This would wipe file system signatures only.


With these two ideas in mind, I lean towards keeping the user interface simpler.  In this situation I think that option 3.1 would be the preferred method.


4)  My preference would be to use the hdparm command.  The reason being that it would be easier for a user to manually recreate the steps shown in the gparted_details.htm log file.  Ideally I like GParted to be as transparent about operations as possible.  Having said that I do realize that with some partition operations that the user would have to resort to fdisk or gdisk to emulate the same functionality.


With GParted, I think we should try to do the "right" thing by default for our users.  This means that they are not required to know when to wipe file system signatures in order to successfully use GParted.  I think that performing TRIM on SSDs fits this definition too.  By this I mean that rather than having a menu option to "TRIM SSD", we do this on behalf of the user at the appropriate time (e.g., before formatting a partition with a file system).

15 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

Curtis, you've got exactly what I was trying to get across. 

For naming in (3.1) how about "unformatted" and "cleared" (or "cleaned", "emptied", "erased", "wiped").

In terms of actually choosing between (3.1), (3.2) or anything else I'm not yet sure.  I like user interfaces that is as obvious as possible, as least likely to be misunderstanding as possible.  To me this may often be the simpler, but not always.  I'll see if I can come up with any other ideas.

Anyway I'll create two bug reports and go from there.

16 Reply by gedakc (edited by gedakc 2012-11-21 19:35:41)

  • gedakc
  • Developer
  • Offline
  • Registered: 2007-11-30
  • Posts: 4,593

Re: Idea to improved SSD handling by using TRIM

mfleetwo wrote:

For naming in (3.1) how about "unformatted" and "cleared" (or "cleaned", "emptied", "erased", "wiped").

I like this suggestion.  "cleared" implies that something is done to the contents of the partition.  In my mind "wiped" might suggest that the partition has been securely erased, which is not the case.

EDIT:
It would be nice to have these two options grouped together.

17 Reply by mfleetwo

  • mfleetwo
  • Developer
  • Offline
  • Registered: 2012-05-18
  • Posts: 477

Re: Idea to improved SSD handling by using TRIM

Created the following bug reports: