Can APFS special files save storage space?

I’ve long been critical of some of the best-selling utilities for the Mac, that set out to perform deduplication of files by detecting which appear to be identical, and removing ‘spare’ copies. This is because APFS introduced clone files, and in the right circumstances those take up no space in storage, as their data is common and not duplicated at all. As it’s practically difficult to tell whether two files are clones, any utility or command tool that claims to save space by removing duplicates can’t tell you the truth, and in most cases won’t save as much space as it claims.

Claims made by those utilities are often exaggerated. This is because they calculate how much space they think they’ve saved by adding the sizes of the potential duplicates they have deleted. That’s not correct when a clone file is deleted, as that doesn’t actually free any space at all, even though the clone file has exactly the same nominal size as the original.

Benefitting from clone files

I’m delighted to see the eminent John Siracusa turn this on its head and finally make better use of clone files in his app Hypercritical, available from the App Store. Instead of deleting clones, his app can replace duplicate copies with clones, and so achieve real space savings. This comes down to simple arithmetic:

if you have two copies (not clones) of a file in the same APFS volume, the total size they take on disk is twice the size of one of them;
if you have two clones (not copies) of a file in the same APFS volume, the total size they take on disk is only the size of one of them, as its clone takes no additional space at all.

Hypercritical thus checks all the files in a selected folder, identifies which are identical copies, and (where suitable) will replace those copies (except an original) with clones, so saving real storage space.

I also think it has the most user-friendly payment scheme: download Hypercritical free of charge and check your Mac with it. If it doesn’t find sufficient savings, and you decide not to use it to replace any duplicates with clones, then it costs you nothing. If you want to reclaim that space, then you can opt to pay according to the amount of space it saves, by subscription, or with a one-time payment. On that basis, I unhesitatingly recommend everyone to download it from the App Store, and at least check their Home folder to see if it’s worth paying to reclaim space. You have absolutely nothing to lose.

In my case, perhaps because I tend to clone files using the Finder’s Duplicate command, the savings that it offered were of little benefit, but your Home folder could be different and release 100 GB or more.

Sparse files

The other space-saving special file type in APFS is the sparse file. Although it can bring great savings in storage space, that’s largely up to the app(s) that create and maintain the file, rather than the user. Devising an app that could go round converting plain to sparse files is harder, and risks incompatibility with those apps that access those files.

Fitting 285 GB into 16.5 GB

As a demonstration of how effective APFS special files are in saving disk space, I built myself a 100 GB partition (APFS Container) on an SSD and tried to fill it with clone and sparse files until I got bored.

At this stage, the 100 GB partition contains:

One 16.5 GB IPSW image file, with nine clones of it, created using the Duplicate command.
Eleven 10 GB sparse files and one clone, created using my app Sparsity.

Add those file sizes together and they come to 285 GB, yet the 100 GB partition only has 16.5 GB stored on it, and still has over 83 GB free. No compression is involved here, of course.

As the saying goes, there ain’t such as thing as a free lunch, and that free space could vanish quickly depending on what happens to those files. The worst case is for an app not to recognise sparse files, and write one to disk in plain format, so swallowing 10 GB at once. Editing the cloned files would be a more gradual way of their growing in size. Only changed data would then need to be saved, so free disk space would steadily fall as more changes were made to the clone.

Clone and sparse files are by no means unique to APFS, but they can be impressive, and above all they’re effective at reducing excess erase-write cycles that age SSDs, whatever you do with the storage they free.

I’m very grateful to Duncan for drawing my attention to Hypercritical, and to John Siracusa for an outstanding app.