APFS: Special file types

Compared to its predecessor HFS+, APFS brings several new types of file and link. This article gathers together brief summaries of those we currently know about, even though accounts in the APFS Reference may be cursory or omitted altogether.

Clone files

I have already given an account of how clone files are used in APFS. Instead of making an ordinary copy of a file with each of its components duplicated, whenever possible APFS will instead make a copy of the inode and its attributes, together with file extent information, as a clone of the original.

This can be verified by demonstrating that their inode numbers are different, as is information in the attributes such as the file’s name. There’s a flag in the file’s attributes to indicate that cloning has taken place, but working out their current degree of overlap requires comparison of their extents, and isn’t available outside the file system.

Dataless files

These have been introduced more recently to support non-replicating FileProviders, including iCloud Drive. When the local copy of a file also stored remotely in the cloud is evicted, its data is removed from local storage, rendering it dataless.

When that file is to be used locally again, its data has to be downloaded from the cloud service, and the local dataless file is materialised by adding its data back. Because that local file never lost its metadata, those remain intact, as should any locally stored versions. Although the APFS Reference details flags for dataless snapshots, it doesn’t contain any information about dataless files, which do have a flag to mark that state in their attributes, as explained here.

Sparse files

In common with many other modern file systems, APFS provides a special format for files consisting of substantial quantities of null data, sparse files. The APFS Reference provides no information about their format or handling, though, only details of flags used to distinguish them.

Sparse files aren’t created by a process writing large amounts of zeroed data, but require specific operations using a FileHandle. These might start with opening the FileHandle for writing, then writing blocks of non-null data and moving the write pointer through the null data using seek(). An example is shown in the code below.

FileHandles can’t be used to create a sparse file from an existing file that already has blank data stored on disk.

One notable disadvantage of sparse files is their propensity to explode to full size when copied to another file system such as HFS+. As transparent conversion to the sparse file formats of other file systems isn’t available, there’s currently no way avoid that.

Trimmed disk images as sparse files

macOS Monterey brought together sparse files, UDRW read-write disk images, and APFS trimming in a powerful combination that results in high-efficiency storage of disk images that were previously of fixed size.

When a UDRW disk image is first created, it’s written as a single file of the size set by the user. When that disk image is mounted for a second time, it’s transformed into a sparse file, with all its unused blocks trimmed on mounting and converted to skipped space. This doesn’t happen during its initial mount, because trimming is only triggered automatically when its file system is mounted. This isn’t described in the otherwise detailed account given in man hdiutil.

Evidence for this appears in the mount sequence in the log, which begins with its preliminaries (times given in decimal seconds from an arbitrary start):
2.015645 dev_init:299: disk6 device_handle block size 4096 block count 1169499 features 16 external
2.015826 nx_mount:1224: disk6 initializing cache w/hash_size 8192 and cache size 32768
2.023228 nx_checkpoint_find_valid_checkpoint:565: disk6 sanity checking all recently-changed container state… please be patient.
2.023567 nx_mount:1553: disk6 checkpoint search: largest xid 1, best xid 1 @ 1
2.023574 nx_mount:1580: disk6 stable checkpoint indices: desc 0 data 0

That’s followed by Spaceman, the APFS Space Manager, scanning for free blocks, and trimming them:
2.023580 spaceman_metazone_init:110: disk6 no metazone for device 0, of size 4790267904 bytes, block_size 4096
2.023783 spaceman_scan_free_blocks:3311: disk6 scan took 0.000179 s (no trims)
2.023813 apfs_newfs:30416: disk6s1 FS will NOT be encrypted.
2.024045 spaceman_scan_free_blocks:3293: disk6 scan took 0.000256 s, trims took 0.000035 s
2.024048 spaceman_scan_free_blocks:3295: disk6 1164953 blocks free in 1 extents
2.024050 spaceman_scan_free_blocks:3303: disk6 1164953 blocks trimmed in 1 extents (35 us/trim, 28571 trims/s)
2.024052 spaceman_scan_free_blocks:3306: disk6 trim distribution 1:0 2+:0 4+:0 16+:0 64+:0 256+:1

This feature is limited to disk images containing APFS and HFS+ file systems, as they have trim support, unlike FAT or ExFAT. That in itself can appear confusing, because HFS+ as a file system doesn’t support sparse images, although here a disk image containing an HFS+ file system enjoys the same preferential treatment, and is converted into a sparse file when hosted on APFS.

Further economy in the use of disk space can be seen when such sparse disk images are cloned, for example in cloning a macOS virtual machine hosted on APFS.

Symbolic links

Symlinks are special files in APFS, and the volume superblock maintains a total number of symlinks on that volume. Rather than consisting simply of text data, they contain at least one extended attribute named com.apple.fs.symlink containing the path in the link. That extended attribute is marked as being owned by the file system, although that doesn’t affect the symlink’s permissions.

Firmlinks

These were introduced with macOS 10.15 Catalina to form bidirectional links between locations in the System and Data volumes in a boot volume group. These are apparently marked with another extended attribute named com.apple.fs.firmlink. Apple doesn’t provide any further details.

System firmlinks appear to be required to be created early in the life of a volume, and can’t apparently be repaired or recreated later. This caused problems for many using Big Sur, for instance, who ended up with orphaned Data volumes that had lost their firmlinks to their paired System volume. There are currently no user tools for working with system firmlinks, nor any method for repairing or recreating them once both volumes have been created.

Since their introduction, the list of system firmlinks given in /usr/share/firmlinks has remained the same:
/Applications <-> Applications
/Library <-> Library
/System/Library/Caches <-> System/Library/Caches
/System/Library/Assets <-> System/Library/Assets
/System/Library/PreinstalledAssets <-> System/Library/PreinstalledAssets
/System/Library/AssetsV2 <-> System/Library/AssetsV2
/System/Library/PreinstalledAssetsV2 <-> System/Library/PreinstalledAssetsV2
/System/Library/CoreServices/CoreTypes.bundle/Contents/Library <-> System/Library/CoreServices/CoreTypes.bundle/Contents/Library
/System/Library/Speech <-> System/Library/Speech
/Users <-> Users
/Volumes <-> Volumes
/cores <-> cores
/opt <-> opt
/private <-> private
/usr/local <-> usr/local
/usr/libexec/cups <-> usr/libexec/cups
/usr/share/snmp <-> usr/share/snmp
/AppleInternal <-> AppleInternal (on Apple engineering systems only)
in each case shown as the system volume path and the Data volume path that are firmlinked.

Users can create synthetic firmlinks, which appear at root volume level and link to folders or symbolic links on the Data volume. These are stored in an /etc/synthetic.conf file, and are documented in man synthetic.conf. Rich Trouton has published an excellent tutorial on how to do this. Users who previously had their own custom folders at the root level of the boot volume can use this to create the same effect in Catalina and later.

Accounting for space

It appears that the primary motivation for space-saving file formats in clone and sparse files isn’t to return more free space for the user, but to minimise on the amount of data that has to be written to an SSD to store each file, so extending its working life. This also improves performance: cloning a 100 GB file takes but an instant, whereas copying 100 GB of data from one file to another on the same volume requires an appreciable amount of time.

Accounting for the space taken up by sparse and clone files becomes complicated and may appear misleading:

for individual sparse files, the actual (reduced) space taken on disk is given;
for individual clone files, the original (unreduced) space taken on disk is given;
for sparse and clone files within a volume, the actual (fully reduced) total space taken is given;
for sparse and clone files within a volume, the decision as to whether it’s possible to copy another file to that volume can be made on either reduced or unreduced space.

Two examples illustrate the inconsistencies that can result. These are given for a single APFS volume in a container of 100 GB, with 100 GB available space in that volume initially:

A single sparse file of 120 GB unreduced size, but only 50 KB actual space taken, can’t be copied to that volume because there’s insufficient space.
Five or more clones of a 21 GB sparse file, occupying 2 GB of disk space for the original, can be made without error. At the end, total space used is reported as 2 GB, leaving 98 GB free space available.

Thus there are occasions when sparse and clone files do allow the user to make full use of the disk space they save, and others when the user is blocked from doing so. While these conflicting behaviours may arise in layers well above the file system, they suggest that APFS may not provide consistent guidance.

Articles in this series

1. Files and clones
2. Directories and names
3. Containers and volumes
4. Snapshots
5. Encryption and sealing

References

Apple’s APFS Reference (PDF), last revised 22 June 2020.