Happy birthday APFS, 7 years old today
Seven years ago, on 27 March 2017, Apple introduced one of the most fundamental changes in its operating systems since Mac OS X 10.0 Cheetah was released 16 years earlier. On that day, those who updated iOS to version 10.3 had their iPhone’s storage silently converted to the first release of Apple File System, APFS. Six months later, with the release of macOS 10.13 High Sierra on 25 September, Mac users followed suit.
At the time, these were huge gambles. The potential for disaster was significant, and at the last minute, Apple decided that those Macs with Fusion Drives had to stay with its old Mac OS Extended File System (HFS+) for another year before they too could start up from APFS. In some markets, notably South Korea, even Apple’s own software failed to cope with changes in the way that file names were handled by the new file system.
Although APFS has certainly had its moments over the last seven years, Apple’s gambles have paid off, and proved key to the success of Apple silicon Macs. Had there been no APFS, many of the fundamental technologies like Secure Boot and the Signed System Volume (SSV) would have been far tougher if not impossible to implement. Macs and Apple’s devices had been in dire need of a modern file system for years; while there was a time when it looked as if that could have been ZFS, in 2014 Apple decided to write its own file system from scratch, with Dominic Giampaolo as lead engineer.
Prehistory
The very first Macs shipped with MFS, designed for 400 KB floppy disks and unable to cope with storage larger than 20 MB. In 1985 that was replaced with the Hierarchical File System HFS, and that evolved into HFS+ in 1998, intended to address hard disk seek times, fragmentation, and limited storage space. In 2002, journalling was added to improve its reliability.
Apple announced APFS at WWDC in 2016, when it was expected to be released in another 18 months if development and testing went smoothly. Those who had hoped for ZFS were disappointed and many remain so today. But over the last seven years, APFS has overcome most of its initial problems, is now one of the most widely used file systems, and has proved itself more than capable on anything from a watch to extensive enterprise networks of Macs.
History
macOS Sierra already had a pre-release version for those who wanted to preview it. That ended its life in 10.12.6 as version 0.3, or 249.7.6 in the subsequent numbering system. As we discovered when we upgraded to High Sierra, that was a far cry from what was to come.
After a promising period in beta, Apple discovered fundamental problems between APFS and its popular Fusion Drives. The first release of macOS 10.13 shipped with APFS version 748.1.46, but suddenly dropped support for those, so converted only those startup volumes on SSDs and hard disks. Snapshots were wobbly at first, and it quickly became clear that APFS was never going to perform well on rotating disks.
High Sierra had a stormy early release history, marred by a series of security gaffes. Vulnerabilities were fixed in the Supplemental Update released less than two weeks after 10.13, leaving snapshots to be improved in 10.13.1 on 31 October. Many expected problems with Fusion Drives would be fixed quickly, but those weren’t addressed until the following September. Another problem that troubled the introduction of APFS to all platforms was the refusal during beta-testing to incorporate Unicode normalisation; this had to be resolved in later versions of macOS 10.13 and iOS 10, as explained here and here.
In September 2018, Apple at last released Mojave 10.14 with support for Fusion Drives, accompanied by the first version of the Apple File System Reference. Although a long and detailed document, developers soon realised how incomplete it was, in spite of the long delay in its publication. At last third-party file system developers had some hard information to work with, and users started assuming that third-party disk maintenance and repair tools were imminent.
Catalina brought major changes to APFS, with the use of expanded volume roles to form System Volume Groups, with their separate but firmlinked System and Data volumes. macOS 10.15.5 fixed a serious bug preventing the transfer of very large amounts of data to RAID volumes. At that time, Apple released an updated version of the Apple File System Reference, building expectations that third-party tools were just round the corner, at least among those who weren’t aware of how much information was still missing. Nearly four years later, it’s still that edition dated 22 June 2020 which remains the latest information released by Apple about APFS.
Further major changes came with Big Sur 11.0.1 when it was released in November 2020, introducing the sealed and signed snapshot now used to boot macOS. This was also the first release to support making Time Machine backups to APFS volumes, and to support Apple silicon Macs. Since then, macOS has continued to take advantage of the many new features in APFS. Apple’s ongoing failure to inform even developers of changes still catches us out: apparently Monterey added a feature to trim UDRW disk images, only discovered just over a year ago. Who knows what still remains hidden in APFS in Sonoma?
Free space or wear?
Perhaps the most common everyday annoyance with APFS is its apparent inability to give a clear estimate of free space available in a container. This is the downside to its complexity, making free space dynamic, so that much of it may be temporary and conditional. I and others have written repeatedly about our lack of confidence in any of the estimates of free space currently available in macOS, which is worst when it’s most critical, such as when trying to update macOS.
What has attracted almost no attention, though, is how APFS is designed to prolong the working life of SSDs by minimising the number of write/erase cycles. Although the user may be unable to realise the space efficiency achieved with the use of sparse and clone files, for instance, by reducing the number of storage blocks actually written in order to store files, with APFS each write/erase cycle should store unique data rather than needless copies or null data.
This is of little or no comfort to those condemned to use hard disks for storage, even if only for Time Machine backups. Although the great majority of APFS users’ devices will never be connected to a hard disk, for the small minority running it in macOS, they remain the medium of choice for backups and other bulk storage, and vulnerable to performance degradation resulting from fragmentation by the file system.
Third-party tools
Although Apple dropped early hints that APFS might be released as open source, after seven years information about its internals released by Apple still appears to be insufficient to allow third-party developers to create maintenance tools independent of those bundled in macOS. This isn’t just about the potential for someone outside Apple to improve on the performance of fsck or Disk Utility, but also concerns features that Apple shows no sign of supporting.
While Apple may be content with checking integrity only in the file system itself, many users also want to be assured that critical data doesn’t become corrupted or modified. In the absence of adequate documentation, third-party support for integrity checking remains relatively clumsy.
Another significant and lasting omission is the fundamental ability to copy Time Machine backups to a different storage device. Encouraged by reports, I recently attempted (again) to do this using Disk Utility’s Restore feature, only to be told repeatedly that my Time Machine backup storage volume wasn’t of the right type to be copied elsewhere.
Apple’s reluctance to document APFS internals may stem from the deep involvement between the file system and macOS security. If so, then it’s a strange choice that we should be deprived of better tools for safeguarding data on our Macs for the sake of their security by obscurity. Perhaps if APFS had proved less of a success, then we might have been allowed to know more of its secrets.
Version numbering
APFS version numbers are important, as they’re referred to in the file system and may be quoted in terms of which version created a given container or volume, and which last modified it.
Its major version numbers change with major version of macOS:
macOS 10.12 has APFS version 0.3 or 249.x.x
10.13 has 748.x.x
10.14 has 945.x.x
10.15 has 1412.x.x
11 has 1677.x.x
12 has 1933.x.x until 12.2, thereafter 1934.x.x.
13 has 2142.x.x.
14 has 2235.x.x. until 14.3, thereafter 2236.x.x
Minor version numbers increment according to the minor version of macOS, and patch numbers wander without pattern. Changing version numbers thus aren’t any indication of the scope or magnitude of changes made to APFS. As Apple seldom provides any information on changes made to APFS, it’s anyone’s guess as to what is going on.
