Sonoma’s log gets briefer and more secretive

If you used Macs before Sierra, chances are that you were happy to browse their logs when necessary, for example to check whether your Time Machine backups were working correctly, or quietly throwing errors. At that time, the logs were traditional text-based records like those in other variants of Unix, and browsed using the Console app in /Applications/Utilities.

Traditional text logs in OS X 10.10 Yosemite before they changed in Sierra.

I started to become irritated by them as long ago as El Capitan, writing:
“Having excised most of the old dross and accumulated extensions, and running a fairly clean El Capitan system, my All Messages [in Console] clocks up around 4000 messages every 8 or 9 hours. At its worst – straight after the OS X 10.11.5 update, for example – it can fill those 4000 message slots in a minute or two, but even overnight when the system is not asleep it will seldom stretch to much longer than 12 hours. A traditional Unix wizard would have the screaming ab-dabs at this. We used to tweak, tune, and pare down systems until they barely coughed into their logs. It used to be the mark of a well-loved system.”

Little did we realise then that Sierra was going to change all that, and by Mojave we’d be enduring 4,000 and more log entries in a second, when our Macs were feeling loquacious. That was because Apple introduced the Unified log, with its entries written not in plain text but compressed binary format. This was the death-blow for the casual reader of logs: for a start, the replacement Console app was unable to access any log entries made in the past, and its tools were, and remain, woefully inadequate for tackling the increasing torrent of log entries.

Despite its many great strengths, the Unified log has suffered two problems that are limiting its usefulness in Sonoma: its diminishing period of coverage, and censorship.

Period of coverage

Rather than rolling its log files by their age, and keeping the last five days of logs, macOS maintains them by size. Like its predecessors, that means logd, its maintainer, only keeps the last 520 MB or so of regular Persist log entries collected at any time, and purges older entries. Some older log entries are retained, but it’s pot luck as to whether those you’re interested in are among them, and even if they are, other related entries may have been removed already. If you’re interested to see how this works, logd keeps records in the logdata.statistics.[n].txt files in /var/db/diagnostics, but those too are purged so you can only look back a few days.

In the early days of the Unified log, I was amazed to discover that it kept log records for as long as 20 days or more. In Sonoma, my main working Mac is struggling to keep them for as long as 24 hours. There’s no way for the user to increase that period to compensate for the increased rate of writing log entries by macOS. If I don’t catch an event well within 24 hours of it happening, chances are that its log entries will already have been purged when I go to look for them.

Privacy

Censorship to maintain privacy is an important feature in the Unified log, and is responsible for so many of its entries being peppered with <private> instead of revealing potentially sensitive data. The mechanism for this is robust: when collecting entries for the log, sensitive content is excluded, then displayed with that marker. As that sensitive content isn’t stored in the first place, there’s no way to unmask that content once it has been collected and written to the log.

Currently, the only way to see unmasked log entries is through a special profile. Once that has been enabled, and until it’s disabled again, all privacy should be removed from the log, and those <private> markers replaced with real data.

Only Apple has now decided that some log content is too private to reveal even when privacy has been removed. I came across this recently, in multiple log entries, when researching Web Apps, in entries like
ShareKit Start performing service <private> with items: <private>
ShareKit filteredItemsFromItems:<private> [-1]–> <private>
ShareKit Final shared items for <private>: <private>
ShareKit Perform custom action (<private> [<private>])
ShareKit Custom action (<private>) does not support async, running sync block

Of course, it may be possible to turn this additional privacy off, but the documentation leads us once again to a guessing game. Apple documents the Enable-Private-Data profile key here, where it refers to it controlling “private data logging”. But in its account on log privacy settings, Apple refers to two distinct privacy settings that can be applied to the content of log entries: the “standard option” ‘private’, and sensitive, described as “the option to always redact interpolated values that contain sensitive information.” Does this mean that setting Enable-Private-Data to false doesn’t unmask content designated sensitive, only that set as ‘private’?

Like so many great ideas, the Unified log is suffering from profligate overuse. Who really needs to see full dictionary dumps from RunningBoard? Is there no better way to record that information than in the log?