Watch your background: background activities with DAS-CTS
Much of what macOS does happens in the background. You don’t have to refresh the Spotlight indexes when you edit and change documents, nor should you have to run backups manually, or flush caches. Those are all tasks that are intended to be performed by services within macOS, in the background, out of sight and out of mind. They are also tasks that can go wrong or cause problems, so it’s valuable to understand how they work. This article is the first of a small series looking at how background tasks are run, and what can go wrong with them.
In broad terms, tasks that run in the background fall into three types:
Background services, that are launched during startup, appear in the list of processes in Activity Monitor, and work on demand. Examples include cfprefsd to manage preferences and mds_stores to manage Spotlight’s indexes. Activity Monitor normally lists a total of around 500.
Background activities, that are typically run at flexible time intervals according their need. Examples include Time Machine backups and XProtect Remediator scans. There’s normally a total of over 500.
Background threads, that are created and controlled within an app, and are counted in the thread totals given in Activity Monitor. Most apps now use these to take advantage of the multiple CPU cores in Macs. There’s an average of around 3-5 threads per process.
Of those, I’m going to start by looking at background activities, as they’re least-known and barely documented, although some provide the most important services to the user.
Background activities
If you open Activity Monitor shortly after you’ve logged into your Mac and watch the processes taking up most % CPU and in the CPU History window, you’ll see a succession that become very active for a while, then disappear. Although some are background services running all the time, many stay for a while, do their work, then vanish: most are background activities, and in Apple silicon Macs usually run on the E cores alone.
Most background activities are run because they’re set up using property lists stored in the LaunchAgents or LaunchDaemons folders in one of the Library folders, /System/Library, /Library or ~/Library. Daemons are run as root (user 0), while Agents are run as a user (normally the current user, 501). One notable feature that you’ll see in their property lists is the key-value pair
<key>ProcessType</key>
<string>Background</string>
Apple explains this in man launchd.plist:
This optional key describes, at a high level, the intended purpose of the job. The system will apply resource limits based on what kind of job it is. If left unspecified, the system will apply light resource limits to the job, throttling its CPU usage and I/O bandwidth. This classification is preferable to using the HardResourceLimits, SoftResourceLimits and Nice keys. The following are valid values:
Background
Background jobs are generally processes that do work that was not directly requested by the user. The resource limits applied to Background jobs are intended to prevent them from disrupting the user experience.
I’ll return to those resource limits in a later article.
In-app background activity scheduling
Most background activities should be set up using a LaunchAgent or LaunchDaemon property list, but code in an app or command tool can also create a background activity using the NSBackgroundActivityScheduler API. Apple suggests those might be appropriate for actions such as automatic saves, backups, periodic content fetching, and installation of updates. I’m not aware of any apps that use that, other than my own experimental utility DispatchRider, which I’m in the process of updating. However, if you know, I’d be very grateful to hear of them.
In Swift, this might be implemented in outline as:
// create the activity
let activity = NSBackgroundActivityScheduler(identifier: “com.orgname.appname.tasks”)
// set it up
activity.repeats = true
activity.interval = TimeInterval( … )
activity.qualityOfService = QualityOfService( … )
activity.tolerance = TimeInterval( … )
// schedule it
activity.schedule() { (completion: NSBackgroundActivityScheduler.CompletionHandler) in
// activity code
completion(NSBackgroundActivityScheduler.Result.finished) }
// stop the activity
activity.invalidate()
Flexible scheduling and dispatch
Normally, when a LaunchAgent or LaunchDaemon is set to run at specified time intervals, launchd will run it at exactly that set time, as long as the Mac is running and awake. When Time Machine was introduced, it backed up using that mechanism, every hour at exactly the same minutes and seconds, as regular as clockwork. If you happened to be in the middle of demanding tasks, then those backups competed with your use of the Mac.
Seven years later, Time Machine was one of the first clients of a new flexible scheduling and dispatch system operated jointly by Duet Activity Scheduler (DAS) and Centralised Task Scheduling (CTS), using lightweight inter-process communication (XPC), DAS-CTS for short.
Background activities are either registered through their property list, or using NSBackgroundActivityScheduler, with XPC Activity (com.apple.xpc.activity). Activities are submitted with their time interval, a tolerance interval around that, and Quality of Service (QoS) to determine how aggressively they will be scheduled. From those, DAS assigns them a priority (different from their QoS), a time window, and an optimal score, used to determine when that activity should be performed.
The activity is then added to the list maintained by DAS. At frequent intervals, DAS checks through its list of activities, rescores each, and decides whether to dispatch that activity via CTS. Various policies are used to determine whether an activity should proceed. One simple example applies to many activities, which aren’t to be run until at least five minutes have elapsed since the most recent boot. DAS simply compares the Minimum seconds after boot required against observed time elapsed.
In some circumstances, DAS checks through the list of running activities to see whether one of those is incompatible with the activity it’s intending to approve for dispatch. That ensures, for example, that two activities competing for the same resource can’t be run together. As Apple promises, “the system can intelligently decide when to perform the task based on the specified criteria,” which also take into account environmental factors such as thermal pressure.
DAS thus arrives at a decision as to whether to proceed and dispatch an activity, recorded as either MNP (must not proceed) or CP (can proceed). In the latter case, it then tells CTS to run the activity, which proceeds via XPC, although XPC isn’t required to be used by the activity. On completion, an activity intended to be repeated is updated in the list maintained by DAS, ready for it to be dispatched for its next run.
Diagnosis and treatment
Since teething problems in macOS Sierra were fixed in High Sierra, the DAS-CTS system has been remarkably reliable, and run successfully for long periods between reboots. That’s just as well, as it has no management utility, and can only be observed through its entries in the log. To see it at work, filter log entries with a predicate for the subsystems com.apple.duetactivityscheduler and com.apple.xpc.activity. This is available as one of the standard log viewers in Mints.
DAS-CTS can only be reset/restarted by restarting the Mac. As we learned during its problems in Sierra, trying anything else is a waste of time.
Summary
Background activities:
are one of three common methods used to run tasks in the background, and used for over 500 activities in macOS Sequoia;
are typically run at flexible time intervals according their need, as in Time Machine backups and XProtect Remediator scans;
are most commonly created from property lists in LaunchAgents or LaunchDaemons folders, where the ProcessType key is set to Background;
can also be created in code using the NSBackgroundActivityScheduler API;
are managed by Duet Activity Scheduler (DAS) and Centralised Task Scheduling (CTS), using XPC, in the DAS-CTS system;
are registered with DAS when created, and added to its list of scheduled activities that are rescored frequently;
when DAS determines an activity can proceed, it tells CTS to run the activity, and it’s updated in the list ready for dispatch next time;
problems can only be diagnosed using the log, and the only way to reset DAS-CTS is to restart.
