Why can’t Spotlight find files in Library folders?

You probably don’t use Spotlight to search files likely to be in one of your Mac’s Library folders, but if you do you may have noticed that it hardly ever finds anything there. I’m grateful to Scott for pointing out that some folders inside those Library folders, like Preferences, can’t be searched at all. This article attempts to discover what’s going on.

How Spotlight finds

When files are created or changed, provided they’re in a location that it indexes, Spotlight’s mdworker processes index data about those files, including metadata about each file itself, such as its name, and about its contents, enabling you to search for terms found in the file’s data. If a Spotlight search is unable subsequently to find known contents of a file, then one of the following has occurred:

The contents of that file weren’t correctly indexed, most commonly because that file is in a location excluded from indexing, as set in Search Privacy… in Spotlight settings.
Although indexes for that volume contain data correct for the file, Spotlight is withholding information about it from search results, most commonly because that category of file (or folder) shouldn’t provide search results, according to Search results settings in Spotlight settings.

I’ll refer to these two causes as unindexed and withheld respectively.

Does Spotlight reindex files that have been moved?

Before trying to establish which folders Spotlight won’t return search results for, I needed to be able to distinguish between those two causes. One way to achieve that might be to ensure that test files were first indexed at a location that is reliably indexed, for which no search results are withheld, then move the test files to a different location. That relies on Spotlight retaining indexed data when files are moved, rather than reindexing them.

I therefore assembled a folder of test files, based on those created and used by Mints, with the addition of a property list containing the search term syzygy999. These were first placed at the top of the user’s Home folder, time allowed for Spotlight to add them to the volume’s indexes, and that was confirmed by performing Mints’ test search, which found them all. That folder was then moved to ~/Documents, with Mints given Full Disk Access to ensure search wouldn’t be blocked by TCC.

Inspection of the log for 30 seconds following the change of location demonstrated that no reindexing occurred, although Spotlight was informed of the changed paths for the files. Thus, any search failures are expected to result from results being withheld, rather than the files being unindexed.

Which folders don’t return expected search results?

I then moved the test folder from ~/Documents to other folders and repeated Mints’ test search for each. Results were:

~/Library and /Library – search only found the term in a plain text file, where it was attached as an extended attribute and not in the text data.
~/Library/Trial – only found in plain text file with the search term in xattr.
~/Library/Preferences – no searches returned successfully.
~/Library/Application Support – all test files were found successfully, including the property list.

Is this specific to the active boot volume group?

All those test were performed on the active Data volume. To test whether this extends to other (inactive) boot volume groups, I connected an external bootable SSD with Sonoma installed and repeated the tests. As Mints only searches the current Data volume, I used the Find feature in the Finder. Results were exactly the same for that Data volume’s single user.

Where is this behaviour controlled?

Within each volume’s Spotlight index folder is VolumeConfiguration.plist, whose name suggests it might contain Spotlight configuration for that volume. Comparison of that file from a regular APFS volume with that from the Data volume of a boot volume group failed to suggest any value that might be responsible for this behaviour, though.

That indicates it’s determined in another property list, or coded into Spotlight.

If you have purchased the Spotlight master tool HoudahSpot, which I strongly recommend, then this may seem familiar, as it’s mentioned in its Help file, suggesting that it’s not new to Sequoia.

Conclusions

Spotlight in Sequoia (at least) withholds the results of searches in Library folders and their contents, except for those in /Library/Application Support.
There’s no way apparent to change that behaviour.

Postscript on dealing with prolonged indexing

At the end of my testing, I tried to unmount the external SSD containing the bootable Sonoma installation, only to be told that it was still in use. This was the inevitable reindexing that has caused similar problems to many others. Rather than forcing ejection in the Finder, I left this to run, in the hope that it would complete reasonably swiftly on a Mac mini M4 Pro. I gave up after over three hours, forced the external Data volume to be unmounted, and disconnected the SSD.

Intensive reindexing activity continued to occupy all the E cores, as if the volume was still present. This was occurring in the /var directory rather than in any volume indexes, and was only terminated by restarting the Mac. At no time did Spotlight indicate that indexing was in progress, nor provide any indication of progress to completion. If this isn’t a bug in Sequoia, then it’s a serious flaw in Spotlight that needs to be addressed. I will see if I can repeat this, and then file a bug report if I can.