Spotlight search can be blocked by extended attributes

There are several well-known methods for excluding items from Spotlight search. This article details one that, as far as I can tell, has remained undocumented for the last 18 years since it was added in Mac OS X 10.5 Leopard, when Spotlight was only two years old, and can still catch you out by hiding files from local Spotlight search. This was discovered by Matt Godden, who has given his account here.

Well-known exclusions

There have been three general methods of excluding folders from Spotlight’s indexing and search, although only two of them still work reliably:

making the folder invisible to the Finder by prefixing a dot ‘.’ to its name;
appending the extension .noindex to the folder name (this earlier worked with .no_index instead);
putting an empty file named .metadata_never_index inside the folder; that no longer works in recent macOS.

Additionally, System Settings offers Spotlight Privacy settings in two sections. Search results won’t normally prevent indexing of those items, but does block them from appearing in search results. Spotlight’s indexing exclusion list is accessed from the Spotlight Privacy… button, where you can add items that you don’t want indexed at all.

Extended attribute

Matt Godden investigated repeated failure of Spotlight search to find some images in his large media library, and discovered that the extended attribute (xattr) named com.apple.metadata:kMDItemSupportFileType was responsible. Images that weren’t returned in a search of that library all had that xattr attached, and when that was removed, those images were found reliably.

According to Apple’s documentation, that xattr was available in Mac OS X 10.5 and has since been deprecated. No further information is given about its function or effect, nor does it appear in an older list of Spotlight metadata attribute keys.

Search of previous mentions of this xattr reveal that it has been found with either of two values, iPhotoPreservedOriginal as described for Matt’s images, and MDSystemFile used with several apps that have proved equally inaccessible to Spotlight search. Images that have this xattr attached appear to have originated in old iPhotos libraries, which may have been migrated to Photos libraries. Searches for files with this xattr suggest that even old collections of images seldom have the xattr present, in my case on only 9 files out of over 800,000 checked, and the MDSystemFile variant wasn’t found in over 100,000 application files.

The mere presence of this xattr is sufficient to exclude a file from Spotlight search: setting its value to the arbitrary word any, for example, was as effective as setting it to either iPhotoPreservedOriginal or MDSystemFile.

Strangely, the method used to search is important: files with the com.apple.metadata:kMDItemSupportFileType xattr can’t be found when using Local Spotlight search in a Find window, but can be found by Mints using a standard search predicate with NSMetadataQuery.

Detection and removal

The simplest way to detect whether your Mac has files with the com.apple.metadata:kMDItemSupportFileType xattr is to use the Crawler tool in my free xattred, with Full Disk Access. Open its window using the Open Crawler… command in the Window menu, paste the xattr name into the Xattr type box. Click on the Scan button and select the volume or folder to check. xattred then crawls the full directory tree within that and reports all files with that xattr.

The xattr can then be removed by dragging the file onto one of xattred’s main windows, selecting the xattr, and clicking on the Cut button. That change will be effective immediately, and the file made available through Spotlight search within a few seconds.

If you have more than a handful of files with the xattr, use xattred’s Stripper to remove them all. Paste the xattr name into the Xattr type box. Click on the Strip button and select the volume or folder to process.

Recommendations

If your Mac is likely to search old images that might have the com.apple.metadata:kMDItemSupportFileType xattr attached, search for and remove all such xattrs to ensure those files aren’t excluded from search.
Whether this behaviour is intentional or not, it’s clearly an undesirable legacy, has been deprecated for many years, and should be removed from Spotlight search.

I’m extremely grateful to Matt Godden for his painstaking research and keeping me informed.