I’ve spent some time exploring how best to take advantage of OS X 10.4’s metadata goodness, familiarizing myself with the new ways in which Spotlight lets you organize files.

I’ve long looked forward to finally being able to use all file attributes to organize my files, rather than just the location attribute. Previously, both in Windows and Mac OS, you could do searches on a limited set of such attributes, like “date created” or “file size”In the Mac OS, there was even a highly rudimentary user-adjustable file attribute you could search with: the color label.. But you could never turn such searches into a permanent filtered window, dynamically showing all the files on your system whose attributes satisfy your criteria.

Mac OS X 10.4 lets you do precisely that, using “Smart Folders”. Now, a file can show up wherever and whenever it is relevant, as determined by its metadata. Apple has made it even better by greatly increasing the different kinds of attributes a file can possess, and also by indexing text-based files, in effect turning content into a (giant) attributeUntil now, I had to resort to BBEdit’s ability to search through the content of multiple files to do something remotely resembling this..

But alas, Spotlight has its limits. Two in particular are identified by John Sircusa in his scarily thorough and much linked-to Ars Technica review of OS X 10.4:

1) Just as I have encountered with the smart folders in iTunes and RSS readers, Spotlight’s user interface severely limits the kind of search query it lets you make. Specifically, all individual queries can only be strung together via logical ANDs. No nests, ORs, or NOTs.

2) Although the underlying BSD operating system now supports it, there is not yet any higher level support for creating your own arbitrary searchable attribute=value pairs, which is precisely what I had been hoping to do to my data — for example by marking up a file thus: “client=si, projectname=designåret, language=english, duedate=20050530, function=invoice”. I would then have been able to group together files in smart folders by client, as well as by projects that are due within a week, and by invoices for the year, come tax time. No longer would I have to choose only one of those methods of viewing my data. I could have it all. But just not yet, it would seem.

But there are workarounds.

Re 1): It turns out that while the Spotlight user interface doesn’t help you structure complex queries, it does give you a way to perform raw queries, just as you would at the command line. While these aren’t pretty, they certainly let you construct as complex a query as you care to make. Restiffbard explains how exactly. There is a picture at the end of the post making it all a bit clearer.

Re 2): Several blogs have noticed that there is a new attribute called “Spotlight Comments” that is visible if you “Get Info” (Command-I) on a file. This attribute provides a text field that is usable in Spotlight searches — for example, you can group together all files whose Spotlight comments contain the word “Polish”.

Spotlight comments has now been mooted by blogger Russell Beattie as a great way of attaching keywords, or “tags”, to files, perhaps because tags are all the rage on the internets right now.

I happen to think that tags are way too shoddy to trust. How would you know if “polish” refers to the people, the language, your shoes or the need for an edit? Smart folders are not so smart that they can tell the difference. This is why it is so important to have attribute=value pairs, making sure that names of attributes are unique.

The “folksonomies” of tags that are used by Technorati and Flickr don’t escape this problem. They too suffer from contextual confusion, especially of the linguistic kind — for example, Swedes will tag pictures of children with “barn”, leaving it to Americans to wonder why their searches for images of a pictoresque barn brings up blond children. The idea is that folksonomies are good enough, and perhaps on the web they are, trading accuracy for spontaneity, but for my own files I need not make such compromises.

Therefore, I think we should go one better than tags and use Spotlight comments as an adhoc repository for our own arbitrary attribute=value pairs. To minimize the potential for contextual confusion, mine are going to look like this:

:client=si: :projectname=designåret:

:language=english: :duedate=20050530:

:function=invoice:

It is far easier to add such metadata than it might look, thanks to the new Automator workflows, which can batch-process files. It’s a trivial matter to create a workflow [.zip] that appends text to the Spotlight comments of selected files and folders (and even nested files and folders [.zip]), as several blogs have done. Just open these workflows in Automator, then save them as Finder plug-ins. To use, select files and/or folders, right-click, and select your workflow of choice. A dialogue then prompts you for the text to append.

It goes without saying that this is a rather rudimentary workaround. For starters, there is no trivial way to batch-change or batch-delete arbitrary text from Spotlight comments (that I have seen — Automator certainly doesn’t provide it), for example by peforming a search and replace on the Spotlight comments of a selection of files. It should be fairly easy to do this in AppleScript, though, given sufficient motivation. It’s a problem that those who plan to use tags face as well.

Eventually, either Apple or a third party will implement OS X 10.4’s built-in support for user-defined attributes, making this ad hoc solution thoroughly obsolete. But when this does happen, it should be trivial to switch to the new and proper menthod, because all you will need to do is select all the items in a smart folder, which already match your predefined criteria, and batch-process them to generate the proper attribute=value pairs.

In the meantime, there is an opportunity here for the right company to create the Conflict Catcher of OS XContext for recent switchers: Conflict Catcher was an indispensable utility for OS 9.. Right now, I wish I were a proper programmer.