Finder Comments (Get Info) internals & bugs in macOS

How to get (and modify) file comments

If you use "Get Info" in macOS, you can edit a comment for the item (file or folder). If you copy the file elsewhere, the comment it copied along.

Now, programmers like me sometimes want to be able to inspect or even modify the comment. For instance, Find Any File lets you search for comments on any disk (volume), so it has to read these comments.

Unfortunately, macOS's operations do not offer a convenient way to access these comments. The safe way is to use AppleScript, especially if you want to alter the comment (see StackOverflow: How to read file comment field and StackOverflow: Upload Comments to File Metadata "Get Info" Mac Command Line).

Now, if you only want to read the comment, using AppleScript is relatively slow. For a program like Find Any File that would search entire disks, this could take hours when it has to query the comments via AppleScript for millions of files.

The other method other people have proposed is to read the comment from an extended attribute (EA), which is much faster.

FAF does this now, and it seems to work quite well.

Until someone contacted me and said that FAF manages to find most but not all comments. After a bit of investigation I found that the comment stored in the EA was different from the one that Finder shows in Get Info! And that's not good for finding comments fast and reliably, at all.

The customer was kind enough to help me investigate this and here's my findings:

How macOS stores file comments

macOS may store the comment for a disk item in these places:

1. In the EA attached to the file.
2. In the hidden .DS_Store file in the same directory as the file in question.
3. In the Spotlight database (this is optional)

It appears that Finder's Get Info, as well as its AppleScript's comment property accessor function, access the .DS_Store file directly to read or alter the comment, and if that happens, it also appears to update the EA alongside, while the Spotlight data gets updated whenever a file change is detected.

How can the comments get out of sync?

That happens due to an apparent bug in Finder or its services. Here's how to reproduce it:

1. Set a comment for a file, using Get Info in Finder, e.g. set it to "original".
2. Copy the file to a different location, e.g. into a folder next to it, keeping the original file name.
3. Modify the comment of the copied file, e.g. to "changed".
4. Copy the copied file back to the original location, choosing to replace the existing.
5. Now check the comment in Finder's Get Info: It'll show "changed", which is the intended result.
6. Now check its EA. In Terminal, use:
   xattr -px com.apple.metadata:kMDItemFinderComment /path/to/original_file | xxd -r -p | plutil -p -. This may show either an empty string or the old comment "original", but should instead be "changed".
7. Even worse, Spotlight's comment may also be wrong. Check with:
   mdls /path/to/original_file | grep kMDItemFinderComment The Spotlight comment may automatically update later, though, e.g. when you open the file or modify it.

(BTW, I suspect that this can also happening when you copy the file in Terminal or other methods that do not involve Finder, though then the result would be the other way around: The EA might be copied to the replaced file location, whereas the .DS_Store file won't get updated.)

Later I found an even easier way to mess up the comment:

After setting the comment with Get Info, rename the file. Check again with Get Info, and you'll see that it still has the comment, but if you now look at the EA, it'll show an empty comment.

So, there appears to be several bugs in Finder's copy operation (I've verified this in both macOS 10.13.6 and 15.4, so it's been around for a while):

Two kind of bugs

1. Replacing or renaming a file should transfer the EAs but doesn't. Maybe Apple has reasons to keep some EAs of the replaced file, but the comment surely isn't one of them - the comment belongs to the file being copied.
2. Copying or renaming a file that replaces another should also trigger an immediate update to the Spotlight importer, making sure it records the new comment. Or maybe Finder even does trigger the importer, but then there's a race condition bug that makes the importer read the outdated or yet non-existing comment in the .DS_Store, with Finder being too slow to update the comment in the .DS_Store in time.

That's my findings so far. I'll file a bug report with Apple but have little hope that this will get addressed, as I've file related bugs in this area before and nothing happened.

What does this mean for apps that search for file comments?

The only reliable way to get a file's comment is to use the slow AppleScript method right now (or read directly from .DS_Store, but that's undocumented and may break any time, because the file's format is private to Apple). Which means I might have to update Find Any File to use the much slower AppleScript method.

But for now, I'm working on a "matching script" for Find Any File that can identify and fix these out-of-sync comments. If one runs this script once on all volumes (which may take a while), the EAs would be up-to-date and FAF could search them quickly.

"Failed to Enable Personal Hotspot on iPhone" – finally, a solution

Some of my older Macs running High Sierra or Mojave were unable to join the Personal Hotspot of one or even two of my iPhones.

I searched the Apple "Community" Support forums and found several related questions, with lots of "I have the same issue" tags, but with no helpful answer:

• https://discussions.apple.com/thread/8153026
• https://discussions.apple.com/thread/8337291
• https://discussions.apple.com/thread/254471591

So I did figure this out on my own.

If you get this error when trying to connect from your Mac to the Personal Hotspot of your iPhone, try the following for a fix:

 There is a file called "com.apple.airport.preferences.plist" in the folder "/Library/Preferences/SystemConfiguration".

 If you cannot locate it directly, use a tool to find it, e.g. with my program Find Any File (no need to pay for it if you only use it once - just download it).

Then make a back-up of the file, in case you get it wrong below! For instance, copy the file to your Desktop.

Then you have two options:

1. Delete this file. But that will also reset all your network interfaces, e.g. if you had set up a LAN connection with a fixed IP address or made other configurations in the System Preferences' Network panel. And after deleting it, you need to restart the Mac so that it sets up the network interfaces again. Then try to connect to the phone again.
2. If you're more adept, edit the file, e.g. with the app PrefEdit or BBEdit, then find all entries that mention "phone", and remove those. After saving the changes, you should be able to reconnect to the phone without even requiring a restart.

If something went wrong, copy the backed-up file back into the folder (you'll have to enter your admin password).

Good luck!

Locating and updating symlinks and Finder Aliases with FAF

Today I renamed one of my internal disks in my Mac Pro. I then realized that I had created a few symlinks to that volume, and those would now become invalid.

For example, if the disk used to be called "Data" and is now called "Backups", then symlinks I may have created would still point to "/Volumes/Data/..." but need now point to the new name instead.

Since I knew that there would only be a handful of such symlink files on my other disks, I could easily update them by hand (using Terminal.app, with the "ln -s" command).

All I needed to do was to find all those symlinks first, making sure I would not miss any.

With Find Any File, this is quite easy. Set up a search like this:

To get the "File Type Code" option, you need to hold down the option (alt) key before clicking the popup-menu. Searching for files of type code "slnk" will address symlinks, and nothing else.

This will then find all matching symlinks, which you can then reveal in the Finder and manually update accordingly.

Similarly, you can also find related Finder Aliases, by searching like this:

After renaming a disk, updating Finder Aliases pointing to that disk is usually not necessary, because Aliases use redundant information to locate moved and renamed files.

However, if you should ever copy all your content to a new (larger) disk, file by file, Finder Aliases won't work any more if the targeted files have also been moved or their disk has been renamed.

So, it can't hurt to update your Aliases right away after moving or renaming the target item. To update your aliases, simply locate or reveal them in Finder, then select the Alias file and hit cmd+R to have it reveal its target. Should the target have been moved or renamed in the mean time, macOS will automatically update all the redundant information.

APFS and fast catalog search

This is about FSCatalogSearch / searchfs support in macOS with the APFS file system.

Updated 3 Oct 2017: Find Any File 1.9 will support fast search on APFS on High Sierra (10.13) by using the searchfs function. Version 1.9 is currently in open beta, see the FAF web site.

Updated 25 July 2017: Clarified why FSCatalogSearch doesn't work on APFS, adds issue about hard links and 64 bit CNID resolving.

Some background on FSCatalogSearch in general

Programs like EasyFind and my own Find Any File (FAF) are able to search for file names (as well as file dates, sizes and a few other rarely needed attributes) on disks in a quite fast manner by using a little-known function macOS offers.

This Carbon level function is known as CatSearch or (FS)CatalogSearch and has been around for more than 25 years. There's also a BSD level function called searchfs.

The advantage of this function is that it performs the search for names at the file system driver level, meaning that when you search for files containing ".png" in their name, the file system can look at the entire directory tree much faster, sorting out the matches, and only report those to the program that initiazes the search and then shows the results to the user.

Without this special function, the search program would have to start at the root of the disk, read each folder (directory) recursively, and then sort out the matches itself, which all takes much more computing time.

For example, a search on a disk with millions of files and folders on it would take only a few seconds with FSCatalogSearch, whereas a classic recursive search would take minutes.

Getting even more technical

Apple added the FSCatalogSearch function in Mac OS long ago, after introducing the HFS file system. This was supported by the fact that  HFS did, unlike Window's FAT, arrange the entire directory tree in one large file on the disk, with interlinked nodes that did not match the hierarchical folder structure. FSCatalogSearch would then iterate over the nodes in a most efficient way, not caring about the folder structure, thereby minimizing disk seek times, which was a significant factor in disk access before SSDs. This also meant that FSCatalogSearch would only work on volume formats that used a single (invisible) file for its entire directory tree, meaning that FSCatalogSearch was never available for FAT disks, for instance. It would also be optimal for NTFS volumes, but since Apple never used NTFS other than to support reading from Bootcamp partitions, they never made the effort to add FSCatalogSearch to their NTFS file system driver.

What about APFS?

Now Apple is about to replace HFS(+) with APFS on macOS. And fans of EasyFind and FAF start wondering: Will I still be able to perform fast disk-wide file name searches the way I'm used to?

The good news is: The APFS file system code has support for the lower level searchfs function, and that's been already added in 2016, apparently, for OS X 10.12. Which ultimately means: Yes, FAF and EasyFind can continue to provide fast search on APFS formatted disks, provided extra work is put into updating the apps accordingly.

However, there are still some issues:

• The high-level FSCatalogSearch does not work on APFS. Both EasyFind and FAF rely on this function and therefore won't find files the fast way on APFS volumes right now. The reason for this is that APFS uses larger values (64 bit) than HFS+ for identifying the files, and the FSCatalogSearch function cannot handle those larger values. (rdar://33454922)
• As of now (10.12.6, 10.13 beta 3), the searchfs function does search case-sensitive and not case-insensitive as it should. That means that searching for ".png" won't find files using ".PNG". I confirmed this with an Apple engineer - it's a known issue, just one with a low priority right now. So, there's a chance that this will get resolved eventually, and I hope it'll be done before 10.13 is released. This issue may not get fixed for 10.12.x, though. We'll have to see what Apple does in this regard. (rdar://33455597)
• Hard links can't be identified correctly - if there are multiple hard links to the same file, then searchfs can't currently tell them apart, and the results will all point to the same directory entry. (rdar://33473247)
• searchfs() returns CNIDs (Catalog Node IDs, 64 bit wide) instead of paths to the found items. This requires resolving these IDs to the paths later. However, there is currently no documented API provided in macOS to do so. There's a hackish way around this, but that's not a proper solution. (rdar://33507188)

What this all means

Current versions of FAF and EasyFind can't fast search on APFS. They need to be rewritten using the searchfs API.

I will be working on a quick-fix version of FAF that'll add fast search on APFS and which I hope to release before 10.13 (High Sierra) is officially released. I have quite a few other improvements for FAF in the works (64 bit app, content search, icon view, server support etc.) which will have to wait so that I can get this APFS issue resolved ASAP.

Recover lost BootCamp Windows partitions on a Mac

I have installed several Windows 7 and 10 version on several of my Macs using Apple's Boot Camp feature.

Recently, I found that almost all of them have disappeared: I was not able to boot from them any more when I held down the option (alt, ⌥) key at startup - the Windows partitions would either not appear at all or not boot up.

The main reason in my case was that the MBR was reset to a plain GUID entry, and my Windows versions do not like that, because they cannot handle the EFI / GUID partition info that the Mac prefers. Why that even happened? Probably from repartitioning operations I frequently perform on my disks - and Apple's Disk Utility is quite ignorant of the needs to keep Windows bootable in this regard.

To fix that, the MBR partition info needed to get fixed, in order to make the Windows NTFS visible again to the Windows boot system.

The easy way

After writing this article initially, I learned that there is a Mac program, WinClone,  that can fix this issue with a few clicks, in case you do not want to mess with it yourself as shown below.

The hard way

I used iBored to edit the partition layout of each disk that contains a BootCamp partition from something like this:

Into this:

Note that this reduces the size of the first partition (you could as well change its size to the minimum, which is 33), and adds a new partition with the start and size matching what you can inquire using the Partitions window (see Disk menu):

This modification makes the Windows partition available in the MBR, and after that, I can boot again from it. And it won't mess with macOS booting because that uses the GUID partition info which isn't getting modified by this procedure (for more info, read my older article on using BootCamp on a non-startup disk).

Using USB barcode scanners on Macs with Xojo

Barcode scanners translate 1D and 2D bar codes, such as Code128 and QR Code, into readable characters.

Usually, handheld bar code scanners with a USB interface simulate a keyboard by default. With that, they work out of the box: If you scan a code with such a reader, it'll send keyboard strokes to the computer.

This makes them easy to use, but it also causes some issues:

• If you have a program that expects input from a scanner, you have to make sure the typed characters are properly caught by your program and not go somewhere they don't belong. E.g, if the user switches accidentally to another program and then scans a bar code, not only will your program not be able to get the code from the scanner, but the keystrokes may cause odd effects in the the program that's active then. It's like your cat walking over your keyboard and wonder what'll go wrong.
• Even worse, it can pose a security threat. Hackers have been able to show scanners customized bar codes that contained key strokes that would lead to remote controlling computers that the user was not supposed to access that way. Imagine a so-called "kiosk" terminal that's supposed to run only a particular application but gets quit by a alt-F4 or cmd-Q keystroke, then opening another program such as a Terminal to mess with the computer. In other words, using a bar code scanner in the "keyboard emulation" mode opens the computer to all kinds of abuse that you may have wanted to prevent by not giving the user a keyboard and mouse.

One way to prevent these issues is to put the scanner into a mode where it does not emulate a keyboard any more. That mode is usually called a "HID POS" or "USB OEM" mode. In such a mode,  the software needs to run special code to communicate with the scanner directly. Another option would be to not use the scanner's USB interface but rather use a serial interface (many scanners can be used with either) and then use a Serial-to-USB adapter to connect the serially operated scanner to the Mac or PC.

In case you want to use the USB HID POS mode with a scanner (e.g. GoDEX models), I've posted some details and a demo project on the Xojo forum a while ago: https://forum.xojo.com/24557

In the meantime, I've also managed to scanners from Datalogic (Gryphon 44xx models) and similar scanners using the sparely documented IBM / Toshiba "OEM" USB protocol. If you are interested in supporting those, contact me directly.

iTunes 11 modal alert dialog overflow

Do you tend to show modal dialogs to show issues to the user when processing his data? Well, think twice, or you might end up upsetting your users like Apple manages with iTunes 11.

See here for a screen recording I created when trying to update my iOS apps after about a week from the last update.

Blogger was supposed to show a video box here,

but that doesn't seem to work.

Instead, watch the video here (Youtube)

Or here (Quicktime)

Admittedly I am suffering from iTunes 11 more than most users because I have two iTunes accounts: A German and a U.S. account. And I have downloaded iOS apps from both accounts. This leads to iTunes wanting to update all apps for both accounts at the same time, even if it only shows me the German updates.

However, before iTunes 11, this was still manageable. iTunes 11 made this much worse.

Also, there are other alerts in there that even the usual Apple customers with just one iTunes account might experience, such as the late-appearing "age restriction" notice that pauses downloading, and messages about problems with particular apps without naming those apps.

Overall, iTunes 11 does this all plain terribly. Apple used to excel at good user interfaces. Nowadays, I get the impression they take the kids right from college, let them work on these programs with no supervision from anyone who still has a clue about how to avoid such bad user experiences.

(I've also reported this to Apple via their bugreporter, see http://www.openradar.me/15350892)