Cloning APFS Volumes & Containers ("APFS inverter failed")

(This is an older article that I hadn't published back then because it might not be fully accurate, i.e. the steps may not be applicable. Yet, it contains some valuable information so I decided to publish it now. Read with a grain of salt. If you have corrections, don't hesitate to comment or email me.)

TL;DR

If Disk Utility fails to clone an APFS container, giving "APFS inverter failed" as error message, a work-around solution is to copy the partition with the "dd" command into a same-sized partition, then fix the cloned container's UUIDs.

About cloning (copying) entire Mac volumes

Usually, when you want to clone a Mac volume, you'd use Disk Utility's Restore operation. It performs a sector-by-sector copy (while skipping unused sector . The alternative would be to perform a file-by-file copy, as it's done by 3rd party tools like Carbon Copy Cloner.

A clone operation by copying individual files has several disadvantages over a sector copy:

• It's much slower.
• If you're using Finder Alias files, these may not work any more afterwards (that's because they rely on each file's unique ID, and those IDs change when copying files over to the destination).
• More programs may request re-activation (re-registration).

However, when I recently tried to make an identical copy of my macOS Mojave system from my MacBook Air (2015), copying it to an external SSD, I ran into problems:

After the copy and verification operation was already apparently finished, an additional inverter process needs to be run on cloned APFS volumes. While I do understand that that's necessary when I copy only a single volume out of an APFS container, or copy the volume into a target APFS container without replacing it entirely, but even if I try to clone the entire container, with erasing the target, it still wants to run an inversion process - and that makes no sense to me.

Now, the error that I kept seeing is: APFS inverter failed to invert the volume

And I'm not the only one, see here and here.

I tried many things, including First Aid and removal of all snapshots, and running the cmdline tool "asr", which showed me more detailed error messages. Still, no success. I keep getting variations of the same issue.

What I'm showing now is a way to clone a complete APFS container (with all contained volumes) the way it should work.

How to clone an APFS container

Note: This may not work with encrypted volumes. Or it might. I have not tried.

We simply copy the entire partition (which contains the APFS container) sector by sector. (Small disadvantage over Disk Utility's Restore operation: This will also copy unused sectors, so it'll take a bit longer.)

Afterwards, we need to change the UUIDs of the cloned container and its volumes, or the Mac (and especially Disk Utility) may get confused when both the original and the cloned volumes are present on the same Mac.

Perform the sector copy operation

In case you want to copy your bootable macOS system, you will have to first start up from a different macOS system. If you have no other external disk or partition with another macOS system, simply start up from the Recover system.

When ready, connect the target disk, then start Terminal.app.

Get an overview of our disk names by entering (and always pressing Return afterwards):

diskutil list

Here's a sample output of my Mac that has four partitions, two of which use HFS+ ("AirElCap", "AirData") and the other two use APFS ("AirMojave", "AirHighSierra"):

/dev/disk0 (internal):

   #:                       TYPE NAME                    SIZE       IDENTIFIER

   0:      GUID_partition_scheme                         121.3 GB   disk0

   1:                        EFI EFI                     314.6 MB   disk0s1

   2:                 Apple_APFS Container disk1         40.9 GB    disk0s2

   3:                  Apple_HFS AirElCap                20.1 GB    disk0s3

   4:                 Apple_Boot Boot OS X               134.2 MB   disk0s4

   5:                 Apple_APFS Container disk2         39.8 GB    disk0s5

   6:                  Apple_HFS AirData                 19.9 GB    disk0s6

/dev/disk1 (synthesized):

   #:                       TYPE NAME                    SIZE       IDENTIFIER

   0:      APFS Container Scheme -                      +40.9 GB    disk1

                                 Physical Store disk0s2

   1:                APFS Volume AirMojave               20.7 GB    disk1s1

   2:                APFS Volume Preboot                 47.1 MB    disk1s2

   3:                APFS Volume Recovery                512.7 MB   disk1s3

   4:                APFS Volume VM                      2.1 GB     disk1s4

/dev/disk2 (synthesized):

   #:                       TYPE NAME                    SIZE       IDENTIFIER

   0:      APFS Container Scheme -                      +39.8 GB    disk2

                                 Physical Store disk0s5

   1:                APFS Volume AirHighSierra           23.2 GB    disk2s1

   2:                APFS Volume Preboot                 20.9 MB    disk2s2

   3:                APFS Volume Recovery                519.0 MB   disk2s3

   4:                APFS Volume VM                      3.2 GB     disk2s4

I plan to clone "AirMojave", so the disk identifier would be disk1, because that is the container for that volume. In your case it may be a different identifier. I will write the commands below using diskS for the source container and diskTsP for the target partition.

Now unmount all volumes of our source container. In Terminal, enter:

diskutil unmountDisk diskS

If this does not succeed, then you either have programs or files open on one of the volumes or you're trying to unmount the boot volume (in that case, you should have restarted into a different system as explained above). Do not continue if the mount was not successful or you're likely to end up with a corrupted clone.

diskutil apfs list diskS

+-- Container disk2 5099518D-36C0-47CC-B034-0CDA52C51CE8

    ====================================================

    APFS Container Reference:     disk2

    Size (Capacity Ceiling):      39838416896 B (39.8 GB)

    Capacity In Use By Volumes:   27042205696 B (27.0 GB)

    Capacity Not Allocated:       12796211200 B (12.8 GB)

sudo diskutil resizeVolume diskTsP size

In place of size, use the number that's show after "Size (Capacity Ceiling):".

diskutil unmount diskTsP

This is the command that will perform the copying:

sudo dd bs=1m if=/dev/diskS of=/dev/diskTsP

(See also Adrian's comment below, suggesting to use "rdisk" instead of "disk" for higher speed.)

While dd is doing its work, it won't show any progress on its own. And it can take hours, even days. To check where it's at, type ctrl-T in the Terminal window. That will print a line showing its current progress.

Fix the UUIDs

Enter in Terminal:

sudo /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -s /dev/diskT

You'll likely not get any response from this command, telling you success or failure.

To check whether the UUIDs have been changed successfully, enter in Terminal:

diskutil apfs list

In the output, find your source and target disks, and compare the UUIDs of their container and container volumes. They should all be different. If the UUIDs of source and target volumes show the same code, then the fix did not work. Try again.

Make the system volume bootable

The last step is to make the cloned volume bootable again (in case it was before). For that, you need to mount the Preboot volume, then rename the single folder inside, which contains the old UUID, into the new UUID of the main volume.

To mount it:

diskutil mount diskTsX

(X stands for the partition number containing the "Preboot" volume)

To show it in Finder:

open /Volumes/Preboot

Now you should see a Finder window with a folder named after the UUID of the original volume you copied. Rename that folder into the new UUID. To verify that you used the correct UUID, open System Preferences, Startup Disk, select the cloned bootable volume and click the Restart button. If it says that it can't bless it, you got it wrong. Otherwise, the system should now restart, meaning the UUID was correctly set to a bootable volume.

Performance considerations when reading directories on macOS

(Latest update: 17 Feb 2025 (2), see end of text)

I'm developing (and selling) a fairly popular file search program for the Mac called Find Any File, or just FAF.

It works differently from Spotlight, the Mac's primary search tool, in that it always scans the live file system instead of using a database. This makes it somewhat slower in many cases, but has the advantage that it looks at every file on the targeted disk (whereas Spotlight skips system files by default, for instance).

My primary goal is to make the search as fast as possible.

Fast search built into macOS

Until recently, this went quite well, because Mac disks (volumes) were formatted in HFS+ (aka Mac OS Extended), and Apple provides a special file search operation (CatalogSearch or searchfs) for these volumes, by which FAF could ask for the file name the user is looking for, and macOS would search the volume's directory on itself and only return the matching files. This is very fast.

Unfortunately, with Apple's new file system APFS, and the fact that any macOS running High Sierra or Mojave got their startup volume converted from HFS+ to APFS, search performance has decreased by factor 5 to 6! Where searching the entire startup disk for a file like "hosts" did take just 5 seconds on a fast Mac with a HFS volume, it now takes half a minute or more on APFS.

Besides, the old network file server protocol AFP does also support the fast search operation, but only on real Mac servers - some NAS systems pretend to support this as well, but my experience shows that this is very unreliable. The newer SMB protocol, OTOH, does not appear to support searchfs.

Searching the classic way

When the searchfs operation is not available, unreliable or inefficient, FAF falls back to looking at every directory entry itself, looking for matches to the search, then looking at the contents of every subdirectory, and so on. This is called a recursive search (whereas searchfs performs a flat search over all directory entries of a volume).

There are several ways to read these directories. I'll list the most interesting ones:

• -[NSFileManager contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error:]
• opendir() & readdir_r()
• getattrlistbulk()
• fts_open() && fts_read()

The first is the standard high-level (Foundation) function. It lets you choose which attributes (besides the file name) it shall fetch alongside. This is useful if you want to look at the file sizes, for instance. If you let them fetch along, it'll cache them in the NSURL objects, thereby increase performance if you call [NSURL getResourceValue] later to read the value.

readdir() is a very old UNIX / POSIX function to read a directory's file names, and nothing else, one by one.

getattrlistbulk() is a special Mac BSD function that's an extension to the older getattrlist(). It is supposed to be optimized for faster reading, as it can fetch the entire contents of a directory at once, along with attributes such as file dates and sizes. [NSFileManager contentsOfDirectoryAtURL...] supposedly uses this function, thereby making use of its performance advantage.

fts_open() is a long-existing BSD or POSIX function that is specialized on traversing directory trees. I've added this only after the initial tests, i.e. its discussion is a bit more brief below.

Test methods

I've tried to find out which of the various methods of reading directories, looking only for file names, is the fastest: I had to scan the same directory tree with every method separately.

Testing performance is a bit difficult because macOS tends to cache recently read directories for a short while. For instance, the first time I scan a directory tree with a 100,000 items, it may take 10s, and when I run the same test again withing a few seconds, it'll take only 2s. If I wait half a minute, it may again take 10s. And if I'm searching on a file server, that server may also cache the information in RAM. For instance, my NAS, equipped with Hard Disks, will be rather loud the first time I search on it, due to the HDs performing lots of seeking, whereas a repeat search will make hardly any noise due to little to no seeking, which also increases the search performance.

Therefore, I performed the tests twice in succession: Once after freshly mounting the target volume (so that the cache was clear) and once again right after. This would give me both the worst and best case performances. I repeated this several times and averaged the results.

I also had to test on different media (e.g. fast SSD vs. slower HD) and formats (HFS+, APFS, NTFS) and network protocols (AFP vs. SMB, from both a NAS and another Mac) because they all behave quite differently.

The Xcode project I used for timing all three scanning functions can be downloaded here.

Test results

Most tests were performed on macOS 10.13.6. The NAS is a Synology DS213j with firmware DSM 6.2.1, connected over 1 GBit Ethernet, and both AFP and SMB tests were made on the same NAS directory. The Terminal cmd "smbutil statshares -a" indicates that the latest SMB3 protocol was used. The remote Mac ran macOS 10.14.4, and the targeted directory on it was on a HFS+ volume so that I could compare performance between AFP and SMB (APFS vols can't be shared over AFP). I also did a few tests on the 10.14.4 Mac, though I only recorded the best case results as the others were difficult to create (I'd have had to reboot between every test and I wasn't too keen on that).

I was expecting that contentsOfDirectoryAtURL would always be as fast as its low level version getattrlistbulk, whereas readdir would be slower as it wasn't optimized for this purpose. Surprisingly, this was not always the case. (I did not include the fts method when I did this run of tests - its results will instead be discussed in a separate chapter below).

The values show passed time in seconds for completing a search of a deep folder structure. The values are only comparable in each line, but not across lines, because the folder contents were different. The exception are the network volumes, where the same folders were used for AFP and SMB.

The green fields point out the best results. The red one points out an anomaly.

	contentsOfDirectoryAtURL		getattrlistbulk		opendir/readdir
	worst case	best case	worst case	best case	worst case	best case
HD HFS+	12.4	2.8	12.4	2.3	12.4	2.35
SSD HFS+	4.9	2.8	4.6	2.26	4.6	2.47
SSD APFS	11.2	10.6	10	6.8	8.6	3.2
SSD NTFS	28	6	28	6	8	4.7
10.14 APFS		12		10		8
10.14 HFS+		4.1		3.8		4
NAS via AFP	5.6	2.5	4.8	2.14	5.6	2.7
NAS via SMB	15	15	17	15	9	5.7
Mac via SMB	4.4	5.4	6.8	5.5	6.5	5
Mac via AFP	5.3	3.6	5.1	3.7	5.9	4.3

Observations

• HD vs. SSD shows that the initial search takes much longer on HDs, which makes sense because HDs have a higher latency. Once the data is in the cache, though, both are equally fast (which makes sense as well).
• contentsOfDirectoryAtURL and getattrlistbulk perform equally indeed, just as predicted, with the latter usually being a bit faster once the data comes from the cache.
• On APFS, NTFS and SMB, readdir() is significantly faster than the other methods, which is quite surprising to me.
• SMB performance is worse than AFP (regardless, Apple declared AFP obsolete) in nearly all cases.
• When accessing a Mac via SMB, contentsOfDirectoryAtURL is faster than the other methods, but only on the first run (see red field). Once the caches have been filled, it's slower. I can't make sense of it, but it's a very consistent effect in my tests.

The fts functions

fts_open() / fts_read() are, in most cases, faster than readdir(), contentsOfDirectoryAtURL and getattrlistbulk. Exceptions are network protocols, where especially the retrieval of additional attributes makes it slower than the other methods.

Fetching additional attributes

When extra attributes such as file dates or sizes, are needed during the scan, the timing of the various methods changes as follows:

• For contentsOfDirectoryAtURL and getattrlistbulk, there is little impact if these extra attributes are requested with the function call.
• For readdir(), fetching additional attributes (through lstat()) turns it into the slowest method.
• The fts functions are the least affected by getting attributes that are also available through the lstat() function if a local file system is targeted. However, for network volumes via AFP, they become about 20% slower in my tests, whereas getattrlistbulk stays faster.

Differences between macOS versions

When searching the same volumes (both HFS+ and APFS) from Sierra (10.12.6), High Sierra (10.13.6) and Mojave (10.14.4), I measure a consistent worse performance on Mojave. Meaning that scanning directories got slower in 10.14 vs. 10.13, by about 15%.

Also, getting additional attributes 10.12,  compared to 10.13 and later, takes about twice as long, across all methods. Which could mean that something improved in 10.13 regarding fetching attributes.

Conclusion

It appears that for optimal performance, I need to implement several methods, and select them depending on which file system or protocol I talk to.

Here's my current list of fastest method per file system:

• HFS+: Always fts
• APFS: Always fts
• AFP: Always getattrlistbulk
• SMB: If not attributes needed: readdir, otherwise fts or getattrlistbulk

Update on 29 Apr 2019

When traversing a directory tree, one must take care not to cross over into other volumes, which can happen if you encounted mounted file systems in your path, such as when you parse "/" into "/Volumes".

The safe way to check for this is to determine the volume a folder is on before you dive into it. To identify volumes is to get their volume or device ID. One way is to call stat(), then check its st_dev value, another is to get the NSURLVolumeIdentifierKey. Or, in the case of fts_read, it's already provided - which adds to its superior efficiency.

My testing shows an unpleasant performance impact, though:

When traversing with contentsOfDirectoryAtURL, calling stat() is less efficient than getting the value for NSURLVolumeIdentifierKey. That makes sense, because the stat() fetches more data, and that could cause additional disk I/O.

OTOH, the file system layer should know the ID of the volume without the need to perform disk I/O.

Meaning, getting the value for NSURLVolumeIdentifierKey should cost no significant time at all, because the information is known to the upper file system level, before even passing the request on to the actual file system driver for the parcular volume. Therefore the value should be readily available at a much higher level - regardless, fetching this volume ID takes about as much time as getting an actual value from the lowest level, such as a file size or date.

However, when I add fetching this volume ID to every encountered file & folder, the scan time increases by over 30%. Fortunately, for the scanning, one only has to fetch this value for directories, not for files, which makes this have a smaller overall impact. Still, the performance of this could be better if Apple engineering would consider this, I believe. After all, identifying  the volume ID is needed by almost any directory scanner.

Update on 11 May 2019

When discussing my findings on an Apple forum (actually, on one of the few remaining Apple mailing lists), Jim Luther pointed out to try enumeratorAtURL. And, indeed, this function does better than any of the others, at least with my tests on local disks, both on HFS+ and APFS. Like fts_read, it takes care of staying on the same volume, so that I do not have to check the volume ID myself.

I have updated my test project with the use of this function.

Update on 17 Feb 2025

I have further improved my DirScanner code by avoiding the creation of NSURL object unless necessary and fetching some properties (file size and mount point flag) more efficiently. This also fixes an issue where searching on the root volume would find items twice because it could dive into /System/Volumes/Data accidentally.

Another update later this day was made: NSURLs are now created more efficiently. The key to this is creating them not from NSStrings but from C-Strings using the fileSystemRepresentation initializer, thus avoiding the need for internal conversion operations.

New version of the Xcode project is available at the same location (see below).

Comments, concerns?

Feel free to download the Xcode project and run your own tests.

Comments are welcome here or on Twitter to me: @tempelorg

APFS and fast catalog search

This is about FSCatalogSearch / searchfs support in macOS with the APFS file system.

Updated 3 Oct 2017: Find Any File 1.9 will support fast search on APFS on High Sierra (10.13) by using the searchfs function. Version 1.9 is currently in open beta, see the FAF web site.

Updated 25 July 2017: Clarified why FSCatalogSearch doesn't work on APFS, adds issue about hard links and 64 bit CNID resolving.

Some background on FSCatalogSearch in general

Programs like EasyFind and my own Find Any File (FAF) are able to search for file names (as well as file dates, sizes and a few other rarely needed attributes) on disks in a quite fast manner by using a little-known function macOS offers.

This Carbon level function is known as CatSearch or (FS)CatalogSearch and has been around for more than 25 years. There's also a BSD level function called searchfs.

The advantage of this function is that it performs the search for names at the file system driver level, meaning that when you search for files containing ".png" in their name, the file system can look at the entire directory tree much faster, sorting out the matches, and only report those to the program that initiazes the search and then shows the results to the user.

Without this special function, the search program would have to start at the root of the disk, read each folder (directory) recursively, and then sort out the matches itself, which all takes much more computing time.

For example, a search on a disk with millions of files and folders on it would take only a few seconds with FSCatalogSearch, whereas a classic recursive search would take minutes.

Getting even more technical

Apple added the FSCatalogSearch function in Mac OS long ago, after introducing the HFS file system. This was supported by the fact that  HFS did, unlike Window's FAT, arrange the entire directory tree in one large file on the disk, with interlinked nodes that did not match the hierarchical folder structure. FSCatalogSearch would then iterate over the nodes in a most efficient way, not caring about the folder structure, thereby minimizing disk seek times, which was a significant factor in disk access before SSDs. This also meant that FSCatalogSearch would only work on volume formats that used a single (invisible) file for its entire directory tree, meaning that FSCatalogSearch was never available for FAT disks, for instance. It would also be optimal for NTFS volumes, but since Apple never used NTFS other than to support reading from Bootcamp partitions, they never made the effort to add FSCatalogSearch to their NTFS file system driver.

What about APFS?

Now Apple is about to replace HFS(+) with APFS on macOS. And fans of EasyFind and FAF start wondering: Will I still be able to perform fast disk-wide file name searches the way I'm used to?

The good news is: The APFS file system code has support for the lower level searchfs function, and that's been already added in 2016, apparently, for OS X 10.12. Which ultimately means: Yes, FAF and EasyFind can continue to provide fast search on APFS formatted disks, provided extra work is put into updating the apps accordingly.

However, there are still some issues:

• The high-level FSCatalogSearch does not work on APFS. Both EasyFind and FAF rely on this function and therefore won't find files the fast way on APFS volumes right now. The reason for this is that APFS uses larger values (64 bit) than HFS+ for identifying the files, and the FSCatalogSearch function cannot handle those larger values. (rdar://33454922)
• As of now (10.12.6, 10.13 beta 3), the searchfs function does search case-sensitive and not case-insensitive as it should. That means that searching for ".png" won't find files using ".PNG". I confirmed this with an Apple engineer - it's a known issue, just one with a low priority right now. So, there's a chance that this will get resolved eventually, and I hope it'll be done before 10.13 is released. This issue may not get fixed for 10.12.x, though. We'll have to see what Apple does in this regard. (rdar://33455597)
• Hard links can't be identified correctly - if there are multiple hard links to the same file, then searchfs can't currently tell them apart, and the results will all point to the same directory entry. (rdar://33473247)
• searchfs() returns CNIDs (Catalog Node IDs, 64 bit wide) instead of paths to the found items. This requires resolving these IDs to the paths later. However, there is currently no documented API provided in macOS to do so. There's a hackish way around this, but that's not a proper solution. (rdar://33507188)

What this all means

Current versions of FAF and EasyFind can't fast search on APFS. They need to be rewritten using the searchfs API.

I will be working on a quick-fix version of FAF that'll add fast search on APFS and which I hope to release before 10.13 (High Sierra) is officially released. I have quite a few other improvements for FAF in the works (64 bit app, content search, icon view, server support etc.) which will have to wait so that I can get this APFS issue resolved ASAP.