11 May 2019

Cloning APFS Volumes & Containers ("APFS inverter failed")

(This is an older article that I hadn't published back then because it might not be fully accurate, i.e. the steps may not be applicable. Yet, it contains some valuable information so I decided to publish it now. Read with a grain of salt. If you have corrections, don't hesitate to comment or email me.)

TL;DR

If Disk Utility fails to clone an APFS container, giving "APFS inverter failed" as error message, a work-around solution is to copy the partition with the "dd" command into a same-sized partition, then fix the cloned container's UUIDs.

About cloning (copying) entire Mac volumes

Usually, when you want to clone a Mac volume, you'd use Disk Utility's Restore operation. It performs a sector-by-sector copy (while skipping unused sector . The alternative would be to perform a file-by-file copy, as it's done by 3rd party tools like Carbon Copy Cloner.

A clone operation by copying individual files has several disadvantages over a sector copy:

  • It's much slower.
  • If you're using Finder Alias files, these may not work any more afterwards (that's because they rely on each file's unique ID, and those IDs change when copying files over to the destination).
  • More programs may request re-activation (re-registration).
However, when I recently tried to make an identical copy of my macOS Mojave system from my MacBook Air (2015), copying it to an external SSD, I ran into problems:

After the copy and verification operation was already apparently finished, an additional inverter process needs to be run on cloned APFS volumes. While I do understand that that's necessary when I copy only a single volume out of an APFS container, or copy the volume into a target APFS container without replacing it entirely, but even if I try to clone the entire container, with erasing the target, it still wants to run an inversion process - and that makes no sense to me.

Now, the error that I kept seeing is: APFS inverter failed to invert the volume

And I'm not the only one, see here and here.

I tried many things, including First Aid and removal of all snapshots, and running the cmdline tool "air", which showed me more detailed error messages. Still, no success. I keep getting variations of the same issue.

What I'm showing now is a way to clone a complete APFS container (with all contained volumes) the way it should work.

How to clone an APFS container

Note: This may not work with encrypted volumes. Or it might. I have not tried.

We simply copy the entire partition (which contains the APFS container) sector by sector. (Small disadvantage over Disk Utility's Restore operation: This will also copy unused sectors, so it'll take a bit longer.)

Afterwards, we need to change the UUIDs of the cloned container and its volumes, or the Mac (and especially Disk Utility) may get confused when both the original and the cloned volumes are present on the same Mac.

Perform the sector copy operation

In case you want to copy your bootable macOS system, you will have to first start up from a different macOS system. If you have no other external disk or partition with another macOS system, simply start up from the Recover system.

When ready, connect the target disk, then start Terminal.app.


Get an overview of our disk names by entering (and always pressing Return afterwards):

diskutil list
Here's a sample output of my Mac that has four partitions, two of which use HFS+ ("AirElCap", "AirData") and the other two use APFS ("AirMojave", "AirHighSierra"):

/dev/disk0 (internal):

   #:                       TYPE NAME                    SIZE       IDENTIFIER

   0:      GUID_partition_scheme                         121.3 GB   disk0

   1:                        EFI EFI                     314.6 MB   disk0s1

   2:                 Apple_APFS Container disk1         40.9 GB    disk0s2

   3:                  Apple_HFS AirElCap                20.1 GB    disk0s3

   4:                 Apple_Boot Boot OS X               134.2 MB   disk0s4
   5:                 Apple_APFS Container disk2         39.8 GB    disk0s5
   6:                  Apple_HFS AirData                 19.9 GB    disk0s6

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +40.9 GB    disk1
                                 Physical Store disk0s2
   1:                APFS Volume AirMojave               20.7 GB    disk1s1
   2:                APFS Volume Preboot                 47.1 MB    disk1s2
   3:                APFS Volume Recovery                512.7 MB   disk1s3
   4:                APFS Volume VM                      2.1 GB     disk1s4

/dev/disk2 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +39.8 GB    disk2
                                 Physical Store disk0s5
   1:                APFS Volume AirHighSierra           23.2 GB    disk2s1
   2:                APFS Volume Preboot                 20.9 MB    disk2s2
   3:                APFS Volume Recovery                519.0 MB   disk2s3
   4:                APFS Volume VM                      3.2 GB     disk2s4

I plan to clone "AirMojave", so the disk identifier would be disk1, because that is the container for that volume. In your case it may be a different identifier. I will write the commands below using diskS for the source container and diskTsP for the target partition.

Now unmount all volumes of our source container. In Terminal, enter:

diskutil unmountDisk diskS
If this does not succeed, then you either have programs or files open on one of the volumes or you're trying to unmount the boot volume (in that case, you should have restarted into a different system as explained above). Do not continue if the mount was not successful or you're likely to end up with a corrupted clone.
diskutil apfs list diskS
+-- Container disk2 5099518D-36C0-47CC-B034-0CDA52C51CE8
    ====================================================
    APFS Container Reference:     disk2
    Size (Capacity Ceiling):      39838416896 B (39.8 GB)
    Capacity In Use By Volumes:   27042205696 B (27.0 GB) (67.9% used)
    Capacity Not Allocated:       12796211200 B (12.8 GB) (32.1% free)
sudo diskutil resizeVolume diskTsP size
In place of size, use the number that's show after "Size (Capacity Ceiling):".
diskutil unmount diskTsP
This is the command that will perform the copying:
sudo dd bs=65536 if=/dev/diskS of=/dev/diskTsP
While dd is doing its work, it won't show any progress on its own. And it can take hours, even days. To check where it's at, type ctrl-T in the Terminal window. That will print a line showing its current progress.

Fix the UUIDs

Enter in Terminal:
sudo /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -s /dev/diskT
You'll likely not get any response from this command, telling you success or failure.

To check whether the UUIDs have been changed successfully, enter in Terminal:
diskutil apfs list
In the output, find your source and target disks, and compare the UUIDs of their container and container volumes. They should all be different. If the UUIDs of source and target volumes show the same code, then the fix did not work. Try again.

Make the system volume bootable

The last step is to make the cloned volume bootable again (in case it was before). For that, you need to mount the Preboot volume, then rename the single folder inside, which contains the old UUID, into the new UUID of the main volume.

To mount it:
diskutil mount diskTsX
(X stands for the partition number containing the "Preboot" volume)

To show it in Finder:
open /Volumes/Preboot
Now you should see a Finder window with a folder named after the UUID of the original volume you copied. Rename that folder into the new UUID. To verify that you used the correct UUID, open System Preferences, Startup Disk, select the cloned bootable volume and click the Restart button. If it says that it can't bless it, you got it wrong. Otherwise, the system should now restart, meaning the UUID was correctly set to a bootable volume.

10 comments:

  1. After performign the DD command I'm not getting any kind of output. Is this normal? Does it just take a long time?

    ReplyDelete
    Replies
    1. Yes, dd makes no output by default. And yes, it probably takes a while (up to hours), but should show a summary of how many blocks were copied once it's done.

      Delete
    2. Thanks Thomas - actually took about ten days hah but worked - thanks so much!

      Delete
    3. Yikes! Conncted via USB 2, I assume? How large was the disk?

      In fact, one can ask dd to show its progress, as I just found out via the terminal cmd "man dd": Hit ctrl-T while dd is running and it'll output where it's at. I've updated the post accordingly.

      Delete
    4. I'm not sure why but its cloned it as a HFS file system and not APFS!

      Delete
  2. If you get "inverter failed" errors with asr you should be able to run the command with "--verbose --debug" flags to maybe get more specific info on the issue you are facing.

    btw. I my case (macOS Catalina 1.15.1) the issue was that my original APFS container had some APFS Volumes with APFS snapshots in them (created by TimeMachine). Deleting those snapshots allowed me to run asr restore successfully. See my article here:
    https://gist.github.com/darwin/3c92ac089cf99beb54f1108b2e8b4b9f

    ReplyDelete
  3. You can make dd copy much more quickly by doing three things. Use a larger block size, for example 1m. Read from the physical partition, disk0s2 in your example, instead of the synthesized disk, disk1. And use the raw disk devices, prefix r. So the command becomes

    sudo dd bs=1m if=/dev/rdisk0s2 of=/dev/rdisk0s5

    The copying will then go at the speed of the media. In your example, suppose that the speed of the 128GB SSD in the MacBook Air is 300MB/s. Then the copy will go at 150MB/s reading and 150MB/s writing and 40GB will take between 4 and 5 minutes. The only situation where the media speed does not apply is when you are copying between partitions on the same HDD because there will be a lot of head-seeking activity.

    The procedure you describe also works with encrypted volumes if you do not change the volume UUID. The volume UUID is used in the decryption, therefore it must be preserved. I wonder whether the purpose of apfs.util -s is to produce an immediate secure erase. I have successfully mounted and unlocked the cloned volume. It is not possible to choose whether the original or the clone will boot the computer, because the UUIDs are the same. The computer will choose. I had the clone on an external drive and my computer always chose the internal drive. So I have not been able to prove that the clone is bootable. The only way to do that would be to disconnect the internal drive. Note the cloning procedure works with software encryption but I expect it would not work with hardware encryption.

    In principle it should be possible to modify the container's keybag, and to rewrap the container's keybag and the volume's keybag, to suit new UUIDs. However I do not know of a utility which does this. Such a utility might weaken security.

    ReplyDelete
  4. How has the target disk be prepared before cloning? Should it be partitioned first with an (empty target) APFS container? Or could it be a normal HPFS partition to which the following dd will copy the source APFS container?
    (The commandline "sudo diskutil resizeVolume diskTsP " indicates for me that diskTsP is only a HPFS partition. Otherwise it should be "diskutil apfs resizeContainer diskTsP ".

    ReplyDelete
    Replies
    1. Since we're copying an entire container, which is practically the entire partition, it does not matter much how the target partition was pre-formatted, as it gets overwritten anyway. What may need updating is the partition type and UUID in the GUID partition map entry, afterwards. For that reaons, pre-formatting it already with an empty APFS container would be useful, as then the partition will have the correct type set already, at least, I reckon.

      Delete
  5. This comment has been removed by the author.

    ReplyDelete