11 May 2019

Cloning APFS Volumes & Containers ("APFS inverter failed")

(This is an older article that I hadn't published back then because it might not be fully accurate, i.e. the steps may not be applicable. Yet, it contains some valuable information so I decided to publish it now. Read with a grain of salt. If you have corrections, don't hesitate to comment or email me.)

TL;DR

If Disk Utility fails to clone an APFS container, giving "APFS inverter failed" as error message, a work-around solution is to copy the partition with the "dd" command into a same-sized partition, then fix the cloned container's UUIDs.

About cloning (copying) entire Mac volumes

Usually, when you want to clone a Mac volume, you'd use Disk Utility's Restore operation. It performs a sector-by-sector copy (while skipping unused sector . The alternative would be to perform a file-by-file copy, as it's done by 3rd party tools like Carbon Copy Cloner.

A clone operation by copying individual files has several disadvantages over a sector copy:

  • It's much slower.
  • If you're using Finder Alias files, these may not work any more afterwards (that's because they rely on each file's unique ID, and those IDs change when copying files over to the destination).
  • More programs may request re-activation (re-registration).
However, when I recently tried to make an identical copy of my macOS Mojave system from my MacBook Air (2015), copying it to an external SSD, I ran into problems:

After the copy and verification operation was already apparently finished, an additional inverter process needs to be run on cloned APFS volumes. While I do understand that that's necessary when I copy only a single volume out of an APFS container, or copy the volume into a target APFS container without replacing it entirely, but even if I try to clone the entire container, with erasing the target, it still wants to run an inversion process - and that makes no sense to me.

Now, the error that I kept seeing is: APFS inverter failed to invert the volume

And I'm not the only one, see here and here.

I tried many things, including First Aid and removal of all snapshots, and running the cmdline tool "asr", which showed me more detailed error messages. Still, no success. I keep getting variations of the same issue.

What I'm showing now is a way to clone a complete APFS container (with all contained volumes) the way it should work.

How to clone an APFS container

Note: This may not work with encrypted volumes. Or it might. I have not tried.

We simply copy the entire partition (which contains the APFS container) sector by sector. (Small disadvantage over Disk Utility's Restore operation: This will also copy unused sectors, so it'll take a bit longer.)

Afterwards, we need to change the UUIDs of the cloned container and its volumes, or the Mac (and especially Disk Utility) may get confused when both the original and the cloned volumes are present on the same Mac.

Perform the sector copy operation

In case you want to copy your bootable macOS system, you will have to first start up from a different macOS system. If you have no other external disk or partition with another macOS system, simply start up from the Recover system.

When ready, connect the target disk, then start Terminal.app.


Get an overview of our disk names by entering (and always pressing Return afterwards):

diskutil list
Here's a sample output of my Mac that has four partitions, two of which use HFS+ ("AirElCap", "AirData") and the other two use APFS ("AirMojave", "AirHighSierra"):
/dev/disk0 (internal):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                         121.3 GB   disk0
   1:                        EFI EFI                     314.6 MB   disk0s1
   2:                 Apple_APFS Container disk1         40.9 GB    disk0s2
   3:                  Apple_HFS AirElCap                20.1 GB    disk0s3
   4:                 Apple_Boot Boot OS X               134.2 MB   disk0s4
   5:                 Apple_APFS Container disk2         39.8 GB    disk0s5
   6:                  Apple_HFS AirData                 19.9 GB    disk0s6

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +40.9 GB    disk1
                                 Physical Store disk0s2
   1:                APFS Volume AirMojave               20.7 GB    disk1s1
   2:                APFS Volume Preboot                 47.1 MB    disk1s2
   3:                APFS Volume Recovery                512.7 MB   disk1s3
   4:                APFS Volume VM                      2.1 GB     disk1s4

/dev/disk2 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +39.8 GB    disk2
                                 Physical Store disk0s5
   1:                APFS Volume AirHighSierra           23.2 GB    disk2s1
   2:                APFS Volume Preboot                 20.9 MB    disk2s2
   3:                APFS Volume Recovery                519.0 MB   disk2s3
   4:                APFS Volume VM                      3.2 GB     disk2s4

I plan to clone "AirMojave", so the disk identifier would be disk1, because that is the container for that volume. In your case it may be a different identifier. I will write the commands below using diskS for the source container and diskTsP for the target partition.

Now unmount all volumes of our source container. In Terminal, enter:

diskutil unmountDisk diskS
If this does not succeed, then you either have programs or files open on one of the volumes or you're trying to unmount the boot volume (in that case, you should have restarted into a different system as explained above). Do not continue if the mount was not successful or you're likely to end up with a corrupted clone.
diskutil apfs list diskS
+-- Container disk2 5099518D-36C0-47CC-B034-0CDA52C51CE8
    ====================================================
    APFS Container Reference:     disk2
    Size (Capacity Ceiling):      39838416896 B (39.8 GB)
    Capacity In Use By Volumes:   27042205696 B (27.0 GB)
    Capacity Not Allocated:       12796211200 B (12.8 GB)
sudo diskutil resizeVolume diskTsP size
In place of size, use the number that's show after "Size (Capacity Ceiling):".
diskutil unmount diskTsP
This is the command that will perform the copying:
sudo dd bs=1m if=/dev/diskS of=/dev/diskTsP

(See also Adrian's comment below, suggesting to use "rdisk" instead of "disk" for higher speed.)

While dd is doing its work, it won't show any progress on its own. And it can take hours, even days. To check where it's at, type ctrl-T in the Terminal window. That will print a line showing its current progress.

Fix the UUIDs

Enter in Terminal:
sudo /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -s /dev/diskT
You'll likely not get any response from this command, telling you success or failure.

To check whether the UUIDs have been changed successfully, enter in Terminal:
diskutil apfs list
In the output, find your source and target disks, and compare the UUIDs of their container and container volumes. They should all be different. If the UUIDs of source and target volumes show the same code, then the fix did not work. Try again.

Make the system volume bootable

The last step is to make the cloned volume bootable again (in case it was before). For that, you need to mount the Preboot volume, then rename the single folder inside, which contains the old UUID, into the new UUID of the main volume.

To mount it:
diskutil mount diskTsX
(X stands for the partition number containing the "Preboot" volume)

To show it in Finder:
open /Volumes/Preboot
Now you should see a Finder window with a folder named after the UUID of the original volume you copied. Rename that folder into the new UUID. To verify that you used the correct UUID, open System Preferences, Startup Disk, select the cloned bootable volume and click the Restart button. If it says that it can't bless it, you got it wrong. Otherwise, the system should now restart, meaning the UUID was correctly set to a bootable volume.

16 comments:

  1. After performign the DD command I'm not getting any kind of output. Is this normal? Does it just take a long time?

    ReplyDelete
    Replies
    1. Yes, dd makes no output by default. And yes, it probably takes a while (up to hours), but should show a summary of how many blocks were copied once it's done.

      Delete
    2. Thanks Thomas - actually took about ten days hah but worked - thanks so much!

      Delete
    3. Yikes! Conncted via USB 2, I assume? How large was the disk?

      In fact, one can ask dd to show its progress, as I just found out via the terminal cmd "man dd": Hit ctrl-T while dd is running and it'll output where it's at. I've updated the post accordingly.

      Delete
    4. I'm not sure why but its cloned it as a HFS file system and not APFS!

      Delete
  2. If you get "inverter failed" errors with asr you should be able to run the command with "--verbose --debug" flags to maybe get more specific info on the issue you are facing.

    btw. I my case (macOS Catalina 1.15.1) the issue was that my original APFS container had some APFS Volumes with APFS snapshots in them (created by TimeMachine). Deleting those snapshots allowed me to run asr restore successfully. See my article here:
    https://gist.github.com/darwin/3c92ac089cf99beb54f1108b2e8b4b9f

    ReplyDelete
  3. You can make dd copy much more quickly by doing three things. Use a larger block size, for example 1m. Read from the physical partition, disk0s2 in your example, instead of the synthesized disk, disk1. And use the raw disk devices, prefix r. So the command becomes

    sudo dd bs=1m if=/dev/rdisk0s2 of=/dev/rdisk0s5

    The copying will then go at the speed of the media. In your example, suppose that the speed of the 128GB SSD in the MacBook Air is 300MB/s. Then the copy will go at 150MB/s reading and 150MB/s writing and 40GB will take between 4 and 5 minutes. The only situation where the media speed does not apply is when you are copying between partitions on the same HDD because there will be a lot of head-seeking activity.

    The procedure you describe also works with encrypted volumes if you do not change the volume UUID. The volume UUID is used in the decryption, therefore it must be preserved. I wonder whether the purpose of apfs.util -s is to produce an immediate secure erase. I have successfully mounted and unlocked the cloned volume. It is not possible to choose whether the original or the clone will boot the computer, because the UUIDs are the same. The computer will choose. I had the clone on an external drive and my computer always chose the internal drive. So I have not been able to prove that the clone is bootable. The only way to do that would be to disconnect the internal drive. Note the cloning procedure works with software encryption but I expect it would not work with hardware encryption.

    In principle it should be possible to modify the container's keybag, and to rewrap the container's keybag and the volume's keybag, to suit new UUIDs. However I do not know of a utility which does this. Such a utility might weaken security.

    ReplyDelete
  4. How has the target disk be prepared before cloning? Should it be partitioned first with an (empty target) APFS container? Or could it be a normal HPFS partition to which the following dd will copy the source APFS container?
    (The commandline "sudo diskutil resizeVolume diskTsP " indicates for me that diskTsP is only a HPFS partition. Otherwise it should be "diskutil apfs resizeContainer diskTsP ".

    ReplyDelete
    Replies
    1. Since we're copying an entire container, which is practically the entire partition, it does not matter much how the target partition was pre-formatted, as it gets overwritten anyway. What may need updating is the partition type and UUID in the GUID partition map entry, afterwards. For that reaons, pre-formatting it already with an empty APFS container would be useful, as then the partition will have the correct type set already, at least, I reckon.

      Delete
    2. Hi Thomas,
      yes, you're right. I had the same thoughts, that it would not matter which type the target partition has, but after copying with dd the target partition was not recognized as an APFS container. Because I had no ideas how to change partition types in macOS I decided that it's best to format the target disk with an empty APFS container before copying the source container with dd. Perhaps you could change the tutorial and the command line to adjust the target container size accordingly?

      Anyway, everything else worked as expected, thanks for the tutorial! I would only suggest changing the dd command line in your tutorial to the syntax suggested from Adrian (with rdisk and 1M it took only 3 minutes instead of 47 minutes on my system with your syntax).

      In addition I'd like to add a comment regarding the startup process in recovery: It's best to use the recovery of an independent system like a DVD or an install media. I tried to use the recovery of the source system first, but in this case the unmount command failed (IIRC the dd command refused to work).

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
  6. Thanks for this post. Extremely clarifying, and verified my understandings based on what I read elsewhere. It was good to hear that you made work what I had hypothesized -- that you could restore all volumes by using dd.

    Question for you: in your comment from 19-Feb-2020, 22:22, you referred to the container as "practically" the entire partition. Do you mean practically like "for all intents and purposes", or like "nearly"? Because, if it's the latter, I'd like to acquire a better understanding of how an APFS container is different, since I thought it was just a a partition of type Apple_APFS, with the name "container" implying multiple volume capability, unlike HFS+, but not technically being anything different than a partition of a disk.

    ReplyDelete
    Replies
    1. Yes, a container occupies a partition and is identified by the type ID in the GUID partition entry. IIRC, a container maintains its own size and thus could theoretically be smaller than the partition it sits in. An APFS container uses a B*Tree, similarly to HFS, for maintaining all its contents across the multiple volumes it manages. Does that answer your question?

      Delete
    2. I think it does! So, APFS sounds similar to HFS, then -- i.e. a GUID Partition Table could presumably define a partition of type either Apple_HFS or Apple_APFS, and under normal circumstances the data structures of either would agree with the GPT, but there's nothing that forces them to do so. (IIRC, if you use dd and not asr to block-copy an HFS partition to a different-sized partition on another disk, the copied volume will incorrectly report the original partition's size.) So, in my mind, that makes "container" completely synonymous with "partition for APFS"; if the container happens to disagree with what the GPT says about the partition size, that just means there's something wrong with the partition/container data, just as is the case with HFS. It doesn't meant that the concepts are really different. Or at least that's how it makes sense to me! Please feel free to tell me if I'm thinking about it wrongly. And thank you again for your article, and your quick response.

      As a experiment, I wonder what would happen if you tried to use dd to restore a (non-system) APFS container to an *APM* partition. I'm sure something would protest somewhere, but in theory, it's just a different kind of partition map that the Mac understands, but with APFS partition (container) data, rather than HFS partition data.

      Delete
    3. If you try that experiement, please share what you find out. I'm curious, but not much as it's not really helpful, IMO :)

      Delete
    4. Not really helpful at all! I didn't actually do it, because it occurred to me that the partition would have to be identified in the table as type APFS, so it could be known to be handled as an APFS container, and of course APM doesn't offer an APFS partition type.

      What I did do, just for laughs and science, was see what happens if an APFS container occupied a partition incorrectly indicated as HFS. Big surprise: it didn't work. (Though, if it had, I might have carried on with the APM experiment.)

      I made a sector-dump image (CD/DVD master format) of a test APFS container. I then erased the original disk, and made new partitions, all HFS, of different sizes and number, but I made one have the identical size as the original APFS container. I then used dd to clone the image file directly to that identically-sized partition.

      (I didn't need to mount the image first, as I would have if it were a .dmg, because the .cdr file is just a straight sector dump, like an .iso. The fact that the length of the file is identical to the partition size, as revealed by "gpt show", furthers the idea that there's no meaningful distinction between "APFS container" and "APFS-formatted partition".)

      Anyway, with the container still identified in the GPT as a partition of type HFS, macOS was not thrilled, and ignored the whole partition. It was absent from "diskutil list" and present only as a greyed out "disk2s4" in Disk Utility.

      But when I changed the partition type to APFS -- which I achieved by using "gpt remove" followed by "gpt add" -- everything was exactly as expected. The volumes within the container, as block-copied from the image file, showed right up.

      Delete