Universal disk image format

from Wikipedia, the free encyclopedia
Universal disk image format
File extension : .dmg, .udif
MIME type : application / x-apple-diskimage-udif application / x-apple-diskimage
Magic number : 6B 6F 6C 79 hex
1,802,464,377

(koly)

Developed by: Apple, Inc.
Initial release: 2000
Current version: 4th
Type: Memory image , optionally compressed and encrypted
Container for: APM , FAT , GPT , HFS , HFS + , ISO 9660 , MFS , MBR , UDF etc. a.
Contained in: macOS
Standard (s) : proprietary
Website : support.apple.com

The Universal Disk Image Format (UDIF) is one of Apple Computer, Inc. for Mac OS X developed proprietary file format for memory dumps . It replaced the New Disk Image Format (NDIF) previously introduced with Disk Copy 6.0 , in which the metadata was stored in the resource fork . With UDIF, all data - data fork , resource fork and metadata - are saved in the file itself, which facilitates data exchange via the Internet or non-Macintosh media.

The first release of Mac OS X Public Beta (10.0, "Kodiak," 2000) already included operating system support for the Universal Disk Image Format , which was associated with the filename extension .udif . Starting with Mac OS X 10.2 (“Jaguar,” 2002), the extension is usually .dmgused, which Apple calls “ disk image ” (translated as “image file”). UDIF is not part of Darwin .

Under classic Mac OS UDIF is not supported. However, there is a developer version of Disk Copy 6.4 and beta version 6.5 that support uncompressed, unencrypted UDIF memory images under Mac OS 9 .

For other operating systems there are reverse engineering programs for converting UDIF images into other formats.

Integration in Mac OS X / OS X / macOS

In Mac OS X, which was renamed to OS X in 2012 and macOS in 2016 , a UDIF memory dump can be created using multiple programs. A UDIF file can be created and viewed on the command line with hdiutil. Up to Mac OS X 10.2 ("Jaguar," 2002) Disk Copy was graphically responsible, which was replaced by the disk utility as of Mac OS X Panther (10.3, 2003) .

Also starting in Mac OS X Panther takes care of the DiskImageMounter.apparound the hook of supported memory dumps. Up to Mac OS X Snow Leopard (10.6, 2009) the formats used by the classic Mac OS such as NDIF and Disk Copy 4.2 are also supported. With Mac OS X Lion (10.7, 2011) Apple removed support for these formats.

The DiskImageMounter can, however, also use simple memory images , such as those that can also be ddcreated with . The hard disk service program can also create such 1: 1 copies of the data in a data storage device: although these also have the extension .dmgin the file name, they are not UDIF files.

construction

A UDIF memory image consists of several, partly compressed sections. A partition table is usually also included in the memory image . The data is because Mac OS X originally designed for the PowerPC - architecture was developed in Big Endian coded.

The file usually begins with the binary memory image, followed by metadata such as the resource fork known from NDIF and a property list , followed by a Koly block. The resource fork is 1: 1 what was still stored in an alternative data stream in the file system with NDIF . Since this data is now stored in the file itself, a UDIF file can be copied safely and without loss of metadata via the Internet or non-Macintosh media. In newer versions of Mac OS X, the resource fork in this binary form is usually omitted because the data is completely plistmapped in the . The property list ( plistwhich translates as a list of properties ) is available in 7-bit XML format and stores metadata such as the resource fork and block lists.

A UDIF file can contain several such segments, so that several independent memory images can be contained per UDIF file. Likewise, more than one partition can be contained in a memory image by means of the partition table.

A typical structure looks like this:

  1. Memory image ( data fork ), differently compressed and / or encrypted in blocks
  2. Metadata ( resource fork , was only used up to Mac OS X 10.1 and is no longer created as of Mac OS X Tiger 10.4.7)
  3. Block table ( plist; also contains the other data of the resource fork )
  4. Koly block

Memory dump

The data that is saved as an image of a data memory as UDIF can be available in different forms, for example as a 1: 1 copy or as an encrypted partition. For UDIF it is only important that there is also a block table and a Koly block that describes the form of the data. Most of the time the data is compressed.

In this respect, UDIF is not restricted to a specific partition table or a specific file system. Images of ISO-9660 media (also as a hybrid) such as CD-ROMs or DVDs are just as common as small installation images for Mac applications, which are usually from APM or GPT partition tables with a data partition in the Apple-standard HFS + or APFS are built.

Block table

The block table makes it easier to find the correct data or partition within the memory map by providing block addresses. The list is written as a property list in XML and in most cases contains a division into partitions. In Mac OS X / OS X / macOS the two most used partition tables are the Apple Partition Map , therefore, can be found in plistthe partition entries Driver Descriptor Mapand Apple_partition_map, and the GUID partition table , then the entries Protective Master Boot Record, GPT Header, GPT Partition Data, EFI System Partition. In both cases, the typical Mac partitions of type Apple_HFSand Apple_Freeare available, but generally any partition table supported by the operating system and any supported file system (with the associated partition type) can be used. For some file systems, such as ISO 9660 or UDF , additional partitioning is not required.

In any case, the property list has an entry for each area, such as a partition or a file system, within the blkxkey, which contains the block list. 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>resource-fork</key>
    <dict>
        <key>blkx</key>
        <array>
            <dict>
                <key>Attributes</key>
                <string>0x0050</string>
                <key>CFName</key>
                <string>Protective Master Boot Record (MBR : 0)</string>
                <key>Data</key>
                <data>
                bWlza…
                </data>
                <key>ID</key>
                <string>-1</string>
                <key>Name</key>
                <string>Protective Master Boot Record (MBR : 0)</string>
            </dict>
        </array>
        <key>plst</key>
        <array>
            <dict>
                <key>Attributes</key>
                <string>0x0050</string>
                <key>Data</key>
                <data>
                …
                </data>
                <key>ID</key>
                <string>0</string>
                <key>Name</key>
                <string></string>
            </dict>
        </array>
    </dict>
</dict>
</plist>

The Dataseparate block list for each partition within the key results in the advantage that a single block (or sector) can be decoded more quickly because it is not the entire partition that has to be decompressed, but rather only the connected blocks. It is also possible to simply leave out unwritten (empty) areas ( Zero-Fill). Areas that are difficult to compress anyway can simply be saved as a 1: 1 copy unchanged ( RAW, unchanged raw data), which ultimately brings a speed advantage.

The area inside contains <key>Data</key> <data> … </data> </key>the Base64 -coded block list. This always begins with the signature mish, which bWlzaresults in Base64 encoding . Since the number of blocks is not determined in advance, the list continues until an entry of the type 0xffffffffmarks the last block. The structure of the block list in C is as follows: 

typedef struct {
        uint32_t Signature;          // Magic ('mish')
        uint32_t Version;            // Current version is 1
        uint64_t SectorNumber;       // Starting disk sector in this blkx descriptor
        uint64_t SectorCount;        // Number of disk sectors in this blkx descriptor

        uint64_t DataOffset;
        uint32_t BuffersNeeded;
        uint32_t BlockDescriptors;   // Number of descriptors

        uint32_t reserved1;
        uint32_t reserved2;
        uint32_t reserved3;
        uint32_t reserved4;
        uint32_t reserved5;
        uint32_t reserved6;

        UDIFChecksum checksum;

        uint32_t NumberOfBlockChunks;
        BLKXChunkEntry [0];
} __attribute__((__packed__)) BLKXTable;

// Where each  BLXKRunEntry is defined as follows:

typedef struct {
        uint32_t EntryType;         // Compression type used or entry type (see next table)
        uint32_t Comment;           // "+beg" or "+end", if EntryType is comment (0x7FFFFFFE). Else reserved.
        uint64_t SectorNumber;      // Start sector of this chunk
        uint64_t SectorCount;       // Number of sectors in this chunk
        uint64_t CompressedOffset;  // Start of chunk in data fork
        uint64_t CompressedLength;  // Count of bytes of chunk, in data fork
} __attribute__((__packed__)) BLKXChunkEntry;

The following block types of the variable EntryTypecould blxxpreviously be reverse engineered as type :

blxx-Type (Hex) Scheme description introduction
0x00000000 - Zero-Fill , omitted (no data saved) - an area in which there are only zeros (empty blocks) Mac OS X 10.0 ("Cheetah," 2001)
0x00000001 UDRW, UDRO RAW or NULL compression , uncompressed raw data Mac OS X 10.0 ("Cheetah," 2001)
0x00000002 - unknown, ignored -
? UDRo write-protected (read-only, obsolete) ?
? UDCo compressed (obsolete) ?
0x80000004 UDCO Apple Data Compression (ADC) , compressed Mac OS X 10.0 ("Cheetah," 2001)
0x80000005 UDZO zLib data compression , compressed Mac OS X 10.1 ("Puma," 2001)
0x80000006 UDBZ bz2lib data compression , compressed Mac OS X Tiger (10.4, 2005)
0x80000007 ULFO LZFSE data compression , compressed OS X El Capitan (11/10/2015)
0x7ffffffe - no blocks, but a comment ( +begand +end) -
0xffffffff - no blocks - marks the end of the blxxlist as the last entry -

Koly block

The last 512-byte block .dmgusually contains a Koly block, which is named after its signature. This contains the essential jump addresses for the structure of the UDIF memory image.

Offset Length (in bytes) Data type content
hex dec
00 00 0 4 char Signature, the magic number of the trailer marks the start of the Koly block. The hexadecimal number 6B 6F 6C 79gives the text koly.
00 04 4 4 uint32_t Version, the UDIF version number; usually 4.
00 08 8 4 uint32_t HeaderSize, the size of the koly block; always has the value 512.
00 0C 12 4 uint32_t Flags
00 10 16 8 uint64_t RunningDataForkOffset
00 18 24 8 uint64_t DataForkOffset, the offset address of the data in the UDIF file. Usually 0because the dates are at the beginning.
00 20 32 8 uint64_t DataForkLength, the size of the (compressed, encrypted) memory image in the UDIF file, in bytes.
00 28 40 8 uint64_t RsrcForkOffset, the offset address for metadata, if any.
00 30 48 8 uint64_t RsrcForkLength, the size of the metadata in bytes, if any.
00 38 56 4 uint32_t SegmentNumber, Number of the segment; usually 1, can also 0be.
00 3C 60 4 uint32_t SegmentCount, Number of segments; normally 1(can also 0be).
00 40 64 16 uuid_t SegmentID, 128-bit GUID of the segment (if SegmentNumbernot 0is).
00 50 80 4 uint32_t DataChecksumSize, Information about the checksum .
00 54 84 132 32 × uint32_t DataChecksum, up to 128 bytes large checksum of the data ( data fork ).
00 D8 216 8 uint64_t XMLOffset, Offset address from the beginning of the file to plist(property list, in XML format).
00 E0 224 8 uint64_t XMLLength, the size of the plistin bytes.
00 E8 232 120 120 × uint8_t reserved (not used)
01 60 352 4 uint32_t ChecksumSize, Information about the checksum of the uncompressed memory dump.
01 64 356 132 32 × uint32_t Checksum, up to 128 bytes large checksum of the uncompressed data.
01 E8 488 4 uint32_t ImageVariant, usually 1.
01 EC 492 8 uint64_t SectorCount, Number of sectors in the uncompressed memory image.
01 F4 500 4 uint32_t reserved (unused)
01 F8 504 4 uint32_t reserved (unused)
01 FC 508 4 uint32_t reserved (unused)

Since the Koly block is almost always at the end of the file, it can be used relatively easily to find the property list: The values ​​for XMLOffsetand XMLLengthindicate the position of the plistin the UDIF file.

recognition

Since the information that the file is in Universal Disk Image Format is located as a Koly block at the end of the file after a wide variety of binary data, a file with the extension .dmgor .udifthe Unix command is file not recognized as UDIF. Instead, it specifies filethe type of data at the beginning of the file, such as zlib compressed dataor bzip2 compressed data. However, there is often a false detection, for example VAX COFF executable.

Under macOS, the command outputs hdiutil imageinfo Datei.dmgdetailed information on the UDIF file (in the example with the file name Datei.dmg). 

user@Mac:~$ hdiutil imageinfo Datei.dmg | grep Format
Format Description: UDIF read-only compressed (zlib)
Format: UDZO

compatibility

In general, UDIF files cannot be used in older Mac OS X versions if they do not support the compression algorithm used, the partition table or the file system used. The proprietary compression methods ADC and LZFSE are not available on other operating systems, so that only zLib and bz2 compressed data can be processed.

Under the very old versions of Mac OS X, 10.0 ("Cheetah," 2001) and 10.1 ("Puma," 2001), only UDIF files with an existing resource fork are supported. As of Mac OS X Tiger 10.4.7 (June 2006), however, when a UDIF file is created, the RsrcForkOffsetand fields RsrcForkLengthin the Koly block are no longer automatically filled, which consequently results in UDIF images created with Mac OS X prior to 10.2 ("Jaguar," 2002) makes incompatible.

Web links

Individual evidence

  1. ↑ Disk Utility for Mac: disk image (English), Apple Support, quote: “ A disk image (.dmg file) is a file that looks and acts like a mountable device or volume. “Retrieved November 1, 2016.
  2. ↑ Disk Utility for Mac: Image file , Apple Support, quote: "An image file (.dmg file) is a file that behaves like an activatable device or volume." Retrieved November 1, 2016.
  3. ↑ puredarwin.org Disk images ( Memento of the original from October 31, 2016 in the Internet Archive ) Info: The archive link was inserted automatically and has not yet been checked. Please check the original and archive link according to the instructions and then remove this notice. (English); accessed on October 31, 2016. @1@ 2Template: Webachiv / IABot / www.puredarwin.org
  4. ↑ DiskCopy 4.2, 6.0, 6.3.3, 6.4, 6.5b13 (English), Macintosh Repository; accessed on October 20, 2016.
  5. ↑ a b Demystifying the DMG File Format (English), Jonathan Levin, June 12, 2013; accessed on October 30, 2016.
  6. ↑ Macintosh Disk Images (English); accessed on October 30, 2016.