dd

Moving

Find the name of the drive:

fdisk -l # unix
diskutil list # macos
df -h # show all partitions

Caution

ALWAYS run diskutil list before running operation. disk number can change and you may copy to the wrong destination.

Preparation

Unmount the drive first.

  • If you don't, running further commands will result in dd: /dev/disk_name: Operation not permitted

disk_name is obtained from diskutil list

sudo diskutil unmountDisk /dev/disk_name

dd

dd, copies bits from if to of location.

  • dd is disk/data duplicator/destroyer in Unix-like systems.
sudo dd of=/dev/disk_num1 if=/dev/disk_num2 bs=32M status=progress

bs

bs stands for block size.

  • Larger block size speeds up process for dd overall, but plateaus after a certain point of increase.

Always use bs to make input block size and output block size the same.

ddrescue

Preferred over dd when recovering data.

  • need to install it via brew in macos
brew install ddrescue

Caveat - ddrescue has file number limit.

  • If the disk is large, it may not work.

Logging

dd if=/dev/zero of=/dev/null status=progress |& stdbuf -oL -eL tr '\r' '\n' >>test.txt

Obfuscate/Delete a disk

/dev/zero is a special Unix file filled with zeroes

  • /dev/urandom is filled with random numbers (slower)
# fill with zero
sudo dd if=/dev/zero of=/dev/disk_name bs=1M

# fill with random
sudo dd if=/dev/urandom of=/dev/disk_name bs=1M

Tracing

View progress of dd process while transfer is in progress:

# ad hoc
# send signal to the process to print to its stderr stream
sudo kill -INFO $(pgrep ^dd$)

# continuous
# GNU coreutils >=8.24, add `status=progress`
sudo dd if=/dev/zero of=/dev/disk_name bs=1M status=progress

# Alternative
openssl enc -aes-128-ctr -pass file:/dev/random 2>/dev/null | tail -c+17

Operation Time

Keep in mind, this process can take hours if disk is large

  • On Macbook with M1 processor, starts with 373086326 bytes/sec
  • About 373MB/second
    • 3000 seconds per 1TB
    • 2TB = 6000 seconds = 100 minutes = 1.66 hrs (estimate)
  • 2TB Test, Actual
    • 2000398942208 bytes transferred in 5508.083470 secs (363175132 bytes/sec)
1907730+0 records in
1907729+1 records out
2000398860288 bytes transferred in 5999.167580 secs (333446071 bytes/sec)

Resuming

seek skips n blocks from the beginning of the output file.

skip skips n blocks from the beginning of the input file.

dd if=/dev/null of=./VirtualDisk.img bs=1M seek=1024

Error Handling

original data example:

  +---+---+---+---+---+---+---+---+---+---+---+---+
  | a   b   c   d | e   f   g   h | i   j   k   l |
  +---+---+---+---+---+---+---+---+---+---+---+---+

Suppose we are using block of length 4, and error occurs while reading 'g'.

  +---+---+---+---+---+---+---+---+---+---+---+---+
  | a   b   c   d | e   f  ERR  h | i   j   k   l |
  +---+---+---+---+---+---+---+---+---+---+---+---+

result with default behavior: stops at i/o error

  +---+---+---+---+---+---+
  | a   b   c   d | e   f |
  +---+---+---+---+---+---+

conv=sync: stops at i/o error, and last block is filled with nulls.

  +---+---+---+---+---+---+---+---+
  | a   b   c   d | e   f  \0  \0 |
  +---+---+---+---+---+---+---+---+

conv=noerror: data with error is ignored; original size is lost.

  +---+---+---+---+---+---+---+---+---+---+
  | a   b   c   d | e   f | i   j   k   l |
  +---+---+---+---+---+---+---+---+---+---+

conv=sync, noerror: data is lost, but original length is preserved:

  +---+---+---+---+---+---+---+---+---+---+---+---+
  | a   b   c   d | e   f  \0  \0 | i   j   k   l |
  +---+---+---+---+---+---+---+---+---+---+---+---+