FreeBSD Notes

Restoring Partition Tables

Following occurred on 16-17 Feb-2012.

This section describes the restoration of the partition table on zg-1.softxs.ch, a FreeBSD 9.0 system located at the Datawire data center in Cham. The partition table was corrupted after the system was shutdown and restarted.

The reason for the restart was that the IP addresses of the machine were changed:

  1. 94.231.80.205 --> 94.231.88.100

  2. 94.231.80.206 --> 94.231.88.101

Events on zg-1.softxs.ch leading the problem:

  1. The system was shutdown, without manually un-mounting the encrypted vol002 file system
  2. After over 60 seconds the power was turned off. We did not have monitor connected and we were unable to see in the system was really stopped
  3. The system was turned on again
  4. The system booted but reported a file system error and went into single user mode
  5. Edited the /etc/fstab and commented out the entry for vol002
  6. Rebooted the system fron the console
  7. The system cam up without problems, but the vol002 and vol003 file systems were not mounted

I left the data center and when back on line remotely logged into to zg-1.softxs.ch. Discovered the following:

  1. vol001 could be mounted
  2. There were no /dev files present for the vol002 or vol003 file systems

The /etc/fstab contains the following:

Tried to mount the encrypted vol002 file system:

The /dev entry for the file system was missing, as was that for vol003. Display the disk partitions:

See what's in /dev

Notes:

  1. gm0 is the entire disk, which is actually a mirrored device supported by two physical disks ad0 and ad2

  2. gm0s1 and gm0s2 are FreeBSD slices (which are actually DOS partitions)

  3. gm0s1a to gm0s1g are FreeBSD partitions, which each contain a file system. E.g. the root, swap, /var, /tmp, /usr and /e/vol001 file systems

  4. There should also be two partitions associated with gm0s2 device files. E.g. gm0s2d and gm0s2de. But these devies were not present in /dev

Check the BSD disk labels:

Note that gm0s2 appear to have no partitions defined on it. This is where vol002 and vol003 file systems should be.

Install the scan_ffs to investigate what's there and see if anything can be recovered.

This didn't work as the ports tree was not up to date. Had to download the following distfile by hand scan_ffs-1.2.tar.bz2, which I found at:

Copied the downloaded file to /usr/ports/diskfiles, and ran the make install again.

Run scan_ffs on the gm0s2 slice, which took a long time (nearly 8 hours):

Notes

  1. The partition vol003 was detected, but not the partition vol002
  2. The output format, according to the scan_ffs man page, is in a format that can be read as input by the bsdlabel command

This result indicates that at least some the data in vol002 and vol003 file system is probably recoverable.

Searching in my notes I found of the output from the bsdlabel command, which I had made when I was reconfiguring the vol003 partition (to reformat it and make it a non-encrypted partition).

As mentioned above, the output of the scan_ffs command can be used as input for updating (re-setting) the disk label. This is explained further in the following web page:

I prepared a text file with the offset and size information for partitions vol002 and vol003, taken from the output of the bsdlabel command listed above:

Update the disk label:

The command completed without error and devices files gm0s2d and gm0s2e immediately appeared in the /dev directory

Try and mount vol003, but before mounting it, run a file system check:

This failed. But I tried mounting the file vol003 system anyways:

It worked! The file system was there, but empty. Which was correct.

Try mounting the encrypted file system vol002:

It worked! But it took a long time for the file system check to complete (about 20 minutes).

Lessons learned:

  1. We should probably unmount the vol002 partition by hand before shutting down the system
  2. Make sure we have a copy of the output of the bsdlabel command for all disks on all server command saved in a safe place

FreeBsdNotes (last edited 2013-02-22 11:23:54 by 10)

Copyright 2008-2014, SoftXS GmbH, Switzerland