Differences between revisions 2 and 3

FreeBSD Notes

Contents

FreeBSD Notes
Restoring Partition Tables

Restoring Partition Tables

Following occurred on 16-17 Feb-2012.

This section describes the restoration of the partition table on zg-1.softxs.ch, a FreeBSD 9.0 system located at the Datawire data center in Cham. The partition table was corrupted after the system was shutdown and restarted.

The reason for the restart was that the IP addresses of the machine were changed:

94.231.80.205 --> 94.231.88.100
94.231.80.206 --> 94.231.88.101

Events on zg-1.softxs.ch leading the problem:

The system was shutdown, without manually un-mounting the encrypted vol002 file system
After over 60 seconds the power was turned off. We did not have monitor connected and we were unable to see in the system was really stopped
The system was turned on again
The system booted but reported a file system error and went into single user mode
Edited the /etc/fstab and commented out the entry for vol002
Rebooted the system fron the console
The system cam up without problems, but the vol002 and vol003 file systems were not mounted

I left the data center and when back on line remotely logged into to zg-1.softxs.ch. Discovered the following:

vol001 could be mounted
There were no /dev files present for the vol002 or vol003 file systems

The /etc/fstab contains the following:

cat /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#

/dev/mirror/gm0s1b      none            swap    sw              0       0

/dev/mirror/gm0s1a      /               ufs     rw              1       1
/dev/mirror/gm0s1e      /tmp            ufs     rw              2       2
/dev/mirror/gm0s1f      /usr            ufs     rw              2       2
/dev/mirror/gm0s1d      /var            ufs     rw              2       2

/dev/mirror/gm0s1g      /e/vol001       ufs     rw              2       2

# Note: vol002 is gbde encrypted and must be mounted manually

#/dev/mirror/gm0s2d     /e/vol002       ufs     rw              2       2
/dev/mirror/gm0s2e      /e/vol003       ufs     rw              2       2

# -- CD-ROM

/dev/acd0               /cdrom          cd9660  ro,noauto       0       0

Tried to mount the encrypted vol002 file system:

root@zg-1 # ./mount_gbde.sh vol002
/mount_gbde.sh: mount point: /e/vol002
./mount_gbde.sh: device file: gm0s2d
./mount_gbde.sh: attaching: gm0s2d
Enter passphrase:
gbde: Attach to mirror/gm0s2d failed: Provider not found: "mirror/gm0s2d"
./mount_gbde.sh: ERROR: file not found: /dev/mirror/gm0s2d.bde

The /dev entry for the file system was missing, as was that for vol003. Display the disk partitions:

root@zg-1 # gpart show
=>        63  3907029042  mirror/gm0  MBR  (1.8T)
          63  1363148577           1  freebsd  [active]  (650G)  <--- This is vol001
  1363148640  2543880528           2  freebsd  [active]  (1.2T)  <--- Is this the vol002 & vol003?

See what's in /dev

root@zg-1 # ls -1 /dev/mirror/
gm0
gm0s1         <-- has a label that seems OK
gm0s1a
gm0s1b
gm0s1d
gm0s1e
gm0s1f
gm0s1g
gm0s2         <-- has no label. And no file systems found. This is where vol002 and vol003 should be

Notes:

gm0 is the entire disk (which is actually a mirrored device supported by two physical disks)
gm0s1 and gm0s2 are FreeBSD slices (which are actually DOS partitions)
gm0s1a to gm0s1g are FreeBSD partitions

The gm0s1 devices are the root, swap, /var, /tmp and /usr file systems. There should also be two gm0s2 device files. But they were missing.

Check the BSD disk labels:

root@zg-1 # bsdlabel /dev/mirror/gm0s1 # /dev/mirror/gm0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 2097152 0 4.2BSD 0 0 0 b: 8388608 2097152 swap c: 1363148577 0 unused 0 0 # "raw" part, don't edit d: 20971520 10485760 4.2BSD 0 0 0 e: 20971520 31457280 4.2BSD 0 0 0 f: 52428800 52428800 4.2BSD 0 0 0 g: 1258290977 104857600 4.2BSD 0 0 0 root@zg-1 # bsdlabel /dev/mirror/gm0s2 bsdlabel: /dev/mirror/gm0s2: no valid label found

Note that gm0s2 appear to have no partitions defined on it. This is where vol002 and vol003 file systems should be.

Install the scan_ffs to investigate what's there and see if anything can be recovered.

cd /usr/ports/sysutils/scan_ffs make install

This didn't work as the ports tree was not up to date. Had to download the following distfile by hand scan_ffs-1.2.tar.bz2, which I found at:

http://www.filewatcher.com/m/scan_ffs-1.2.tar.bz2.5119-0.html

Copied the downloaded file to /usr/ports/diskfiles, and ran the make install again.

Run scan_ffs on gm0s2 slice, which took a long time (nearly 8 hours):

root@zg-1 # date Sat Feb 16 00:05:15 CET 2013 root@zg-1 # scan_ffs -l /dev/ad2 ; date X: 2097152 63 4.2BSD 2048 16384 0 # / X: 20971520 10485823 4.2BSD 2048 16384 0 # /var X: 20971520 31457343 4.2BSD 2048 16384 0 # /tmp X: 52428800 52428863 4.2BSD 2048 16384 0 # /usr X: 1258290976 104857663 4.2BSD 2048 16384 0 # /e/vol001 X: 1275100000 2631925600 4.2BSD 2048 16384 0 # /e/vol003 scan_ffs: read: Input/output error Sat Feb 16 07:52:12 CET 2013

Notes

The partition vol003 was detected, but not vol002
The output, according to the scan_ffs man page, is in a format that can be read by the bsdlabel command

This result indicates that at least some the data in vol002 and vol003 file system is probably recoverable.

Searching in my notes I found of the output from the bsdlabel command, which I had made when I was reconfiguring the vol003 partition (to reformat it and make it a non-encrypted partition).

root@zg-1 # bsdlabel mirror/gm0s1 # /dev/mirror/gm0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 2097152 0 4.2BSD 0 0 0 b: 8388608 2097152 swap c: 1363148577 0 unused 0 0 # "raw" part, don't edit d: 20971520 10485760 4.2BSD 0 0 0 e: 20971520 31457280 4.2BSD 0 0 0 f: 52428800 52428800 4.2BSD 0 0 0 g: 1258290977 104857600 4.2BSD 0 0 0 # <-- /e/vol001 root@zg-1 # bsdlabel mirror/gm0s2 # /dev/mirror/gm0s2: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 2543880528 0 unused 0 0 # "raw" part, don't edit d: 1268776960 0 4.2BSD 0 0 0 # <-- /e/vol002 e: 1275103568 1268776960 4.2BSD 0 0 0 # <-- /e/vol003

As mentioned above, the output of the scan_ffs command can be used as input for updating (re-setting) the disk label. This is explained further in the following web page:

http://mikuru.tw/wordpress/archives/468

I prepared a text file with the offset and size information for partitions vol002 and vol003, taken from the output of the bsdlabel command listed above:

root@zg-1 # cat /tmp/vol-label d: 1268776960 0 4.2BSD 0 0 0 # /e/vol002 (605 GB) e: 1275103568 1268776960 4.2BSD 0 0 0 # /e/vol003 (614 GB)

Update the disk label:

root@zg-1 # bsdlabel -R /dev/mirror/gm0s2 /tmp/vol-label

The command completed without error and devices files gm0s2d and gm0s2e immediately appeared in the /dev directory

gm0s2d - vol002 encrypted file system
gm0s2e - vol003 normal file system

Try and mount vol003, but before mounting, run a file system check:

root@zg-1 # fsck /dev/mirror/gm0s2e fsck: Could not determine filesystem type

This failed. But I tried mounting the file vol003 system anyways:

root@zg-1 # mount /e/vol003

It worked! The file system was there, but empty. Which was correct.

Try mounting the encrypted file system vol002:

root@zg-1 # ./mount_gbde.sh vol002 ./mount_gbde.sh: mount point: /e/vol002 ./mount_gbde.sh: device file: gm0s2d ./mount_gbde.sh: attaching: gm0s2d Enter passphrase: ./mount_gbde.sh: found file: /dev/mirror/gm0s2d.bde ./mount_gbde.sh: checking: gm0s2d.bde /dev/mirror/gm0s2d.bde: 338185 files, 269310313 used, 35520820 free (42276 frags, 4434818 blocks, 0.0% fragmentation) ./mount_gbde.sh: mounting: gm0s2d.bde ./mount_gbde.sh: /e/vol002 mounted successfully from gm0s2d.bde Filesystem Size Used Avail Capacity Mounted on /dev/mirror/gm0s2d.bde 581G 514G 62G 89% /e/vol002

It worked! But it took a long time for the file system check to complete (about 20 minutes).

Lessons learned:

We should probably unmount the vol002 partition by hand before shutting down the system
Make sure we have a copy of the output of the bsdlabel for all disks on all server command saved in a safe place

-Deletions are marked like this.
+Additions are marked like this.
 Line 7:
-Following occurred on 16-Feb-2012.
+Following occurred on 16-17 Feb-2012.
 Line 30:
 . There were no /dev files present for the vol002 or vol003 file systems
 Line 72:
-The /dev entry for the file system was missing, as was that for vol003.
+The /dev entry for the file system was missing, as was that for vol003. Display the disk partitions:
 Line 88:
-s1b
+gm0s1b
 Line 95:
+Notes:



  1. '''gm0''' is the entire disk (which is actually a mirrored device supported by two physical disks)

  1. '''gm0s1''' and '''gm0s2''' are FreeBSD slices (which are actually DOS partitions)

  1. '''gm0s1a'' to '''gm0s1g''' are FreeBSD partitions
-Line 117:
+Line 123:
-Note that '''gm0s2''' appear to have no partitions defined on it. This is where vol002 and vol003 should be.
+Note that '''gm0s2''' appear to have no partitions defined on it. This is where vol002 and vol003 file systems should be.
-Line 126:
+Line 132:
 This didn't work as the ports tree was not up to date. Had to download the following distfile by hand '''scan_ffs-1.2.tar.bz2''', which I found at:
-Line 130:
+Line 136:
-Run '''scan_ffs''' on '''gm0s2''', which took a long time (nearly 8 hours):
+Copied the downloaded file to /usr/ports/diskfiles, and ran the make install again.



Run '''scan_ffs''' on '''gm0s2''' slice, which took a long time (nearly 8 hours):
-Line 136:
+Line 144:
 X: 2097152 63 4.2BSD 2048 16384 0 # /
-Line 148:
+Line 156:
-. vol003 was detected, but not vol002
+. The partition vol003 was detected, but not vol002
-Line 153:
+Line 161:
-Searching thought my notes I found of the output from the '''bsdlabel''' command, made when I was reconfiguring the vol003 partition (to reformat it and make it a non-encrypted partition).
+Searching in my notes I found of the output from the '''bsdlabel''' command, which I had made when I was reconfiguring the vol003 partition (to reformat it and make it a non-encrypted partition).
-Line 178:
+Line 186:
-As mentioned above, the output of the scan_ffs command can be used as input for updating (setting) the disk' '''label'''. This is explained in the following web page:
+As mentioned above, the output of the scan_ffs command can be used as input for updating (re-setting) the disk label. This is explained further in the following web page:
-Line 182:
+Line 190:
-I prepared a text file with the offset and size information for vol002 and vol003, taken from the output of the bsdlabel command shown above:
+I prepared a text file with the offset and size information for partitions vol002 and vol003, taken from the output of the bsdlabel command listed above:
-Line 196:
+Line 204:
-The command completed without error and devices files '''gm0s2d''' and '''gm0s2e''' appeared in the /dev directory
+The command completed without error and devices files '''gm0s2d''' and '''gm0s2e''' immediately appeared in the /dev directory
-Line 201:
+Line 209:
-Try and mount vol003, but before mounting run a file system check:
+Try and mount vol003, but before mounting, run a file system check:
-Line 208:
+Line 216:
-This failed. But I tried mounting the file vol003 system anyways.
+This failed. But I tried mounting the file vol003 system anyways:
-Line 223:
+Line 231:
 Enter passphrase:
-Line 226:
+Line 234:
 /dev/mirror/gm0s2d.bde: 338185 files, 269310313 used, 35520820 free
-Line 234:
+Line 242:
-Worked, but it took a long time for the file system check to complete (about 20 minutes).
+It worked! But it took a long time for the file system check to complete (about 20 minutes).