Repairing early UNIX file systems

Repairing early UNIX file systems when they are damaged is often not trivial. Early versions of UNIX had almost no tools for automagically fixing UNIX file system corruption. To do it, one needed to:

• understand how the file system is arranged (see the link below, but it's pretty simple);
• understand what the few available tools (dcheck; icheck; clri) do;
• dive in!

Note that the documentation for 'icheck' contains the following warning: "Notice also that the words in the super-block which indicate the size of the free list and of the i-list are believed. If the super-block has been curdled these words will have to be patched." Early on, manual repair was often needed, but over time, this need was ameliorated.

Different versions of UNIX came with different tools, but the V5 and V6 ones are generally both usable on the other. (But see below.) Some new tools for that generation of file system (below) arrived from CB-UNIX and PWB/UNIX. V7 used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk, or vice versa, will thus trash the disk.

The tools are:

• clri - zeroes the contents of an inode
• check (V5)
• icheck (V6) - ensures that all blocks are assigned to at most one file, and that all unused blocks are in the free list
• dcheck (V6) - ensures that the reference count in each inode matches the number of directory entries for it
• fcheck (CB) - included both the icheck and dcheck functionality, along with other improvements
• fsdb (PWB) - file system debugger, a manual tool to assist with file system damage repair
• fsck (V7) - a later descendant of fcheck

When using 'icheck' and 'dcheck', one will want to run 'icheck' first, and fix any immediate problems revealed by that (see below for problem severity ratings), before moving on to 'dcheck' and the problems it can reveal.

'fcheck' and its later descendant 'fsck' finally moved file system repair beyond the realm of cognoscenti manually patching some problems; they were able to repair most problems. In theory, 'fcheck' should work on V4 and V5 systems, since there are no known differences between the file systems of the three, but this has not been experimentally verified.

The CB-UNIX 'check', PWB's 'fcheck', and 'fsck' all appear to be related, but the exact lineal relationship among the three has not yet been ascertained.

Extra-large (huge) files

V5 and V6 file systems are almost identical (to the point that most tools for the early versions of the file system will work on most instances of either); but V6 includes support for extra-large (huge) files. In these, the 8th block number in the inode is not the number of an indirect block, but rather that of another level of indirect block: each word in that is the number of a 'regular' indirect block.

Without extra-large files, the largest file supported is 512*256*8=1,048,576 bytes; with extra-large files, it is somewhat larger than 256*256*512=33,554,432 bytes - but the length of a file is stored in a 24-bit field in the inode, so in practice a file in a V6 file system can be at most 2^24=16,777,216 bytes long. If the file system does not have any files longer than 512*256*7=917,504 bytes, this distinction can be ignored; all the early tools will work on either kind of early file system.

PWB/UNIX, although mostly V6, does not support extra-large files. 'fcheck' appears to half-support extra-large files ('forallblocks()' seems to handle them, but 'descend()' seems not to).

Manual patching and consistency issues

As indicated, with the earlier tools, some problems will require manual patching to repair errors; 'db' is the best tool to do this, so one will want to study up on the 'db' syntax so one knows how to use it to do that. (The '!' command is the one to examine. It's probably a good idea to practice this on an ordinary file first; one can use 'od' to see the results of one's attempts.) 'db's limited addressing capability may also require the use of 'dd' to extract a copy of the block on which one wants to operate, followed by its use again to put the repaired block back.

If using 'db', one will have to use the non-raw version of the disk, since the raw version can only read/write complete blocks. ('raw' devices use the device controller's DMA capability to transfers block contents direct to and from buffers in the process' address space.) Having made changes to the 'buffered' device, one must then judiciously use 'sync' to flush the updated blocks out to the 'physical' disk.

Contrariwise, there are cases where a tool has operated on the 'raw' device (such as use of 'icheck -s' to re-build the free list), and one has to ensure that the system does not overwrite one's repair by attempting to flush the 'bad' buffered, in-core copy of the changed block(s)' contents out to the disk. In such cases, one will have to stop the machine as soon as the operation is complete, and re-boot it. (There are some corner cases where data is stored elsewhere in the operating system, such as when one is patching the inode of an open file, but these are ignored for the moment, as they are rarely encountered.)

Many of these issues can be bypassed if one has another bootable disk that one can boot from; the damaged disk can then be examined and repaired at leisure, without it being 'mounted' (so the system will not have its own ideas of what the contents of the disk are). If patching by hand, which requires use of the non-raw disk, one still has to flush those changes out, but there will be no issues with the system having a contrary idea of the state of the disk.

Error types

A few words about common error types, including ones which can be safely ignored, and how to fix the ones which cannot be so disdained. It is possible that a damaged file system will contain more than one of these; in such cases, repair them one at a time (the worst one first), and check after each repair, since the repair may have created other problems (e.g. lost blocks after an operation which requires clearing an inode).

Lost blocks

A block which is not in any file, or the free list, can be safely ignored temporarily.

Duplicate blocks

A block being assigned to several different files generally means that all files' contents are likely damaged. A block appearing in the free list, and a file, is also likely to have caused damage, but is easier to repair; a block appearing in the free list two or more times is similarly easy to repair. Both of these latter problems should be fixed ASAP; using the disk in any way before they are fixed is likely to cause further damage to the contents.

The latter cases can be repaired by re-building the free list ('-s' to 'check'/'icheck'). The 'easy' way to fix the first case is i) copy the second file to somewhere else (another disk, to prevent further damage on the 'problem' disk), ii) delete the original of the second file, iii) re-build the free list (because the duplicate block will now be in both the first file, and the free list), iv) examine both files, and see which one has the smashed contents.

Note that 'check'/'icheck' will not tell you what the first file is which is using a duplicate block, because it has already forgotten that by the time it discovers the second claimant - it only keeps a bit array of 'used' blocks. Use of the '-b' option will list all uses of the block, though.

Link count too high

An inode with a link count higher than the number of links to it can be safely ignored temporarily. This, and similar errors below, will often have to be corrected by hand before the 'fcheck'/'fsck' era.

Link count too low

An inode with a link count lower than the number of links to it will not cause an immediate problem (if none of the directory entries are deleted), but it should be repaired fairly quickly, to prevent a problem if that is done.

Link count zero

An inode which is marked as 'allocated', but has a zero link count, has several possible explanations; in general, it can be safely ignored temporarily. The most likely, if on the 'root' file system, is that the system was stopped with pipes open. These may be cleared with 'clri', and any lost blocks retrieved with 'icheck -s'. Other than that, the inode was likely 'lost' as a result of a directory being smashed. A link to it must be created by hand, and the link count adjusted by hand; the contents of the file can then be examined, to see what it was.

Unallocated inode

An inode which is not marked as 'allocated', but which has links to it, is problematic; the problem should be repaired as soon as possible. The exact solution depends on the state of the contents, but an 'easy' repair is to i) delete all the directory entries which link to it, ii) use 'clri' to zero the inode, iii) re-build the free list to re-capture any 'lost' blocks.

External links

• fs(V) - for V5 and before; short but clear detailed specification
• fs(V) - for V6; includes extra-large files
• directory(V)
• db(I)
• clri(VIII)
• check(VIII) - V5's 'check'
• icheck(VIII) - V6's 'icheck'
• dcheck(VIII) - V6's 'dcheck'
• man1_01.pdf - CB-UNIX's 'check' documentation
• fcheck.c - source
• fsdb(VIII) -documentation
• fsdb.c - source
• fsck - source and documentation