Repairing early UNIX file systems - Revision history

Jnc: /* Extra-large (huge) files */ Show where the '16,777,216' comes from

2023-06-12T19:43:12Z

‎Extra-large (huge) files: Show where the '16,777,216' comes from

Jnc: Extra-large files

2023-03-11T21:54:06Z

Extra-large files

Jnc: also vice versa

2023-03-11T11:33:20Z

also vice versa

Jnc: add fsdb

2023-03-07T15:05:28Z

add fsdb

Jnc: 'icheck' and 'dcheck' run order

2023-03-06T14:49:03Z

'icheck' and 'dcheck' run order

Jnc: add mention of CB-UNIX's 'check'; a few more thoughts

2023-03-05T10:32:02Z

add mention of CB-UNIX's 'check'; a few more thoughts

Jnc: /* Duplicate blocks */ listing dup block uses

2023-03-05T00:28:54Z

‎Duplicate blocks: listing dup block uses

Jnc: /* External links */ tag fsck entry

2023-03-05T00:23:17Z

‎External links: tag fsck entry

Jnc: +cat

2023-03-04T10:48:51Z

+cat

Jnc: /* Link count zero */ happens when the system was stopped with pipes open

2023-03-04T10:17:20Z

‎Link count zero: happens when the system was stopped with pipes open

@@ Line 29: / Line 29: @@
 V5 and V6 file systems are almost identical (to the point that most tools for the early versions of the file system will work on most instances of either); but V6 includes support for extra-large (huge) files. In these, the 8th block number in the inode is not the number of an indirect block, but rather that of another level of indirect block: each word in ''that'' is the number of a 'regular' indirect block.
-Without extra-large files, the largest file supported is 512*256*8=1,048,576 bytes; with extra-large files, it is somewhat larger than 256*256*512=33,554,432 bytes - but the length of a file is stored in a 24-bit field in the inode, so in practice a file in a V6 file system can be at most 16,777,216 bytes long. If the file system does not have any files longer than 512*256*7=917,504 bytes, this distinction can be ignored; all the early tools will work on either kind of early file system.
+Without extra-large files, the largest file supported is 512*256*8=1,048,576 bytes; with extra-large files, it is somewhat larger than 256*256*512=33,554,432 bytes - but the length of a file is stored in a 24-bit field in the inode, so in practice a file in a V6 file system can be at most 2^24=16,777,216 bytes long. If the file system does not have any files longer than 512*256*7=917,504 bytes, this distinction can be ignored; all the early tools will work on either kind of early file system.
 PWB/UNIX, although mostly V6, does not support extra-large files. 'fcheck' appears to half-support extra-large files ('forallblocks()' seems to handle them, but 'descend()' seems not to).
 ==Manual patching and consistency issues==

@@ Line 7: / Line 7: @@
 Note that the documentation for 'icheck' contains the following warning: "''Notice also that the words in the super-block which indicate the size of the free list and of the i-list are believed. If the super-block has been curdled these words will have to be patched.''" Early on, manual repair was often needed, but over time, this need was ameliorated.
-Different versions of UNIX came with different tools, but the [[UNIX Fifth Edition|V5]] and [[UNIX Sixth Edition|V6]] ones are generally both usable on the other. Some new tools for that generation of file system (below) arrived from [[CB-UNIX]] and [[PWB/UNIX]]. [[Unix Seventh Edition|V7]] used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk, or vice versa, will thus ''trash the [[disk]]''.
+Different versions of UNIX came with different tools, but the [[UNIX Fifth Edition|V5]] and [[UNIX Sixth Edition|V6]] ones are generally both usable on the other. (But see below.) Some new tools for that generation of file system (below) arrived from [[CB-UNIX]] and [[PWB/UNIX]]. [[Unix Seventh Edition|V7]] used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk, or vice versa, will thus ''trash the [[disk]]''.
 The tools are:
@@ Line 24: / Line 24: @@
 The CB-UNIX 'check', PWB's 'fcheck', and 'fsck' all appear to be related, but the exact lineal relationship among the three has not yet been ascertained.
 ==Manual patching and consistency issues==
@@ Line 71: / Line 79: @@
 ==External links==
-* [http://squoze.net/UNIX/v5man/man5/fs fs(V)] - short but clear detailed specification
+* [http://squoze.net/UNIX/v5man/man5/fs fs(V)] - for V5 and before; short but clear detailed specification
 * [http://squoze.net/UNIX/v5man/man5/directory directory(V)]
 * [http://squoze.net/UNIX/v1man/man1/db db(I)]

@@ Line 7: / Line 7: @@
 Note that the documentation for 'icheck' contains the following warning: "''Notice also that the words in the super-block which indicate the size of the free list and of the i-list are believed. If the super-block has been curdled these words will have to be patched.''" Early on, manual repair was often needed, but over time, this need was ameliorated.
-Different versions of UNIX came with different tools, but the [[UNIX Fifth Edition|V5]] and [[UNIX Sixth Edition|V6]] ones are generally both usable on the other. Some new tools for that generation of file system (below) arrived from [[CB-UNIX]] and [[PWB/UNIX]]. [[Unix Seventh Edition|V7]] used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk will thus ''trash the [[disk]]''.
+Different versions of UNIX came with different tools, but the [[UNIX Fifth Edition|V5]] and [[UNIX Sixth Edition|V6]] ones are generally both usable on the other. Some new tools for that generation of file system (below) arrived from [[CB-UNIX]] and [[PWB/UNIX]]. [[Unix Seventh Edition|V7]] used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk, or vice versa, will thus ''trash the [[disk]]''.
 The tools are:

@@ Line 7: / Line 7: @@
 Note that the documentation for 'icheck' contains the following warning: "''Notice also that the words in the super-block which indicate the size of the free list and of the i-list are believed. If the super-block has been curdled these words will have to be patched.''" Early on, manual repair was often needed, but over time, this need was ameliorated.
-Different versions of UNIX came with different tools, but the [[UNIX Fifth Edition|V5]] and [[UNIX Sixth Edition|V6]] ones are generally both usable on the other. [[Unix Seventh Edition|V7]] used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk will thus ''trash the [[disk]]''.
+Different versions of UNIX came with different tools, but the [[UNIX Fifth Edition|V5]] and [[UNIX Sixth Edition|V6]] ones are generally both usable on the other. Some new tools for that generation of file system (below) arrived from [[CB-UNIX]] and [[PWB/UNIX]]. [[Unix Seventh Edition|V7]] used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk will thus ''trash the [[disk]]''.
 The tools are:
@@ Line 15: / Line 15: @@
 * icheck (V6) - ensures that all blocks are assigned to at most one [[file]], and that all unused blocks are in the free list
 * dcheck (V6) - ensures that the reference count in each [[inode]] matches the number of [[directory]] entries for it
-* fcheck (V6) - apparently  product of CMU, it included both the icheck and dcheck functionality, along with other improvements
+* fcheck (CB) - included both the icheck and dcheck functionality, along with other improvements
 * fsck (V7) - a later descendant of fcheck
@@ Line 22: / Line 23: @@
 'fcheck' and its later descendant 'fsck' finally moved file system repair beyond the realm of cognoscenti manually patching some problems; they were able to repair ''most'' problems. In theory, 'fcheck' should work on V4 and V5 systems, since there are no known differences between the file systems of the three, but this has not been experimentally verified.
-The V6 'fcheck', 'fsck', and [[CB-UNIX]]'s 'check' appear to be related, but the exact lineal relationship among the three has not yet been ascertained.
+The CB-UNIX 'check', PWB's 'fcheck', and 'fsck' all appear to be related, but the exact lineal relationship among the three has not yet been ascertained.
 ==Manual patching and consistency issues==
@@ Line 77: / Line 78: @@
 * [https://minnie.tuhs.org/cgi-bin/utree.pl?file=V6/usr/man/man8/icheck.8 icheck(VIII)] - V6's 'icheck'
 * [https://minnie.tuhs.org/cgi-bin/utree.pl?file=V6/usr/man/man8/dcheck.8 dcheck(VIII)] - V6's 'dcheck'
 * [https://www.tuhs.org/Archive/Distributions/USDL/CB_Unix/cbunix_man1_01.pdf man1_01.pdf] - CB-UNIX's 'check' documentation
 * [https://www.tuhs.org/cgi-bin/utree.pl?file=V7addenda/fsck fsck] - source and documentation

@@ Line 15: / Line 15: @@
 * icheck (V6) - ensures that all blocks are assigned to at most one [[file]], and that all unused blocks are in the free list
 * dcheck (V6) - ensures that the reference count in each [[inode]] matches the number of [[directory]] entries for it
-* fcheck (V6) - apparently  product of CMU, it included both the icheck and icheck functionality, along with other improvements
+* fcheck (V6) - apparently  product of CMU, it included both the icheck and dcheck functionality, along with other improvements
 * fsck (V7) - a later descendant of fcheck
 'fcheck' and its later descendant 'fsck' finally moved file system repair beyond the realm of cognoscenti manually patching some problems; they were able to repair ''most'' problems. In theory, 'fcheck' should work on V4 and V5 systems, since there are no known differences between the file systems of the three, but this has not been experimentally verified.
@@ Line 42: / Line 44: @@
 ===Duplicate blocks===
-A block being assigned to several different files generally means that all files' contents are likely damaged. A block appearing in the free list, and a file, is also likely to have caused damage, but is easier to repair; a block appearing in the free list two or more times is similarly easy to repair. Both of these latter problems should be fixed ASAP; using the disk is likely to cause further damage to the contents.
+A block being assigned to several different files generally means that all files' contents are likely damaged. A block appearing in the free list, and a file, is also likely to have caused damage, but is easier to repair; a block appearing in the free list two or more times is similarly easy to repair. Both of these latter problems should be fixed ASAP; using the disk in ''any'' way before they are fixed is likely to cause further damage to the contents.
-The latter cases can be repaired by re-building the free list ('-s' to 'check'/'icheck'). The 'easy' way to fix the first case is i) copy the second file to somewhere else, ii) delete the original of the second file, iii) re-build the free list (because the duplicate block will now be in both the first file, and the free list), iv) examine both files, and see which one has the smashed contents.
+The latter cases can be repaired by re-building the free list ('-s' to 'check'/'icheck'). The 'easy' way to fix the first case is i) copy the second file to somewhere else (another disk, to prevent further damage on the 'problem' disk), ii) delete the original of the second file, iii) re-build the free list (because the duplicate block will now be in both the first file, and the free list), iv) examine both files, and see which one has the smashed contents.
 Note that 'check'/'icheck' will not tell you what the ''first'' file is which is using a duplicate block, because it has already forgotten that by the time it discovers the second claimant - it only keeps a bit [[array]] of 'used' blocks. Use of the '-b' option will list all uses of the block, though.

@@ Line 1: / Line 1: @@
-'''Repairing early UNIX file systems''' when they are damaged is not trivial. Early versions of [[UNIX]] had almost no tools for automagically fixing [[UNIX file system]] corruption. To do it, one needs to:
+'''Repairing early UNIX file systems''' when they are damaged is often not trivial. Early versions of [[UNIX]] had almost no tools for automagically fixing [[UNIX file system]] corruption. To do it, one needed to:
 * understand how the file system is arranged (see the link below, but it's pretty simple);
 * understand what the few available tools (dcheck; icheck; clri) do;
 * dive in!
 Different versions of UNIX came with different tools, but the [[UNIX Fifth Edition|V5]] and [[UNIX Sixth Edition|V6]] ones are generally both usable on the other. [[Unix Seventh Edition|V7]] used a different file system format from V5/V6 (they are very similar, but block numbers are 16 bits on V5/V6, and 32 bits on V7); use of V5/V6 tools on a V7 disk will thus ''trash the [[disk]]''.
@@ Line 9: / Line 11: @@
 The tools are:
 * check (V5)
 * icheck (V6) - ensures that all blocks are assigned to at most one [[file]], and that all unused blocks are in the free list
 * dcheck (V6) - ensures that the reference count in each [[inode]] matches the number of [[directory]] entries for it
-* clri (V6) - zeroes the contents of an inode
+* fcheck (V6) - apparently  product of CMU, it included both the icheck and icheck functionality, along with other improvements
 * fsck (V7) - a later descendant of fcheck
-The documentation for 'icheck' contains the following warning: "''Notice also that the words in the super-block which indicate the size of the free list and of the i-list are believed. If the super-block has been curdled these words will have to be patched.''"
+'fcheck' and its later descendant 'fsck' finally moved file system repair beyond the realm of cognoscenti manually patching some problems; they were able to repair ''most'' problems. In theory, 'fcheck' should work on V4 and V5 systems, since there are no known differences between the file systems of the three, but this has not been experimentally verified.
 ==Manual patching and consistency issues==
-As indicated, some problems will require manual patching to repair  errors; 'db' is the best tool to do this, so one will want to study up on the 'db' syntax so one knows how to use it to do that. (The '!' command is the one to examine. It's probably a good idea to practice this on an ordinary file first; one can use 'od' to see the results of one's attempts.) 'db's limited addressing capability may also require the use of 'dd' to extract a copy of the block on which one wants to operate, followed by its use again to put the repaired block back.
+As indicated, with the earlier tools, some problems will require manual patching to repair  errors; 'db' is the best tool to do this, so one will want to study up on the 'db' syntax so one knows how to use it to do that. (The '!' command is the one to examine. It's probably a good idea to practice this on an ordinary file first; one can use 'od' to see the results of one's attempts.) 'db's limited addressing capability may also require the use of 'dd' to extract a copy of the block on which one wants to operate, followed by its use again to put the repaired block back.
 If using 'db', one will have to use the non-raw version of the disk, since the raw version can only read/write complete blocks. ('raw' devices use the [[device controller]]'s [[Direct Memory Access|DMA]] capability to transfers block contents direct to and from [[buffer]]s in the [[process]]' [[address space]].) Having made changes to the 'buffered' device, one must then judiciously use 'sync' to flush the updated blocks out to the 'physical' disk.
@@ Line 42: / Line 46: @@
 The latter cases can be repaired by re-building the free list ('-s' to 'check'/'icheck'). The 'easy' way to fix the first case is i) copy the second file to somewhere else, ii) delete the original of the second file, iii) re-build the free list (because the duplicate block will now be in both the first file, and the free list), iv) examine both files, and see which one has the smashed contents.
-(Note that 'check'/'icheck' will not tell you what the ''first'' file is which is using a duplicate block, because it has already forgotten that by the time it discovers the second claimant - it only keeps a bit [[array]] of 'used' blocks.) Use of the '-b' option will list all uses of the block, though.
+Note that 'check'/'icheck' will not tell you what the ''first'' file is which is using a duplicate block, because it has already forgotten that by the time it discovers the second claimant - it only keeps a bit [[array]] of 'used' blocks. Use of the '-b' option will list all uses of the block, though.
 ===Link count too high===
-An inode with a link count higher than the number of [[hard link|links]] to it can be safely ignored temporarily. This, and similar errors below, will generally have to be corrected by hand.
+An inode with a link count higher than the number of [[hard link|links]] to it can be safely ignored temporarily. This, and similar errors below, will often have to be corrected by hand before the 'fcheck'/'fsck' era.
 ===Link count too low===
-An inode with a link count lower than the number of links to it will not cause an immediate problem (if none of the directory entries are deleted), but it should be repaired as soon as possible, to prevent a problem if that is done.
+An inode with a link count lower than the number of links to it will not cause an immediate problem (if none of the directory entries are deleted), but it should be repaired fairly quickly, to prevent a problem if that is done.
 ===Link count zero===
-An inode which is marked as 'allocated', but has a zero link count, has several possible explanations; in general, it can be safely ignored temporarily. The most likely, if on the root file system, is that the system was stopped with [[pipe]]s open. These may be cleared with 'clri', and any lost blocks retrieved with 'icheck -s'. Other than that, the inode was likely 'lost' as a result of a directory being smashed. A link to it must be created by hand, and the link count adjusted by hand; the contents of the file can then be examined, to see what it was.
+An inode which is marked as 'allocated', but has a zero link count, has several possible explanations; in general, it can be safely ignored temporarily. The most likely, if on the 'root' file system, is that the system was stopped with [[pipe]]s open. These may be cleared with 'clri', and any lost blocks retrieved with 'icheck -s'. Other than that, the inode was likely 'lost' as a result of a directory being smashed. A link to it must be created by hand, and the link count adjusted by hand; the contents of the file can then be examined, to see what it was.
 ===Unallocated inode===
-An inode which is not marked as 'allocated', but which has links to it, is problematic. The exact solution depends on the state of the contents, but an 'easy' repair is to i) delete all the directory entries which link to it, ii) use 'clri' to zero the inode, iii) re-build the free list to re-capture any 'lost' blocks.
+An inode which is not marked as 'allocated', but which has links to it, is problematic; the problem should be repaired as soon as possible. The exact solution depends on the state of the contents, but an 'easy' repair is to i) delete all the directory entries which link to it, ii) use 'clri' to zero the inode, iii) re-build the free list to re-capture any 'lost' blocks.
 <!--
 ==See also==
@@ Line 72: / Line 76: @@
 * [https://minnie.tuhs.org/cgi-bin/utree.pl?file=V6/usr/man/man8/dcheck.8 dcheck(VIII)] - V6's 'dcheck'
 * [http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/fcheck.c fcheck.c] - source
 * [https://www.tuhs.org/cgi-bin/utree.pl?file=V7addenda/fsck fsck] - source and documentation

← Older revision		Revision as of 10:48, 4 March 2023
Line 76:		Line 76:
	[[Category: File Systems]]		[[Category: File Systems]]
	[[Category: UNIX]]		[[Category: UNIX]]
		+	[[Category: UNIX Practical Guides]]