Today's article is on how to run fsck on a Linux system that won't boot.
I'm prompted to write this article because of a problem I experienced with my Devuan Daedalus 5 operating system which is based on Debian12 without systemd. Systemd is a suite of basic building blocks for a Linux system. It provides a system and service manager and starts the rest of the operating system.
My Devuan system would not boot and the file system was corrupted. I had to run fsck manually. See the screenshot below.
I apparently experienced an 'unexpected inconsistency' when my Devuan install attempted to boot.
Unexpected inconsistency is a elaborate term for a not so graceful shutdown.
Whenever there is a power surge or a power outage it can cause a unexpected inconsistency to occur.
Power interruptions can cause booting problems and may corrupt the root file system partition.
These interruptions in loss of power will cause fault to the files by an application or daemon.
There had been two surges in the power in my area while my system was up and running. I suspected that this was the culprit.
It's important here to remember a very important detail. In order to run fsck to repair a Linux file system the partition of that distribution can not be mounted.
There are a few ways to run fsck to repair a Linux file system.
You can use the downward arrow key on the keyboard to select "Advanced Options" listed in your Grub Menu for the distribution that needs repair. And proceed that way by entering "Rescue Mode".
Another way is, if you are in a dual boot or triple boot with other Linux operating systems on one machine, you can boot into one of those distributions and run fsck from the command-line.
You can also boot up a Live Linux distro on a USB thumb drive or other device and run fsck that way.
Lastly, you can schedule a fsck at boot time. To do that create a file called "forcefsck" in the root partition of your system. As root, run: touch /forcefsck. Then you can force or schedule a reboot of your system.
After your system boots run ls /forefsck and see if it still exists. IF it is 'still in place' it's a good idea to remove it otherwise, fsck will run on every system boot.
In my case I used a Live version of Linux Mint Debian Edition 6 ready to go on a USB thumb drive. It worked flawlessly. Upon running these commands (see below) I was very quickly able to reboot into the Devuan system and the error message in the screenshot above vanished.
Once the Live version of LMDE6 booted to the desktop I launched the terminal and ran:
sudo fsck -y /dev/nvme0n1p1
***Keep in mind that you need the exact path to your device and the root file system partition. If you have multiple drives than type the path and double check yourself.***
Running that command with the -y flag will say yes to each query and try to fix the file system without any interaction from the user. Without using the -y flag fsck will stop at each problem and ask you to fix it or not. Who wants redundancy when your system isn't booting and running correctly?
The output was pleasing.
int@mint:~$ sudo fsck -y /dev/nvme0n1p1
fsck from util-linux 2.38.1
e2fsck 1.47.0 (5-Feb-2023)
/dev/nvme0n1p1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found. Fix? yes
Inode 12845062 was part of the orphaned inode list. FIXED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -37735445 -(53473241--53473243)
Fix? yes
Free blocks count wrong for group #1151 (8687, counted=8688).
Fix? yes
Free blocks count wrong for group #1631 (22756, counted=22759).
Fix? yes
Free blocks count wrong (56359388, counted=56355876).
Fix? yes
Inode bitmap differences: -12589184 -12845062
Fix? yes
Free inodes count wrong for group #1536 (0, counted=1).
Fix? yes
Free inodes count wrong for group #1568 (8170, counted=8171).
Fix? yes
Free inodes count wrong (14903719, counted=14903646).
Fix? yes
/dev/nvme0n1p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p1: 300706/15204352 files (0.6% non-contiguous), 4443356/60799232 blocks
mint@mint:~$ sudo fsck -c /dev/nvme0n1p1
fsck from util-linux 2.38.1
e2fsck 1.47.0 (5-Feb-2023)
Checking for bad blocks (read-only test): done
/dev/nvme0n1p1: Updating bad block inode.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nvme0n1p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p1: 300706/15204352 files (0.6% non-contiguous), 4443356/60799232 blocks
mint@mint:~$Running fsck <or> e2fsck with the -c option invokes the badblocks program that will mark unusable parts of the disk for non-use. This way the rest of the usable disk can still be used with a lower capacity.
As always run your Linux safe and have a great weekend!
Alex