Bad Blocks HowTo, by Bruce Allen

http://smartmontools.sourceforge.net/BadBlockHowTo.txt

THIS DOCUMENT SHOWS HOW TO IDENTIFY THE FILE ASSOCIATED WITH AN
UNREADABLE DISK SECTOR, AND HOW TO FORCE THAT SECTOR TO REALLOCATE.

Assumptions: Linux OS, ext2 or ext3 file system.

Bruce Allen

Thanks to Sergey Vlasov, Theodore Ts'o, Michael Bendzick, and others
for explaining this to me. I would like to add text showing how to do
this for other file systems, in particular ReiserFS, XFS, and JFS:
please email me if you can provide this information.

NOTE: Starting with GNU coreutils release 5.3.0, dd on Linux includes
options 'iflag=direct' and 'oflag=direct'. Using these with the dd commands
below should be helpful, because adding these flags should avoid any interaction
with the block buffering IO layer in Linux and permit direct reads/writes
from the raw device. Use 'dd --help' to see if your version of dd supports
these options. If not, build the latest code from
fttp://alpha.gnu.org/gnu/coreutils.

In this example, the disk is failing self-tests at Logical Block
Address LBA = 0x016561e9 = 23421417. The LBA counts sectors in units
of 512 bytes, and starts at zero.

-----------------------------------------------------------------------------------------------
root]# smartctl -l selftest /dev/hda:

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 217 0x016561e9
-----------------------------------------------------------------------------------------------

Note that other signs that there is a bad sector on the disk can be
found in the non-zero value of the Current Pending Sector count:
-----------------------------------------------------------------------------------------------
root]# smartctl -A /dev/hda
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 1
-----------------------------------------------------------------------------------------------

First Step: We need to locate the partition on which this sector of
the disk lives:
-----------------------------------------------------------------------------------------------
root]# fdisk -lu /dev/hda

Disk /dev/hda: 123.5 GB, 123522416640 bytes
255 heads, 63 sectors/track, 15017 cylinders, total 241254720 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 63 4209029 2104483+ 83 Linux
/dev/hda2 4209030 5269319 530145 82 Linux swap
/dev/hda3 5269320 238227884 116479282+ 83 Linux
/dev/hda4 238227885 241248104 1510110 83 Linux
-----------------------------------------------------------------------------------------------

The partition /dev/hda3 starts at LBA 5269320 and extends past the
'problem' LBA. The 'problem' LBA is offset 23421417 - 5269320 =
18152097 sectors into the partition /dev/hda3.

To verify the type of the file system and the mount point, look in
/etc/fstab:
-----------------------------------------------------------------------------------------------
root]# grep hda3 /etc/fstab
/dev/hda3 /data ext2 defaults 1 2
-----------------------------------------------------------------------------------------------
You can see that this is an ext2 file system, mounted at /data.

Second Step: we need to find the blocksize of the file system
(normally 4096 bytes for ext2):
-----------------------------------------------------------------------------------------------
root]# tune2fs -l /dev/hda3 | grep Block
Block count: 29119820
Block size: 4096
-----------------------------------------------------------------------------------------------
In this case the block size is 4096 bytes.

Third Step: we need to determine which File System Block contains this
LBA. The formula is:
b = (int)((L-S)*512/B)
where:
b = File System block number
B = File system block size in bytes
L = LBA of bad sector
S = Starting sector of partition as shown by fdisk -lu
and (int) denotes the integer part.

In our example, L=23421417, S=5269320, and B=4096. Hence the
'problem' LBA is in block number
b = (int)18152097*512/4096 = (int)2269012.125
so b=2269012.

Note: the fractional part of 0.125 indicates that this problem LBA is
actually the second of the eight sectors that make up this file system
block.

Fourth Step: we use debugfs to locate the inode stored in this block,
and the file that contains that inode:
-----------------------------------------------------------------------------------------------
root]# debugfs
debugfs 1.32 (09-Nov-2002)
debugfs: open /dev/hda3
debugfs: icheck 2269012
Block Inode number
2269012 41032
debugfs: ncheck 41032
Inode Pathname
41032 /S1/R/H/714197568-714203359/H-R-714202192-16.gwf
-----------------------------------------------------------------------------------------------

In this example, you can see that the problematic file (with the mount
point included in the path) is:
/data/S1/R/H/714197568-714203359/H-R-714202192-16.gwf

To force the disk to reallocate this bad block we'll write zeros to
the bad block, and sync the disk:
-----------------------------------------------------------------------------------------------
root]# dd if=/dev/zero of=/dev/hda3 bs=4096 count=1 seek=2269012
root]# sync
-----------------------------------------------------------------------------------------------

NOTE: THIS LAST STEP HAS PERMANENTLY AND IRRETREVIABLY DESTROYED SOME
OF THE DATA THAT WAS IN THIS FILE. DON'T DO THIS UNLESS YOU DON'T
NEED THE FILE OR YOU CAN REPLACE IT WITH A FRESH OR CORRECT VERSION.

Now everything is back to normal: the sector has been reallocated.
Compare the output just below to similar output near the top of this
article:
-----------------------------------------------------------------------------------------------
root]# smartctl -A /dev/hda
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 1
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 1
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 1
-----------------------------------------------------------------------------------------------

Note: for some disks it may be necessary to update the SMART Attribute values by using
smartctl -t offline /dev/hda

The disk now passes its self-tests again:

-----------------------------------------------------------------------------------------------
root]# smartctl -t long /dev/hda [wait until test completes, then]
root]# smartctl -l selftest /dev/hda

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 239 -
# 2 Extended offline Completed: read failure 90% 217 0x016561e9
# 3 Extended offline Completed: read failure 90% 212 0x016561e9
# 4 Extended offline Completed: read failure 90% 181 0x016561e9
# 5 Extended offline Completed without error 00% 14 -
# 6 Extended offline Completed without error 00% 4 -
-----------------------------------------------------------------------------------------------

and no longer shows any offline uncorrectable sectors:

-----------------------------------------------------------------------------------------------
root]# smartctl -A /dev/hda
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 1
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 1
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
-----------------------------------------------------------------------------------------------

A SECOND EXAMPLE

On this drive, the first sign of trouble was this email from smartd:

To: ballen
Subject: SMART error (selftest) detected on host: medusa-slave166.medusa.phys.uwm.edu

This email was generated by the smartd daemon running on host:
medusa-slave166.medusa.phys.uwm.edu in the domain: master001-nis

The following warning/error was logged by the smartd daemon:
Device: /dev/hda, Self-Test Log error count increased from 0 to 1

Running smartctl -a /dev/hda confirmed the problem:

Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 80% 682 0x021d9f44

Note that the failing LBA reported is 0x021d9f44 (base 16) = 35495748 (base 10)

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 3
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 3

and one can see above that there are 3 sectors on the list of pending
sectors that the disk can't read but would like to reallocate.

The device also shows errors in the SMART error log:

Error 212 occurred at disk power-on lifetime: 690 hours
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 12 46 9f 1d e2 Error: UNC 18 sectors at LBA = 0x021d9f46 = 35495750

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Timestamp Command/Feature_Name
-- -- -- -- -- -- -- -- --------- --------------------
25 00 12 46 9f 1d e0 00 2485545.000 READ DMA EXT

Signs of trouble at this LBA may also be found in SYSLOG:

[root]# grep LBA /var/log/messages | awk '{print $12}' | sort | uniq
LBAsect=35495748
LBAsect=35495750

So I decide to do a quick check to see how many bad sectors there
really are. Using the bash shell I check 70 sectors around the trouble
area:

[root]# export i=35495730
[root]# while [ $i -lt 35495800 ]
> do echo $i
> dd if=/dev/hda of=/dev/null bs=512 count=1 skip=$i
> let i+=1
> done

35495734
1+0 records in
1+0 records out
35495735
dd: reading `/dev/hda': Input/output error
0+0 records in
0+0 records out

35495751
dd: reading `/dev/hda': Input/output error
0+0 records in
0+0 records out
35495752
1+0 records in
1+0 records out

which shows that the seventeen sectors 35495735-35495751 (inclusive)
are not readable.

Next, we identify the files at those locations. The partitioning
information on this disk is identical to the first example above, and
as in that case the problem sectors are on the third partition
/dev/hda3. So we have:
L=35495735 to 35495751
S=5269320
B=4096
so that b=3778301 to 3778303 are the three bad blocks in the file
system.

[root]# debugfs
debugfs 1.32 (09-Nov-2002)
debugfs: open /dev/hda3
debugfs: icheck 3778301
Block Inode number
3778301 45192
debugfs: icheck 3778302
Block Inode number
3778302 45192
debugfs: icheck 3778303
Block Inode number
3778303 45192
debugfs: ncheck 45192
Inode Pathname
45192 /S1/R/H/714979488-714985279/H-R-714979984-16.gwf
debugfs: quit

And finally, just to confirm that this is really the damaged file:

[root]# md5sum /data/S1/R/H/714979488-714985279/H-R-714979984-16.gwf
md5sum: /data/S1/R/H/714979488-714985279/H-R-714979984-16.gwf: Input/output error

Finally we force the disk to reallocate the three bad blocks:
[root]# dd if=/dev/zero of=/dev/hda3 bs=4096 count=3 seek=3778301
[root]# sync

We could also probably use:
[root]# dd if=/dev/zero of=/dev/hda bs=512 count=17 seek=35495735

At this point we now have:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0

which is encouraging, since the pending sectors count is now zero.
Note that the drive reallocation count has not yet increased: the
drive may now have confidence in these sectors and have decided not to
reallocate them..

A device self test:
[root#] smartctl -t long /dev/hda
(then wait about an hour) shows no unreadable sectors or errors:

Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 692 -
# 2 Extended offline Completed: read failure 80% 682 0x021d9f44

[USEFUL HINTS ADDED BY OTHERS]

---------------------------------------------------------------------------

From: Kay Diederichs

I read your badblocks-howto at
http://smartmontools.sourceforge.net/BadBlockHowTo.txt and greatly
benefitted from it. One thing that's (maybe) missing is that often the
"smartctl -t long" scan finds a bad sector which is _not_ assigned to
any file. In that case it does not help to run debugfs, or rather
debugfs reports the fact that no file owns that sector. Furthermore,
it is somewhat laborious to come up with the correct numbers for
debugfs, and debugfs is slow ...

So what I suggest in the case of presence of
Current_Pending_Sector/Offline_Uncorrectable errors is to create a
huge file on that filesystem.
dd if=/dev/zero of=/some/mount/point bs=4k
creates the file. Leave it running until the partition/filesystem is
full. This will make the disk reallocate those sectors which do not
belong to a file. Check the "smartctl -a" output after that and make
sure that the sectors are reallocated. If any remain, use the debugfs
method. Of course the usual caveats apply - back it up first, and so
on.

---------------------------------------------------------------------------

From: Frederic BOITEUX

HOW TO LOCATE AND REPAIR BAD BLOCKS ON AN LVM VOLUME

* Smartd reports an error in a short test :
-------------------------------------------

# smartctl -a /dev/hdb
...
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: read failure 90% 66 37383668

So the disk has a bad block located in LBA block 37383668

* In which physical partition is the bad block ?
------------------------------------------------

# sfdisk -lu /dev/hdb

Disk /dev/hdb: 9729 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/dev/hdb1 63 996029 995967 82 Linux swap / Solaris
/dev/hdb2 * 996030 1188809 192780 83 Linux
/dev/hdb3 1188810 156296384 155107575 8e Linux LVM
/dev/hdb4 0 - 0 0 Empty

It's in the /dev/hdb3 partition, a LVM2 partition.
From the LVM2 partition beginning, the bad block has an offset of
(37383668 - 1188810) = 36194858

We have to find in which LVM2 logical partition the block belongs to.

* In which logical partition is the bad block ?
-----------------------------------------------

*IMPORTANT* : LVM2 can use different schemes dividing its physical
partitions to logical ones : linear, striped, contiguous or
not... The following example assumes that allocation is linear !

The physical partition used by LVM2 is divided in PE (Physical Extent)
units of the same size, starting at pe_start' 512 bytes blocks from
the beginning of the physical partition.

The 'pvdisplay' command gives the size of the PE (in KB) of the
LVM partition :
# part=/dev/hdb3 ; pvdisplay -c $part | awk -F: '{print $8}'
4096

To get its size in LBA block size (512 bytes or 0.5 KB), we multiply this
number by 2 : 4096 * 2 = 8192 blocks for each PE.

To find the offset from the beginning of the physical partition is a
bit more difficult : if you have a recent LVM2 version, try :
# pvs -o+pe_start $part

Either, you can look in /etc/lvm/backup :
# grep pe_start $(grep -l $part /etc/lvm/backup/*)
pe_start = 384

Then, we search in which PE is the badblock, calculating the PE rank
in which the faulty block of the partition is :
physical partition's bad block number / sizeof(PE) =
36194858 / 8192 = 4418.3176

So we have to find in which LVM2 logical partition is used the PE
number 4418 (count starts from 0) :
# lvdisplay --maps |egrep 'Physical|LV Name|Type'
LV Name /dev/WDC80Go/racine
Type linear
Physical volume /dev/hdb3
Physical extents 0 to 127
LV Name /dev/WDC80Go/usr
Type linear
Physical volume /dev/hdb3
Physical extents 128 to 1407
LV Name /dev/WDC80Go/var
Type linear
Physical volume /dev/hdb3
Physical extents 1408 to 1663
LV Name /dev/WDC80Go/tmp
Type linear
Physical volume /dev/hdb3
Physical extents 1664 to 1791
LV Name /dev/WDC80Go/home
Type linear
Physical volume /dev/hdb3
Physical extents 1792 to 3071
LV Name /dev/WDC80Go/ext1
Type linear
Physical volume /dev/hdb3
Physical extents 3072 to 10751
LV Name /dev/WDC80Go/ext2
Type linear
Physical volume /dev/hdb3
Physical extents 10752 to 18932

So the PE #4418 is in the /dev/WDC80Go/ext1 LVM logical partition.

* Size of logical block of filesystem on /dev/WDC80Go/ext1 :
------------------------------------------------------------

It's a ext3 fs, so I get it like this :
# dumpe2fs /dev/WDC80Go/ext1 | grep 'Block size'
dumpe2fs 1.37 (21-Mar-2005)
Block size: 4096

* bad block number for the filesystem :
---------------------------------------

The logical partition begins on PE 3072 :
(# PE's start of partition * sizeof(PE)) + parttion offset[pe_start] =
(3072 * 8192) + 384 = 25166208
512b block of the physical partition, so the bad block number for the
filesystem is :
(36194858 - 25166208) / (sizeof(fs block) / 512)
= 11028650 / (4096 / 512) = 1378581.25

* Test of the fs bad block :

dd if=/dev/WDC80Go/ext1 of=block1378581 bs=4096 count=1 skip=1378581

If this dd command succeeds, without any error message in console or
syslog, then the block number calculation is probably wrong ! *Don't*
go further, re-check it and if you don't find the error, please
renunce !

* Search / correction follows the same scheme as for simple
partitions :

- find possible impacted files with debugfs (icheck ,
then ncheck ).

- reallocate bad block writing zeros in it, *using the fs block size* :

dd if=/dev/zero of=/dev/WDC80Go/ext1 count=1 bs=4096 seek=1378581

Et voilà !

---------------------------------------------------------------------------

This document is version $Id: BadBlockHowTo.txt,v 1.9 2006/06/12 02:16:50 ballen4705 Exp $
It is Copyright Bruce Allen (2004-6) and distributed under GPL2.

Dear community, What is the

Dear community,

What is the sequence of steps, if bad block is not a data node, bit inode or part of any other FS structures? How to tell FS to free/reallocate this block?

Well, that depends on the

Well, that depends on the filesystem.

EXT3/4 filesystems have many backup superblocks and you can mount the filesystem using one of them. NTFS has only one backup and you can use Testdisk to restore that.

Either way, I would make an image and work on a copy of the image.

How to you tell the FS to fix the problem? Use fsck or chkdisk.