A good, but bad disk... badblocks alone can never be trusted.

[tr0910]

A number of us will soon have cheap BF drives needing extensive testing and preclearing.  We volunteer to sacrifice our new drives on the altar of knowledge. :-)

[Joe L.]

I think we are missing the point.

If the read fails, it returns -1, This is normal system call protocol.

Not according to POSIX.  You can return a partial read, and set "errno" and not return -1.

Somehow the read or retry of the read is succeeding. Otherwise the read would return -1.

Perhaps many sectors are read and returned in the single "read" prior to the un-readable one.  Remember, I was reading 1024 * 65536 bytes at a time.

I know there are errors in the syslog, Somewhere else something is going wrong or the read is succeeding through retries.

 

We seem to be blaming badblocks for not detecting the read failure, when in fact, if it failed, it would handle it.

If the sectors were pending re-allocation, and the initial "Write" re-allocated them, and the subsequent "Read" verified what was re-allocated, but un-covered additional un-readable sectors, but returned zeros (and zeros were what was being tested) or the read buffer being used still had the prior "read" contents...

I've run this tool on MANY drives that had bad sectors, It finds them and logs them.

If possible the second, third or fourth pass, reallocates them.

 

 

My friend gave me boxes and boxes of hard drives. This tool has worked to ween out the bad ones or make questionable ones useful again.  After the sectors have been reallocated on one of the 4 passes, I do a SMART long test and the drives pass.

 

 

I'm not saying this is the be all / end all tool to detect and/or fix errors, but I've tested on box loads of drives and it works well.

 

 

I would ask Joe L, to try badblocks again without the speed enhancements and see what happens. I know it would take a long time, but that's what I've used and it found errors and/or reallocated them.

I am running another 4-value read/write test.  I did not specify any number of bytes, or block size.  It is going more slowly.  The man page says it uses 1024 bytes and reads 64 blocks at a time by default if I do not set any options.    other than it taking a lot longer, I do not expect any differences.    I've not seen any new un-readable sectors in a few cycles, and based on your experience, I should be able to rely on the drive for my data.

 

 

NOT 

 

Joe L.

[WeeboTech]

I've not seen any new un-readable sectors in a few cycles, and based on your experience, I should be able to rely on the drive for my data.

 

NOT 

 

Joe L.

 

 

Let's be real here. There's more to it then that.

 

 

Based on my experience, if SMART says FAILED or FAILING_NOW, RMA the drive no matter what.

Once SMART says those two words, the vendor deals with an in warranty replacement.

Based on my experience, if there are ANY pending sectors, the drive should not be used for unRAID or you may fail to rebuild another drive during a recovery situation.

[WeeboTech]

I think we are missing the point.

If the read fails, it returns -1, This is normal system call protocol.

Not according to POSIX.  You can return a partial read, and set "errno" and not return -1.

We can enhance the software by clearning errno before the read and testing it after if you think it's getting a partial read.

We can also clear the buffer with something random so that a partial read would not match.

 

 

This is a most interesting case with this drive. Maybe you should hold off on RMA'ing it for a while.

[tr0910]

10 new 3TB drives to be tested and precleared.  Can I help with this process??

[WeeboTech]

10 new 3TB drives to be tested and precleared.  Can I help with this process??

 

 

I would suggest capturing a smartctl log before anything.

do a smart -t short

do a smart -t long (this will take along time, probably 3-5 hours alone)

Then do a badblocks 4 pass write test. (this could take from 24-72 hours).

Capture a smartctl log after everything.

diff the two logs.

 

 

Note the number of reallocated sectors, pending sectors and unreadable sectors.

If there are any pending sectors the drive is not a good candidate for unRAID.

 

 

The preclear tool does most of this in an automated fashion except for the smart -tshort or long and 4 pass write pattern test.

 

 

[UhClem]

Well, I've been traveling/visiting and ate a lot of turkey. Now I'm back, and I have to eat some crow .

 

I was wrong in my assessment of badblocks' behavior regarding the drive errors that JoeL reported/documented.Please note that I have corrected (and mostly retracted) my criticism of badblocks in my earlier post [link]. Summary: drive reports UNCorrectable error to kernel driver (which records it in syslog), but kernel only reports that error to the calling read() (from badblocks) IF the error persists through MAX_RETRIES+1 (6) attempts.

 

Apologies for "The sky is falling ..." alarm. But I hope some of you will have benefitted from the ensuing discussion.

 

I really should not have been "fooled" since the syslog extract that JoeL posted does tell the whole story. But, because every one of those flaky sectors was reported in the same way, as a one-time UNC error (with implied success upon the first retry), I overlooked the non-presence of retry errors along with the non-presence of a final more-detailed error report, for each of the sectors. Sometimes the most important detail is divulged because it is not presented.

 

--UhClem

 

[WeeboTech]

So badblocks is not detecting the error, because the kernel retried and the read finally succeeded.

Looks like we need to check the smart values in addition to the badblocks values.

[UhClem]

So badblocks is not detecting the error, because the kernel retried and the read finally succeeded.

[Hi there--I wish you the best in re-establishing normalcy in your day-to-day.]

 

Correct. But, two things of note: (1) In the scenario being explored here, every one of those UNC sector failures actually succeeded on the first re-try; not only did they not go the whole (1+5=6) attempts [and result in an error return] but they all succeeded on the second attempt (first re-try); that could be insightful "forensics" if one wanted to do an in-depth diagnosis.

(2) Realize that the drive itself makes 10-20+ attempts to read (with valid ECC) the recalcitrant sector before giving the UNC error [to the driver].

Looks like we need to check the smart values in addition to the badblocks values.

Yes, but you want to be "smarter" about how you use badblocks and SMART reports, so that you don't "destroy" valuable diagnostic information in the process. [i don't want to tell you exactly what to do, because I believe you can figure it out, and you should not be deprived of the challenge, and the satisfaction.] Once you do figure out that one, see if you can envision a simple embellishment to badblocks (about 10 lines of C) that will facilitate even further improvement to this endeavor. [i'll be suggesting this enhancement to badblocks' author.]

 

-- UhClem  "If you keep getting sick, but each time seem to completely recover, are you really a healthy individual?"

 

[WeeboTech]

So badblocks is not detecting the error, because the kernel retried and the read finally succeeded.

[Hi there--I wish you the best in re-establishing normalcy in your day-to-day.]

 

Correct. But, two things of note: (1) In the scenario being explored here, every one of those UNC sector failures actually succeeded on the first re-try; not only did they not go the whole (1+5=6) attempts [and result in an error return] but they all succeeded on the second attempt (first re-try); that could be insightful "forensics" if one wanted to do an in-depth diagnosis.

(2) Realize that the drive itself makes 10-20+ attempts to read (with valid ECC) the recalcitrant sector before giving the UNC error [to the driver].

Looks like we need to check the smart values in addition to the badblocks values.

Yes, but you want to be "smarter" about how you use badblocks and SMART reports, so that you don't "destroy" valuable diagnostic information in the process. [i don't want to tell you exactly what to do, because I believe you can figure it out, and you should not be deprived of the challenge, and the satisfaction.] Once you do figure out that one, see if you can envision a simple embellishment to badblocks (about 10 lines of C) that will facilitate even further improvement to this endeavor. [i'll be suggesting this enhancement to badblocks' author.]

 

-- UhClem  "If you keep getting sick, but each time seem to completely recover, are you really a healthy individual?"

 

 

Please save me the CPU cycles and provide the suggestion. LOL!  I really need to think of what needs to be purchased to get back on track. So i'll look for any shortcuts you can provide.

[Barziya]

I wrote all zeros to a disk, then intentionally wrote a few non-zero bytes here and there.  To my dismay, a read-only badblocks pattern test for zeros passed with flying colors:

Checking for bad blocks in read-only mode
Testing with pattern 0x00: done
Pass completed, 0 bad blocks found. (0/0/0 errors)

Yet, I could hexdump the disk and see the non-zero bytes.

 

[WeeboTech]

I wrote all zeros to a disk, then intentionally wrote a few non-zero bytes here and there.  To my dismay, a read-only badblocks pattern test for zeros passed with flying colors:

Checking for bad blocks in read-only mode
Testing with pattern 0x00: done
Pass completed, 0 bad blocks found. (0/0/0 errors)

Yet, I could hexdump the disk and see the non-zero bytes.

 

 

This is an interesting issue. Perhaps the pattern is not actually tested/compared and is only returning success if the read returns success.  Not what I would expect for a read test with a pattern.

 

What command line options were used?

 

 

[WeeboTech]

I wrote all zeros to a disk, then intentionally wrote a few non-zero bytes here and there.  To my dismay, a read-only badblocks pattern test for zeros passed with flying colors:

Checking for bad blocks in read-only mode
Testing with pattern 0x00: done
Pass completed, 0 bad blocks found. (0/0/0 errors)

Yet, I could hexdump the disk and see the non-zero bytes.

 

This is an interesting issue. Perhaps the pattern is not actually tested/compared and is only returning success if the read returns success.  Not what I would expect for a read test with a pattern.

 

What command line options were used?

 

I just ran this on my parity drive where I know there will random data. It reported block errors.

 

# badblocks -vv -t 0x00 -o /tmp/badblocks.out /dev/sdc 102400 0

Checking blocks 0 to 102400

Checking for bad blocks in read-only mode

Testing with pattern 0x00: done                               

Pass completed, 102043 bad blocks found.

root@unRAID /home/rcotrone $wc -l /tmp/badblocks.out

102043 /tmp/badblocks.out

Something is up.

What command line options were used?

What was used to put the bad patterns on the disk?

[Barziya]

I wrote all zeros to a disk, then intentionally wrote a few non-zero bytes here and there.  To my dismay, a read-only badblocks pattern test for zeros passed with flying colors:

Checking for bad blocks in read-only mode
Testing with pattern 0x00: done
Pass completed, 0 bad blocks found. (0/0/0 errors)

Yet, I could hexdump the disk and see the non-zero bytes.

 

This is an interesting issue. Perhaps the pattern is not actually tested/compared and is only returning success if the read returns success.

No, when I ran the same test with a different pattern, it immediately started showing errors. Only the test with the zero pattern passed.

 

I don't have that disk anymore, but I found a way to reproduce a similar thing on a file. Consider this:

# dd if=/dev/zero bs=1M count=100 of=zeros
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.595489 s, 176 MB/s
# 
# cmp /dev/zero zeros
cmp: EOF on zeros
# 
# badblocks -vs -t0 zeros
Checking blocks 0 to 102399
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done
Pass completed, 0 bad blocks found. (0/0/0 errors)
# 
# echo crap>>zeros
#
# cmp /dev/zero zeros
/dev/zero zeros differ: byte 104857601, line 1
# 
# badblocks -vs -t0 zeros
Checking blocks 0 to 102399
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done
Pass completed, 0 bad blocks found. (0/0/0 errors)
# 

 

[WeeboTech]

Must have something to do with short blocks not being read at the end.

I'll have to review this more. It's a concern that needs to be reviewed.

 

$ dd if=/dev/zero bs=1M count=100 of=zeros
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.105639 s, 993 MB/s

$ ls -l zeros
-rw-rw-r-- 1 rcotrone rcotrone 104857600 Aug 14 13:50 zeros

$ badblocks -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 102399
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 0 bad blocks found.

$ badblocks -vs -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 102399
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 0 bad blocks found.


$ dd if=/dev/urandom bs=1M count=1 >> zeros 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.117619 s, 8.9 MB/s

$ ls -l zeros
-rw-rw-r-- 1 rcotrone rcotrone 105906176 Aug 14 13:51 zeros

$ badblocks -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 103423
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 1024 bad blocks found.


$ dd if=/dev/urandom bs=128 count=1 >> zeros    
1+0 records in
1+0 records out
128 bytes (128 B) copied, 0.000125417 s, 1.0 MB/s

ls -l zeros
-rw-rw-r-- 1 rcotrone rcotrone 104857728 Aug 14 13:53 zeros

$ badblocks -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 102399
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 0 bad blocks found.

$ dd if=/dev/urandom bs=512 count=1 >> zeros 
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000199429 s, 2.6 MB/s

$ badblocks -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 102399
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 0 bad blocks found.

$ dd if=/dev/urandom bs=512 count=1 >> zeros   
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000163644 s, 3.1 MB/s

$ badblocks -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 102400
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 1 bad blocks found.

[WeeboTech]

I'm finding a random character in the start or middle is detected.

Random characters at the end that do not complete a block are not read. (unless you alter the block size or fill it out to a full block).

 

rm zeros
$ dd if=/dev/zero bs=1M count=100 >> zeros         
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.109843 s, 955 MB/s
$ dd if=/dev/urandom bs=1 count=1 >> zeros 
1+0 records in
1+0 records out
1 byte (1 B) copied, 5.6979e-05 s, 17.6 kB/s
$ dd if=/dev/zero bs=1M count=100 >> zeros 
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.62282 s, 168 MB/s
$ dd if=/dev/urandom bs=1 count=1 >> zeros 
1+0 records in
1+0 records out
1 byte (1 B) copied, 4.2333e-05 s, 23.6 kB/s

$ badblocks -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 204799
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 1 bad blocks found. <-- unexpected, but also a full block was not written to the file and was probably not read. 

dd if=/dev/zero bs=1M count=1 >> zeros   
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00274552 s, 382 MB/s

$ badblocks -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 205823
Checking for bad blocks in read-only mode
Testing with pattern 0x00: done                                
Pass completed, 2 bad blocks found.  <-- this is what I expect. 

[Barziya]

Must have something to do with short block reads.

I'll have to review this more. It's a concern that needs to be reviewed.

That "thing" may be affecting the write mode too -- it may not be writing the pattern to every single byte of the destination file.

 

[WeeboTech]

Must have something to do with short block reads.

I'll have to review this more. It's a concern that needs to be reviewed.

That "thing" may be affecting the write mode too -- it may not be writing the pattern to every single byte of the destination file.

 

I believe it has more to do with calculating blocks based on a size and only reading a specific amount of those blocks. Anything at the end, is not read and ignored. 

 

My tests at start and in the middle prove it's comparing.

My tests of putting extra bytes at the end of less then a block are not detected.

Fill out the last block and it's detected. I.E. number of blocks are recalculated and read.

[WeeboTech]

 

This shows it's based on a full block at the end.

I put 1 random byte at the end of the final incomplete block thus filling the block.

 

dd if=/dev/zero bs=128 count=1M > zeros
1048576+0 records in
1048576+0 records out
134217728 bytes (134 MB) copied, 1.48001 s, 90.7 MB/s

rcotrone@rgclws:~/log $ dd if=/dev/zero bs=1   count=127 >> zeros  
127+0 records in
127+0 records out
127 bytes (127 B) copied, 0.00042446 s, 299 kB/s

rcotrone@rgclws:~/log $ dd if=/dev/urandom  bs=1   count=1 >> zeros      
1+0 records in
1+0 records out
1 byte (1 B) copied, 4.6294e-05 s, 21.6 kB/s

rcotrone@rgclws:~/log $ badblocks -b 128 -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 1048576
Checking for bad blocks in read-only mode
Testing with pattern 0x00: Weird value (128) in do_read
done                                
Pass completed, 1 bad blocks found.

 

Added 1 more byte for an incomplete block and it's undetected.

This is due to the math of block reads.  Not so much a problem with the program itself, but based on auto calculating blocks to read and not reading past the calculated number of blocks.

 

 

rcotrone@rgclws:~/log $ dd if=/dev/urandom  bs=1   count=1 >> zeros         
1+0 records in
1+0 records out
1 byte (1 B) copied, 7.0351e-05 s, 14.2 kB/s
rcotrone@rgclws:~/log $ badblocks -b 128 -vv -t0x00 -o badblocks.lis zeros  
Checking blocks 0 to 1048576
Checking for bad blocks in read-only mode
Testing with pattern 0x00: Weird value (128) in do_read
done                                
Pass completed, 1 bad blocks found.

[Barziya]

Must have something to do with short block reads.

I'll have to review this more. It's a concern that needs to be reviewed.

That "thing" may be affecting the write mode too -- it may not be writing the pattern to every single byte of the destination file.

 

I believe it has more to do with calculating blocks based on a size and only reading a specific amount of those blocks. Anything at the end, is not read and ignored. 

That is too bad!  I had very high hopes for badblocks.  See, I was investigating alternate ways to do a preclear, before Google brought me to this thread here.  The thing that started me was the sum command preclear uses for the post-read, which is a very very slow way to do that.  I might as well capture and parse the output of a `hexdump /dev/sdX | head` and do the job five times faster.  At the end, I narrowed down my choices to cmp and badblocks.  I especially liked badblocks for its -d option (read delay factor) which could make the system more responsive dirung a long-running preclear.  It's too bad I can't rely on badblocks to go through every single byte on the disk, that's a show-stopper.

 

[Barziya]

This is due to the math of block reads.  Not so much a problem with the program itself, but based on auto calculating blocks to read and not reading past the calculated number of blocks.

Well, dd also uses blocks, but it doesn't ignore stuff at the end. So it kind of is a problem with the program itself.

 

[WeeboTech]

  It's too bad I can't rely on badblocks to go through every single byte on the disk, that's a show-stopper.

 

It goes through every byte that is a full block based on the supplied or calculated block size.

Chances are you would not have an incomplete block at the end of the disk.

 

This is why I asked for the test condition used in your tests.

If I can replicate them, perhaps we can update badblocks to handle conditions better.

 

Random bytes in the start and end are detected, Random bytes at the end are detected up until a full block read at the end. 

 

If you have x count of blocks on a disk, you probably would not have x count + A few extra bytes.

 

It's either that or the block size calculation needs to be refined in badblocks.

 

I've used this tool to weed out drives for years.

I had high hopes of changing bad blocks during the output phase.

 

I.E. Make it pipeable with \n and write messages out via some configurable time instead of every second.

In addition adding the MB/s output so you could see what your throughput is on the drive.

 

 

[WeeboTech]

This is due to the math of block reads.  Not so much a problem with the program itself, but based on auto calculating blocks to read and not reading past the calculated number of blocks.

Well, dd also uses blocks, but it doesn't ignore stuff at the end. So it kind of is a problem with the program itself.

 

If you tell dd to read x number of blocks, it will ignore anything after x number of blocks.

It's the same thing.

 

There's a condition that's not understood.

 

If you use dd with no block count restriction, it will read until EOF.

 

badblocks does not read until EOF, it reads until the ending block count is read.

It's working the way it is supposed to work.

 

What is the exact condition of your test with a hard drive. That's the tell tale.

What commands were made to initialize then add random characters, then test it?

 

The 4 extra bytes at the end of a block that is not calculated to be read is not really an accurate test for the program.

badblocks does not read until EOF as a stream of bytes. It's a block oriented direct read.

 

File reads are not an accurate test unless you are working in disk block size.

[WeeboTech]

Also, I've run badblocks in write mode test many times since it's inception in 90s.

It does detect bad blocks as long as the kernel reports them back.

 

I've weeded out many drives.

 

Running multiple pass write-read mode have made questionable sectors reallocate and marginal sectors get refreshed.  I'm confident it's working as it should.

 

I would really like to see the test case where it's not working on a disk so we can define where the issue really is.

 

[Barziya]

This is due to the math of block reads.  Not so much a problem with the program itself, but based on auto calculating blocks to read and not reading past the calculated number of blocks.

Well, dd also uses blocks, but it doesn't ignore stuff at the end. So it kind of is a problem with the program itself.

 

If you tell dd to read x number of blocks, it will ignore anything after x number of blocks.

It's the same thing.

No, no, I wasn't talking about `count=` number of blocks.  I was talking about dd's `bs=` size of its block.  dd won't ignore bytes that don't constitute a full block. Badblock's equivalent is the -b option, and badblocks seem to ignore bytes that don't constitute a full block.

 

What is the exact condition of your test with a hard drive. That's the tell tale.

We can use Joe.L's example from the original post:

The full badblocks command I used was:

badblocks -c 1024 -b 65536 -vsw -o /boot/badblocks_out_sdl.txt  /dev/sdl

 

The disk he used that on was:  User Capacity:  2,000,398,934,016 bytes

 

2000398934016 is not evenly divisible by 65536.

 

So for his badblocks test he had 30523665 blocks of 65536 bytes each + 24576 bytes.

 

For all he knew, he tested every single byte on his disk.

 

In reality, 24576 bytes on his disk were never written to or tested.  They can contain any random garbage.

 

 

Furthermore, he used -c 1024 option, so he was doing 1024 such blocks at a time.

 

30523665 is not evenly divisible by 1024.

 

So if badblocks is acting the same way with the "blocks at a time" option, that would mean that Joe.L had  29808  passes of 1024  65536-byte blocks, with a leftover of 273 such blocks. Which could mean that he had a total of 17,915,904 bytes at the end that were never written to or tested.

 

 

I don't think the user of badblocks should burden himself with such calculations. I would expect that every single byte of the disk will be tested, or at least a warning be given otherwise.

 

 

I've run badblocks in write mode test many times since it's inception in 90s.

You've mentioned twice already that this tool has been around for many many years. While I see how it's tempting to bring that up, it's not a logical argument for anything.