Cache Drive - BTRFS: errors


Recommended Posts

I get the following errors -

 

I have checked the drive with diagnostics on windows with no errors.

 

I reformatted the drive.

 

Resetup dockers.

 

I have read a few different things - back docker image file so I deleted that and reset it up and same issue comes back. Also saw mention of sata controller could be at fault.

 

Anything else I can test????

 

Apr  6 04:40:03 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 440320

Apr  6 04:40:03 HPTower kernel: btrfs_dev_stat_print_on_error: 15 callbacks suppressed

Apr  6 04:40:03 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23018, flush 1, corrupt 0, gen 0

Apr  6 04:40:03 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 2537472

Apr  6 04:40:03 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23019, flush 1, corrupt 0, gen 0

Apr  6 04:40:03 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 4444712

Apr  6 04:40:03 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23020, flush 1, corrupt 0, gen 0

Apr  6 04:40:03 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 440320

Apr  6 04:40:03 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23021, flush 1, corrupt 0, gen 0

Apr  6 04:40:03 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 2537472

Apr  6 04:40:03 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23022, flush 1, corrupt 0, gen 0

Apr  6 04:40:03 HPTower kernel: XFS (sdf1): xfs_log_force: error -5 returned.

Apr  6 04:40:04 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 440320

Apr  6 04:40:04 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23023, flush 1, corrupt 0, gen 0

Apr  6 04:40:04 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 2537472

Apr  6 04:40:04 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23024, flush 1, corrupt 0, gen 0

Apr  6 04:40:04 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 4444712

Apr  6 04:40:04 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23025, flush 1, corrupt 0, gen 0

Apr  6 04:40:04 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 440320

Apr  6 04:40:04 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23026, flush 1, corrupt 0, gen 0

Apr  6 04:40:04 HPTower kernel: blk_update_request: I/O error, dev loop1, sector 2537472

Apr  6 04:40:04 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1051, rd 23027, flush 1, corrupt 0, gen 0

Apr  6 04:40:08 HPTower kernel: blk_update_request: 15 callbacks suppressed

syslog.1.txt.zip

Link to comment

Well don't do an XFS repair on a btrfs disk, that won't do you any good.  Your log shows signs of a bad disk (likely your cache disk).  Please provide complete system diagnostics (not just your syslog).  This can be obtained from the web interface under the Tools -> Diagnostics page.  Click the download button and upload the full zip file in a reply here.

Link to comment

IF I do xfs_repair -v /dev/sdf

 

Phase 1 - find and verify superblock...

superblock read failed, offset 0, size 524288, ag 0, rval -1

 

fatal error -- Input/output error

i see several issues with the above command
  • it is working on the raw device (sdX) which invalidates parity rather than the mdX devices that are available in Maintenance mode which maintain parity
  • when working with raw devices you need to include the partition number (e.g. /dev/sdf1)
  • it looks like the drive may well have dropped offline

Link to comment

It did drop offline and wasn't available again.

 

I shut it down and changed the sata cable on this drive to see if it was faulty. Like I said I tested the drive in windows machine and no errors/issues/corruption.

 

I will see if it is corrupt again tonight after work.

Link to comment

Errors are back again. Is it the drive, sata card, Docker issue?

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#5 CDB: opcode=0x2a 2a 00 0f b8 d8 80 00 00 10 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 263772288

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#6 CDB: opcode=0x2a 2a 00 0a 7d 9c 90 00 00 40 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 176004240

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000522, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000523, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000524, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000525, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000526, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000527, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000528, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22000529, lost async page write

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#7 CDB: opcode=0x2a 2a 00 0a 7f 42 d0 00 00 58 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 176112336

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22014034, lost async page write

Apr 15 21:07:23 HPTower kernel: Buffer I/O error on dev sdf1, logical block 22014035, lost async page write

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#8 CDB: opcode=0x2a 2a 00 00 42 8a 70 00 00 08 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 4360816

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#9 CDB: opcode=0x2a 2a 00 00 41 dd 80 00 00 08 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 4316544

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#10 CDB: opcode=0x2a 2a 00 0a 7d 9c 50 00 00 40 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 176004176

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 175829887

Apr 15 21:07:23 HPTower kernel: XFS (sdf1): metadata I/O error: block 0xa7af33f ("xlog_iodone") error 5 numblks 64

Apr 15 21:07:23 HPTower kernel: XFS (sdf1): xfs_do_force_shutdown(0x2) called from line 1180 of file fs/xfs/xfs_log.c.  Return address = 0xffffffff81275f2f

Apr 15 21:07:23 HPTower kernel: XFS (sdf1): Log I/O Error Detected.  Shutting down filesystem

Apr 15 21:07:23 HPTower kernel: XFS (sdf1): xfs_log_force: error -5 returned.

Apr 15 21:07:23 HPTower kernel: XFS (sdf1): Please umount the filesystem and rectify the problem(s)

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 175829890

Apr 15 21:07:23 HPTower kernel: XFS (sdf1): metadata I/O error: block 0xa7af342 ("xlog_iodone") error 5 numblks 64

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#13 CDB: opcode=0x2a 2a 00 00 00 00 a0 00 00 20 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 160

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#14 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#14 CDB: opcode=0x2a 2a 00 00 07 cf 00 00 02 00 00

Apr 15 21:07:23 HPTower kernel: blk_update_request: I/O error, dev sdf, sector 511744

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#15 CDB: opcode=0x2a 2a 00 00 27 cf 00 00 02 00 00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#16 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00

Apr 15 21:07:23 HPTower kernel: sd 9:0:0:0: [sdf] tag#16 CDB: opcode=0x2a 2a 00 00 42 a5 d0 00 02 00 00

Apr 15 21:07:23 HPTower kernel: loop: Write error at byte offset 2235449344, length 4096.

Apr 15 21:07:23 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0

Apr 15 21:07:23 HPTower kernel: BTRFS: bdev /dev/loop1 errs: wr 1, rd 0, flush 1, corrupt 0, gen 0

Apr 15 21:07:23 HPTower kernel: XFS (sdf1): xfs_do_force_shutdown(0x2) called from line 1180 of file fs/xfs/xfs_log.c.  Return address = 0xffffffff81275f2f

Apr 15 21:07:23 HPTower kernel: BTRFS: error (device loop1) in write_all_supers:3501: errno=-5 IO failure (errors while submitting device barriers.)

Apr 15 21:07:23 HPTower kernel: BTRFS info (device loop1): forced readonly

Apr 15 21:07:23 HPTower kernel: BTRFS warning (device loop1): Skipping commit of aborted transaction.

Apr 15 21:07:23 HPTower kernel: ------------[ cut here ]------------

Apr 15 21:07:23 HPTower kernel: WARNING: CPU: 3 PID: 18713 at fs/btrfs/super.c:260 __btrfs_abort_transaction+0x4d/0x10e()

Apr 15 21:07:23 HPTower kernel: BTRFS: Transaction aborted (error -5)

Apr 15 21:07:23 HPTower kernel: Modules linked in: xt_nat veth kvm_amd kvm vhost_net vhost macvtap macvlan md_mod xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables tun ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat i2c_piix4 pata_atiixp k10temp r8169 mii ahci libahci acpi_cpufreq [last unloaded: md_mod]

Apr 15 21:07:23 HPTower kernel: CPU: 3 PID: 18713 Comm: btrfs-transacti Not tainted 4.1.18-unRAID #1

Apr 15 21:07:23 HPTower kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-MA785GMT-UD2H/GA-MA785GMT-UD2H, BIOS F10b 07/13/2010

Apr 15 21:07:23 HPTower kernel: 0000000000000009 ffff88040457fc58 ffffffff815f2403 ffffffff8182e140

Apr 15 21:07:23 HPTower kernel: 0000000000000292 ffff88040457fc58 ffff88040457fca8 ffff88040457fc98

Apr 15 21:07:23 HPTower kernel: ffffffff8104778b ffff88040457fce8 ffffffff81286d44 00000000fffffffb

Apr 15 21:07:23 HPTower kernel: Call Trace:

Apr 15 21:07:23 HPTower kernel: [<ffffffff815f2403>] dump_stack+0x65/0x85

Apr 15 21:07:23 HPTower kernel: [<ffffffff8104778b>] warn_slowpath_common+0x97/0xb1

Apr 15 21:07:23 HPTower kernel: [<ffffffff81286d44>] ? __btrfs_abort_transaction+0x4d/0x10e

Apr 15 21:07:23 HPTower kernel: [<ffffffff810477e6>] warn_slowpath_fmt+0x41/0x43

Apr 15 21:07:23 HPTower kernel: [<ffffffff81286d44>] __btrfs_abort_transaction+0x4d/0x10e

Apr 15 21:07:23 HPTower kernel: [<ffffffff812ac3ad>] cleanup_transaction+0x80/0x21d

Apr 15 21:07:23 HPTower kernel: [<ffffffff810724fb>] ? wait_woken+0x7d/0x7d

Apr 15 21:07:23 HPTower kernel: [<ffffffff812ad617>] btrfs_commit_transaction+0xa6c/0xa81

Apr 15 21:07:23 HPTower kernel: [<ffffffff812a93ce>] transaction_kthread+0xfa/0x1cb

Apr 15 21:07:23 HPTower kernel: [<ffffffff812a92d4>] ? btrfs_cleanup_transaction+0x461/0x461

Apr 15 21:07:23 HPTower kernel: [<ffffffff8105c74a>] kthread+0xd6/0xde

Apr 15 21:07:23 HPTower kernel: [<ffffffff8105c674>] ? kthread_create_on_node+0x172/0x172

Apr 15 21:07:23 HPTower kernel: [<ffffffff815f7f12>] ret_from_fork+0x42/0x70

Apr 15 21:07:23 HPTower kernel: [<ffffffff8105c674>] ? kthread_create_on_node+0x172/0x172

Apr 15 21:07:23 HPTower kernel: ---[ end trace 8e8d2967722e7aa1 ]---

Apr 15 21:07:23 HPTower kernel: BTRFS: error (device loop1) in cleanup_transaction:1692: errno=-5 IO failure

Link to comment

Providing whatever data you have or had is backed up somewhere else...

 

remove the drive (loop1) from your array. Hook it up to another PC or Mac and delete every existing partition/volume in it.

 

Leave it unpartitioned/unformatted, all of it.

 

Put it back in your array, reassign it to your cache pool and let unRAID do its thing with it, from scratch.

Link to comment

 

Personally, I've got 4 SSDs in my pool and they're running butterFS marvellously!

Do you have any trouble when you reboot? I have two ssd in cache pool but every time I reboot my array only starts with one ssd and I have to stop it and assign the cache pool again.

Link to comment

 

Personally, I've got 4 SSDs in my pool and they're running butterFS marvellously!

Do you have any trouble when you reboot? I have two ssd in cache pool but every time I reboot my array only starts with one ssd and I have to stop it and assign the cache pool again.

 

No issues when rebooting either. Solid as a rock.

Link to comment

 

Personally, I've got 4 SSDs in my pool and they're running butterFS marvellously!

Do you have any trouble when you reboot? I have two ssd in cache pool but every time I reboot my array only starts with one ssd and I have to stop it and assign the cache pool again.

Use another browser (other than chrome) once to assign the 2nd cache disk, do the complete procedure, assign 2nd cache, start array, stop array and reboot.

 

It should now stick and you can go back to chrome.

Link to comment

Providing whatever data you have or had is backed up somewhere else...

 

remove the drive (loop1) from your array. Hook it up to another PC or Mac and delete every existing partition/volume in it.

 

Leave it unpartitioned/unformatted, all of it.

 

Put it back in your array, reassign it to your cache pool and let unRAID do its thing with it, from scratch.

 

Did that the other day and it has those errors again. So is it the drive or is it the Docker image that is bad. I have a spare non ssd drive I can try.

 

I ran a HD diagnostic on the ssd and it came back clean so can't figure out what is at fault.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.