DZMM Posted September 29, 2018 Share Posted September 29, 2018 (edited) My mover recently has been taking an eternity to run, which has been causing bottlenecks. I've just enabled mover logging and I'm seeing lots of errors for device sdf and when I try to view the smart details they aren't available in the GUI or the temps. Sdf1 is the oldest drive in my cache pool. Sep 29 08:01:54 Highlander kernel: print_req_error: I/O error, dev sdf, sector 558119040 Sep 29 08:01:54 Highlander kernel: sd 5:0:0:0: [sdf] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:54 Highlander kernel: sd 5:0:0:0: [sdf] tag#3 CDB: opcode=0x28 28 00 21 43 89 80 00 00 78 00 Sep 29 08:01:54 Highlander kernel: print_req_error: I/O error, dev sdf, sector 558074240 Sep 29 08:01:54 Highlander kernel: sd 5:0:0:0: [sdf] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:54 Highlander kernel: sd 5:0:0:0: [sdf] tag#4 CDB: opcode=0x28 28 00 21 43 8a 08 00 00 18 00 Sep 29 08:01:54 Highlander kernel: print_req_error: I/O error, dev sdf, sector 558074376 Sep 29 08:01:54 Highlander kernel: sd 5:0:0:0: [sdf] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:54 Highlander kernel: sd 5:0:0:0: [sdf] tag#5 CDB: opcode=0x28 28 00 21 43 8a 30 00 00 38 00 Sep 29 08:01:54 Highlander kernel: print_req_error: I/O error, dev sdf, sector 558074416 Sep 29 08:01:54 Highlander kernel: btrfs_end_buffer_write_sync: 2 callbacks suppressed Sep 29 08:01:54 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:54 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:54 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:54 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:56 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:56 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:56 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:56 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:56 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:56 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:56 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:56 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:56 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:56 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:57 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:57 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:57 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:01:57 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:01:59 Highlander kernel: scsi_io_completion: 2823 callbacks suppressed Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#6 CDB: opcode=0x2a 2a 00 15 09 bf 00 00 00 40 00 Sep 29 08:01:59 Highlander kernel: print_req_error: 2840 callbacks suppressed Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 352960256 Sep 29 08:01:59 Highlander kernel: btrfs_dev_stat_print_on_error: 4885 callbacks suppressed Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117374, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#7 CDB: opcode=0x2a 2a 00 15 09 bf 40 00 00 80 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 352960320 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117375, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#8 CDB: opcode=0x2a 2a 00 15 09 bf c0 00 01 00 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 352960448 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117376, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117377, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#9 CDB: opcode=0x2a 2a 00 15 09 c0 c0 00 00 80 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 352960704 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117378, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#10 CDB: opcode=0x2a 2a 00 15 09 c1 40 00 00 c0 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 352960832 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117379, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117380, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#11 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#11 CDB: opcode=0x2a 2a 00 11 64 1c 80 00 03 08 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 291773568 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117381, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117382, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67117383, rd 60347600, flush 367838, corrupt 0, gen 0 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#12 CDB: opcode=0x2a 2a 00 32 33 74 d0 00 00 38 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 842233040 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#13 CDB: opcode=0x2a 2a 00 11 64 2e d0 00 01 c0 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 291778256 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#14 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#14 CDB: opcode=0x2a 2a 00 11 64 30 e0 00 04 e0 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 291778784 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:01:59 Highlander kernel: sd 5:0:0:0: [sdf] tag#15 CDB: opcode=0x2a 2a 00 11 64 35 c0 00 01 38 00 Sep 29 08:01:59 Highlander kernel: print_req_error: I/O error, dev sdf, sector 291780032 Sep 29 08:01:59 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 ### [PREVIOUS LINE REPEATED 2 TIMES] ### Sep 29 08:01:59 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:00 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:00 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:00 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:00 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:00 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:00 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:00 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 ### [PREVIOUS LINE REPEATED 2 TIMES] ### Sep 29 08:02:00 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:00 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:00 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 ### [PREVIOUS LINE REPEATED 50 TIMES] ### Sep 29 08:02:04 Highlander kernel: scsi_io_completion: 4555 callbacks suppressed Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#12 CDB: opcode=0x2a 2a 00 15 0a fc 20 00 00 20 00 Sep 29 08:02:04 Highlander kernel: print_req_error: 4620 callbacks suppressed Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353041440 Sep 29 08:02:04 Highlander kernel: btrfs_dev_stat_print_on_error: 8082 callbacks suppressed Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122061, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#14 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#14 CDB: opcode=0x2a 2a 00 15 0a fc c0 00 00 80 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353041600 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122062, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#15 CDB: opcode=0x2a 2a 00 15 0a fd 40 00 00 80 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353041728 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122063, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#16 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#16 CDB: opcode=0x2a 2a 00 15 0a fd c0 00 00 80 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353041856 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122064, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#13 CDB: opcode=0x2a 2a 00 15 0a fc 40 00 00 80 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353041472 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122065, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#17 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#17 CDB: opcode=0x2a 2a 00 15 0a fe 40 00 00 80 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353041984 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122066, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#18 CDB: opcode=0x2a 2a 00 15 0a fe c0 00 01 00 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353042112 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122067, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#19 CDB: opcode=0x2a 2a 00 15 0a ff c0 00 00 80 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 353042368 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122068, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122069, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#22 CDB: opcode=0x2a 2a 00 16 ea 49 58 00 00 68 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 384452952 Sep 29 08:02:04 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67122070, rd 60350975, flush 367894, corrupt 0, gen 0 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:04 Highlander kernel: sd 5:0:0:0: [sdf] tag#23 CDB: opcode=0x2a 2a 00 16 ea 49 c0 00 00 80 00 Sep 29 08:02:04 Highlander kernel: print_req_error: I/O error, dev sdf, sector 384453056 Sep 29 08:02:05 Highlander kernel: btrfs_end_buffer_write_sync: 52 callbacks suppressed Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:05 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 Sep 29 08:02:05 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 ### [PREVIOUS LINE REPEATED 31 TIMES] ### Sep 29 08:02:09 Highlander kernel: scsi_io_completion: 4058 callbacks suppressed Sep 29 08:02:09 Highlander kernel: print_req_error: 4117 callbacks suppressed Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 828444184 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: btrfs_dev_stat_print_on_error: 6245 callbacks suppressed Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125270, rd 60354003, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#23 CDB: opcode=0x2a 2a 00 01 9e 42 20 00 00 08 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 27148832 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125271, rd 60354003, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#24 CDB: opcode=0x2a 2a 00 01 a2 cf 48 00 00 08 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 27447112 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125272, rd 60354003, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#25 CDB: opcode=0x2a 2a 00 32 33 62 30 00 00 50 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 842228272 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125273, rd 60354003, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125274, rd 60354003, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#27 CDB: opcode=0x28 28 00 20 b9 05 d8 00 00 28 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 548996568 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125274, rd 60354004, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#28 CDB: opcode=0x28 28 00 21 32 c9 18 00 00 10 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 556976408 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125274, rd 60354005, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#29 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#29 CDB: opcode=0x28 28 00 21 32 e7 58 00 00 38 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 556984152 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125274, rd 60354006, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#30 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#30 CDB: opcode=0x28 28 00 20 b9 06 00 00 00 28 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 548996608 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125274, rd 60354007, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#31 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#31 CDB: opcode=0x28 28 00 21 32 e7 a0 00 00 d8 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 556984224 Sep 29 08:02:09 Highlander kernel: BTRFS error (device sdd1): bdev /dev/sdf1 errs: wr 67125274, rd 60354008, flush 367935, corrupt 0, gen 0 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#0 CDB: opcode=0x28 28 00 21 24 63 90 00 00 18 00 Sep 29 08:02:09 Highlander kernel: print_req_error: I/O error, dev sdf, sector 556032912 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Sep 29 08:02:09 Highlander kernel: sd 5:0:0:0: [sdf] tag#1 CDB: opcode=0x28 28 00 21 32 e8 88 00 00 98 00 Sep 29 08:02:11 Highlander kernel: btrfs_end_buffer_write_sync: 31 callbacks suppressed Sep 29 08:02:11 Highlander kernel: BTRFS warning (device sdd1): lost page write due to IO error on /dev/sdf1 ### [PREVIOUS LINE REPEATED 2 TIMES] ### Sep 29 08:02:11 Highlander kernel: BTRFS error (device sdd1): error writing primary super block to device 3 Sep 29 08:02:11 Highlander kernel: Buffer I/O error on dev sdf1, logical block 16, async page read Coincidentally, I've had a for a while issues stopping and starting dockers so I'm now wondering if it's related. Time to replace the drive? Thanks highlander-diagnostics-20180929-0801.zip Edited September 30, 2018 by DZMM Quote Link to comment
JorgeB Posted September 29, 2018 Share Posted September 29, 2018 SSD dropped offline, so no SMART, but I would first replace both cables to rule them out, if issues persist replace the SSD. 1 Quote Link to comment
DZMM Posted September 29, 2018 Author Share Posted September 29, 2018 (edited) Thanks. I've rebooted and there are SMART errors, but they happened yonks ago and haven't gone up so they aren't a major concern??? I vaguely remember when they happened I did something stupid that I attributed the errors to 187 Reported uncorrect 0x0032 100 100 000 Old age Always Never 56 198 Offline uncorrectable 0x0032 100 100 000 Old age Always Never 192 I've rebooted and there are a lot of "read errors corrected" for sdd and sdf which I assume is a good sign? I've ordered a new SSD just in case which arrives tonight - will return if I don't need. I'll also get some cables now just in case - but I'm going to recheck the current connections as I opened my case this week to add a new 1TB drive and I'm wondering if I dislodged the cables a bit. highlander-diagnostics-20180929-0903.zip Edited September 29, 2018 by DZMM Quote Link to comment
JorgeB Posted September 29, 2018 Share Posted September 29, 2018 You should run a scrub to bring the dropped SSD up to date, but note that any NOCOW shares won't be fixed. 1 Quote Link to comment
DZMM Posted September 29, 2018 Author Share Posted September 29, 2018 scrub completed - all corrected, so good I think: scrub status for c98e8605-16cb-4e52-bd4a-2d4c7e812678 scrub started at Sat Sep 29 09:51:45 2018 and finished after 01:24:10 total bytes scrubbed: 777.04GiB with 12544106 errors error details: verify=2484 csum=12541622 corrected errors: 12544106, uncorrectable errors: 0, unverified errors: 0 I'm going to reboot and check the cables when the kids are eating Quote Link to comment
JorgeB Posted September 29, 2018 Share Posted September 29, 2018 Looks good, as long as there aren't any NOCOW shares on cache with important data you're fine. Quote Link to comment
landS Posted September 29, 2018 Share Posted September 29, 2018 4 hours ago, johnnie.black said: You should run a scrub to bring the dropped SSD up to date, but note that any NOCOW shares won't be fixed. Is a Scrub automated by dynamix trim for SSDs? If not, would you recommend setting up a user scripts automatic Scrub every few months? Quote Link to comment
JorgeB Posted September 29, 2018 Share Posted September 29, 2018 16 minutes ago, landS said: Is a Scrub automated by dynamix trim for SSDs? No. 16 minutes ago, landS said: would you recommend setting up a user scripts automatic Scrub every few months? It's good practice to run one once in a while to confirm all is well, and it should be run after any issue, like an unclean shutdown, dropped disk, etc. 1 Quote Link to comment
DZMM Posted September 29, 2018 Author Share Posted September 29, 2018 Drive is playing up so I'm going to replace it and investigate it offline. Is this the correct way to replace it: 1. Power off server and physically remove drive 2. Put new drive in 3. Boot server and add drive to 3rd cache pool slot 4. Balance I'm trying to avoid doing what I normally do which is copy everything to the array first, and then use 'prefer' to transfer back. Thanks Quote Link to comment
DZMM Posted September 29, 2018 Author Share Posted September 29, 2018 Going to play it safe and copy all to the array Quote Link to comment
JorgeB Posted September 29, 2018 Share Posted September 29, 2018 2 hours ago, DZMM said: Going to play it safe and copy all to the array You can use that one or this one: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=480419 Quote Link to comment
DZMM Posted September 30, 2018 Author Share Posted September 30, 2018 Update: I replaced the drive today as it was still causing problems, the most recent was docker stopping altogether. I successfully replaced it and my other 500GB pool drive with a new 1TB drive, and all is well again. Because I didn't have a spare SATA slot, moving the files to the array and back took around 10 hours (bloody plex) which annoyed my family. I've added the working 500GB drive as a UD doubling my 2nd 'pool' so it's using the last SATA slot, but this at least means I can quickly remove it if I need to replace an array or cache drive in the future - I don't want to have an awful switch-out like today ever again. The unexpected bonus with replacing the drive is that the intermittent problems I've had for a few months stopping and starting dockers have gone and dockers seem more responsive, so the drive has obviously been causing issues for a while. I am also expecting the weird issues where dockers would become inaccessible randomly for a few seconds to almost an hour at times will go away as well, as well as the poor intermittent mover performance to be fixed, as well as problems I wasn't even aware of. Thanks for all the help. Quote Link to comment
DZMM Posted October 6, 2018 Author Share Posted October 6, 2018 Update 2: The replacement drive had to be replaced. Luckily I'm keeping a spare SATA slot now, so I didn't have to move all my cache files to the array @johnnie.black I think this statement could be tweaked a bit to make it clearer that if you've got a pool you don't have to have the drive you're replacing attached e.g. it might have failed completely - you just need one (if two drives in the pool) to be attached so that the files can be synced to the new drive On 7/18/2016 at 9:46 AM, johnnie.black said: This procedure assumes you have enough ports to have both the old and new devices connected at the same time, Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.