So, initially i noticed that some of my containers had stopped.  I thought nothing of it and tried to start them, when i got a Execution error 403.  After some digging and googling I thought my cache drive was broke/corrupted.  When i got home from work I check the connections and rebooted the server.  Everything came back as expected so carried on with my evening.

 

come to this morning, and all the containers had stopped.  Again tried to start them but same error 403.  So with bit more digging and checking the proper logs its not my cache drive but SDK1 of my VM pool.  Both are new bought May'23 1tb SSD's, check enchancedsyslog:

 

Jul 13 23:57:53 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 2

 

Jul 13 23:57:56 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdk1 (-5)

Jul 13 23:57:56 Tower kernel: BTRFS error (device sdk1): bdev /dev/sdk1 errs: wr 59, rd 0, flush 0, corrupt 0, gen 0

Jul 13 23:57:56 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 1

Jul 13 23:57:56 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 2

Jul 13 23:57:56 Tower kernel: BTRFS: error (device sdk1) in write_all_supers:4438: errno=-5 IO failure (2 errors while writing supers)

Jul 13 23:57:56 Tower kernel: BTRFS info (device sdk1: state E): forced readonly

 

SMART & the dashboard show healthy?  So, i guess I need to know is the drive bad? if so I can return and replace if so am I able to run my dockers on the single drive for a few days?

 

appriciate the help

 

tower-smart-20230714-0944.zip tower-diagnostics-20230714-1049.zip

That looks more like a power/connection issue, replace both cables on that SSD, then run an extended SMART test and post new diags after array start.

14 minutes ago, JorgeB said:

That looks more like a power/connection issue, replace both cables on that SSD, then run an extended SMART test and post new diags after array start.

Am I ok to spin down that pool or should I shutdown?

To replace cables you need to power off.

3 hours ago, JorgeB said:

To replace cables you need to power off.

Powered down, as I have 2 of the same brand ssd i wasn't able to match up the serial so I replaced the power for both.  I wasn't able to change the Sata as i'm on a LSI Raid card and only had 8087 SATA cable to hand.

 

I have found the other 8087 so could change if needed.

tower-smart-20230714-1534.zip tower-diagnostics-20230714-1534.zip

No errors so far, run a correcting scrub on the pool.

Ran without errors.  Will leave it and see what happens overnight and check the logs in the morning.

UUID:             0d3eea3a-97ec-4495-aadc-9bc75f3a2749
Scrub started:    Fri Jul 14 16:19:57 2023
Status:           finished
Duration:         0:05:30
Total to scrub:   193.01GiB
Rate:             598.95MiB/s
Error summary:    no errors found

 

Also suggest taking a look here for better pool monitoring.

So, was all good this morning no further issue until this afternoon when i was copying data from my old server to the new VM on this pool.

 

Jul 15 16:41:54 Tower kernel: I/O error, dev sdk, sector 211154464 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Jul 15 16:41:54 Tower kernel: BTRFS info (device sdk1): read error corrected: ino 76350 off 21219844096 (dev /dev/sdk1 sector 211152416)

 

I don't know if this is normal, that it corrected the error?

tower-smart-20230715-1940.zip

Got up this morning to

 

Jul 16 10:24:30 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 1
Jul 16 10:24:30 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 1
Jul 16 10:24:31 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 1

 

 

tower-diagnostics-20230716-1031.zip tower-smart-20230716-1026.zip

Jul 16 08:40:11 Tower kernel: sd 11:0:0:0: [sdk] tag#2183 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Jul 16 08:40:11 Tower kernel: sd 11:0:0:0: [sdk] tag#2183 Sense Key : 0xb [current]
Jul 16 08:40:11 Tower kernel: sd 11:0:0:0: [sdk] tag#2183 ASC=0x0 ASCQ=0x0
Jul 16 08:40:11 Tower kernel: sd 11:0:0:0: [sdk] tag#2183 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
Jul 16 08:40:11 Tower kernel: I/O error, dev sdk, sector 2176 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Jul 16 08:40:11 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdk1 (-5)

 

Device dropped offline, replace cables or try a different SATA port, if the issue follows the device replace it.