TRIM failing on SSD cache pool?

[Drnick5]

Hi folks,

I'm new to unRAID, I have  cache pool of 2 SSD's (a 500gb and a 525GB) I'm using the plugin to allow trim, and both of these SSD's are plugged directly into my motherboards SATA ports (ASrock H370m) 

I got an email alert only containing the following:

fstrim: /mnt/cache: FITRIM ioctl failed: Remote I/O error

 

It contained nothing else. I'm guessing that its saying the trim event I had schedule failed. Any idea what the cause could be. Only thing I can think of is it has something to do with having 2 different sized SSD's in the cache pool. In a normal raid, I'd "lose" the extra 25GB on the 2nd SSD, (which isn't a big deal. ) but it woudl work. not sure if thats the case in unRAID, or if something else is going on. 

[JorgeB]

22 minutes ago, Drnick5 said:

Only thing I can think of is it has something to do with having 2 different sized SSD's in the cache pool.

That won't be a problem, please post the diagnostics, maybe something visible there.

[Drnick5]

I checked the system log, and this is what I see

 

Dec 27 02:00:40 UNRAID kernel: ata6.00: status: { DRDY }
Dec 27 02:00:40 UNRAID kernel: ata6.00: failed command: WRITE FPDMA QUEUED
Dec 27 02:00:40 UNRAID kernel: ata6.00: cmd 61/00:f0:40:09:33/05:00:05:00:00/40 tag 30 ncq dma 655360 out
Dec 27 02:00:40 UNRAID kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 27 02:00:40 UNRAID kernel: ata6.00: status: { DRDY }
Dec 27 02:00:40 UNRAID kernel: ata6.00: failed command: SEND FPDMA QUEUED
Dec 27 02:00:40 UNRAID kernel: ata6.00: cmd 64/01:f8:00:00:00/00:00:00:00:00/a0 tag 31 ncq dma 512 out
Dec 27 02:00:40 UNRAID kernel: res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 27 02:00:40 UNRAID kernel: ata6.00: status: { DRDY }
Dec 27 02:00:40 UNRAID kernel: ata6: hard resetting link
Dec 27 02:00:41 UNRAID kernel: ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 27 02:00:41 UNRAID kernel: ata6.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
Dec 27 02:00:41 UNRAID kernel: ata6.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
Dec 27 02:00:41 UNRAID kernel: ata6.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
Dec 27 02:00:41 UNRAID kernel: ata6.00: supports DRM functions and may not be fully accessible
Dec 27 02:00:41 UNRAID kernel: ata6.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
Dec 27 02:00:41 UNRAID kernel: ata6.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
Dec 27 02:00:41 UNRAID kernel: ata6.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
Dec 27 02:00:41 UNRAID kernel: ata6.00: supports DRM functions and may not be fully accessible
Dec 27 02:00:41 UNRAID kernel: ata6.00: configured for UDMA/133
Dec 27 02:00:41 UNRAID kernel: ata6: EH complete
Dec 27 02:00:41 UNRAID kernel: ata6.00: Enabling discard_zeroes_data
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#0 Sense Key : 0x5 [current] 
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#0 ASC=0x24 ASCQ=0x0 
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#0 CDB: opcode=0x93 93 08 00 00 00 00 33 00 00 22 00 7f ff ff 00 00
Dec 27 02:00:44 UNRAID kernel: print_req_error: critical target error, dev sde, sector 855638050
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#8 Sense Key : 0x5 [current] 
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#8 ASC=0x24 ASCQ=0x0 
Dec 27 02:00:44 UNRAID kernel: sd 6:0:0:0: [sde] tag#8 CDB: opcode=0x93 93 08 00 00 00 00 33 80 00 21 00 7f ff ff 00 00
Dec 27 02:00:44 UNRAID kernel: print_req_error: critical target error, dev sde, sector 864026657
Dec 27 02:00:44 UNRAID kernel: BTRFS warning (device sdd1): failed to trim 1 device(s), last error -121
Dec 27 02:00:44 UNRAID root: /etc/libvirt: 921 MiB (965738496 bytes) trimmed
Dec 27 02:00:44 UNRAID root: /var/lib/docker: 14.2 GiB (15242821632 bytes) trimmed
Dec 27 02:00:44 UNRAID sSMTP[12538]: Creating SSL connection to host
Dec 27 02:00:44 UNRAID sSMTP[12538]: SSL connection using *redacted*
Dec 27 02:00:48 UNRAID sSMTP[12538]: Sent mail for [email protected] (221 2.0.0 closing connection u126sm69083642pgb.2 - gsmtp) uid=0 username=xxxx outbytes=457

 

 

 

 

 

[JorgeB]

Please always post the complete diagnostics, but based on that I would guess a cable issue, replace both cables for that SSD.

[Drnick5]

Sorry about that. I just went to tools diagnostics and did a download of the logs, just not sure which ones are the most helpful to post as there are a ton. 

Attached is the entire syslog. which I think is what you're looking for.

 

All the Sata cables in the machine are brand new, I can try replacing them, just not sure which SSD is having the problem. 

My PSU doesn't have any extra power cables as I'm using all of them. Only way for me to replace that is to get a new power supply. 

syslog.txt

[JorgeB]

10 minutes ago, Drnick5 said:

Sorry about that. I just went to tools diagnostics and did a download of the logs, just not sure which ones are the most helpful to post as there are a ton. 

Attached is the entire syslog. which I think is what you're looking for.

For the future we mean the entire diagnostics, just post the generated zipped file.

 

Syslog appears to confirm there's a cable problem since trim worked sometimes and there are a lot of ATA errors before the latest ones:

 

Dec 21 02:00:13 UNRAID root: /etc/libvirt: 920.9 MiB (965607424 bytes) trimmed
Dec 21 02:00:13 UNRAID root: /var/lib/docker: 15.6 GiB (16754241536 bytes) trimmed
Dec 21 02:00:13 UNRAID root: /mnt/cache: 871 GiB (935236222976 bytes) trimmed
Dec 22 02:00:39 UNRAID kernel: ata6.00: exception Emask 0x0 SAct 0xffffffff SErr 0x0 action 0x6 frozen
Dec 22 02:00:39 UNRAID kernel: ata6.00: failed command: SEND FPDMA QUEUED
Dec 22 02:00:39 UNRAID kernel: ata6.00: cmd 64/01:00:00:00:00/00:00:00:00:00/a0 tag 0 ncq dma 512 out
Dec 22 02:00:39 UNRAID kernel:         res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 22 02:00:39 UNRAID kernel: ata6.00: status: { DRDY }
Dec 22 02:00:39 UNRAID kernel: ata6.00: failed command: SEND FPDMA QUEUED
Dec 22 02:00:39 UNRAID kernel: ata6.00: cmd 64/01:08:00:00:00/00:00:00:00:00/a0 tag 1 ncq dma 512 out
Dec 22 02:00:39 UNRAID kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 22 02:00:39 UNRAID kernel: ata6.00: status: { DRDY }
Dec 22 02:00:39 UNRAID kernel: ata6.00: failed command: SEND FPDMA QUEUED
Dec 22 02:00:39 UNRAID kernel: ata6.00: cmd 64/01:10:00:00:00/00:00:00:00:00/a0 tag 2 ncq dma 512 out
Dec 22 02:00:39 UNRAID kernel:         res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 22 02:00:39 UNRAID kernel: ata6.00: status: { DRDY }
Dec 22 02:00:39 UNRAID kernel: ata6.00: failed command: SEND FPDMA QUEUED
Dec 22 02:00:39 UNRAID kernel: ata6.00: cmd 64/01:18:00:00:00/00:00:00:00:00/a0 tag 3 ncq dma 512 out
Dec 22 02:00:39 UNRAID kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 22 02:00:39 UNRAID kernel: ata6.00: status: { DRDY }
Dec 22 02:00:39 UNRAID kernel: ata6.00: failed command: SEND FPDMA QUEUED
Dec 22 02:00:39 UNRAID kernel: ata6.00: cmd 64/01:20:00:00:00/00:00:00:00:00/a0 tag 4 ncq dma 512 out
Dec 22 02:00:39 UNRAID kernel:         res 40/00:00:00:01:80/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 22 02:00:39 UNRAID kernel: ata6.00: status: { DRDY }

Trim failed when the ATA errors coincided with the trim operation, the SSD with the issues is the Crucial, start by replacing the SATA cable.

 

[Drnick5]

Thanks for the help! I'll give that a shot tonight with another new cable!

I'm assuming if that doesn't fix it... its likely the power cable? (or possibly a failing drive?) 

[JorgeB]

If after replacing both cables issues persist try a different SATA port, very unlikely it's an SSD problem.

[Drnick5]

Thanks! Last question, if I replace the cables and its still an issue, Can I just move that drive to a new SATA port? Will I have to do anything in unRAID to reassign it to the cache pool after I move the port? or will it pick it back up automatically?

[JorgeB]

No need to do anything, Unraid tracks the attached devices by the serial number.