JorgeB

See if the actual disk4 mounts with the UD plugin, you'll need to unassign it first and best to use the read-only mount option.

Looks more like a connection problem, try testing without the expander/backplane, or new cables.

Were there any errors during the rebuild? Please post the diagnostics, ideally before rebooting: Tools -> Diagnostics

Invalid partition doesn't necessarily mean a drive problem, just that the partition doesn't conform to what Unraid expects, though that should not happen without a reason, diagnostics might give more clues.

I don't need them now, was just saying for next time, since the syslog is included as well as much more info, for now see if the failing disks share something other than the HBA and PSU, like a miniSAS cable or power splitter, if not try another PSU or HBA.

Next time please post the full diagnostics, but simultaneous errors on multiple disks Jan 29 10:21:58 PapandriaServer kernel: md: disk6 read error, sector=1332952 Jan 29 10:21:58 PapandriaServer kernel: md: disk6 read error, sector=1332960 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#319 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#319 Sense Key : 0x2 [current] Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#319 ASC=0x4 ASCQ=0x0 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#319 CDB: opcode=0x8a 8a 00 00 00 00 00 00 00 00 40 00 00 00 08 00 00 Jan 29 10:21:58 PapandriaServer kernel: print_req_error: I/O error, dev sdb, sector 64 Jan 29 10:21:58 PapandriaServer kernel: md: disk6 write error, sector=0 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#320 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#320 Sense Key : 0x2 [current] Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#320 ASC=0x4 ASCQ=0x0 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:0:0: [sdb] tag#320 CDB: opcode=0x8a 8a 00 00 00 00 00 00 00 00 58 00 00 00 10 00 00 Jan 29 10:21:58 PapandriaServer kernel: print_req_error: I/O error, dev sdb, sector 88 Jan 29 10:21:58 PapandriaServer kernel: md: disk6 write error, sector=24 Jan 29 10:21:58 PapandriaServer kernel: md: disk6 write error, sector=32 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:2:0: [sdd] tag#323 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:2:0: [sdd] tag#323 Sense Key : 0x2 [current] Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:2:0: [sdd] tag#323 ASC=0x4 ASCQ=0x0 Jan 29 10:21:58 PapandriaServer kernel: sd 6:0:2:0: [sdd] tag#323 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 c0 00 00 00 20 00 00 Jan 29 10:21:58 PapandriaServer kernel: print_req_error: I/O error, dev sdd, sector 192 Jan 29 10:21:58 PapandriaServer kernel: md: disk7 read error, sector=128 Jan 29 10:21:58 PapandriaServer kernel: md: disk7 read error, sector=136 Jan 29 10:21:58 PapandriaServer kernel: md: disk7 read error, sector=144 Jan 29 10:21:58 PapandriaServer kernel: md: disk7 read error, sector=152 suggest a controller, power/connection problem.

Yep.

The reason you need a SAS HBA is because of the SAS expander, an 8 drive chassis could also have one, but it usually doesn't since just two cables are enough to connect all the drives directly.

This thread is meant to replace the now outdated old one about recommended controllers, these are some controllers known to be generally reliable with Unraid: Note: RAID controllers are not recommended for Unraid, this includes all LSI MegaRAID models, doesn't mean they cannot be used but there could be various issues because of that, like no SMART info and/or temps being displayed, disks not being recognized by Unraid if the controller is replaced with a different model, and in some cases the partitions can become invalid, requiring rebuilding all the disks. 2 ports: Asmedia ASM1061/62 (PCIe 2.0 x1) or JMicron JMB582 (PCIe 3.0 x1) 4 ports: Asmedia ASM1064 (PCIe 3.0 x1) or ASM1164 (PCIe 3.0 x4 physical, x2 electrical, though I've also seen some models using just x1) 5 ports: JMicron JMB585 (PCIe 3.0 x4 - x2 electrically) These JMB controllers are available in various different SATA/M.2 configurations, just some examples: 6 ports: Asmedia ASM1166 (PCIe 3.0 x4 physical, x2 electrical) * * There have been some reports that some of these need a firmware update for stability and/or PCIe ASPM support, see here for instructions. These exist with both x4 (x2 electrical) and x1 PCIe interface, for some use cases the PCIe x1 may be a good option, i.e., if you don't have larger slots available, though bandwidth will be limited: 8 ports: any LSI with a SAS2008/2308/3008/3408/3808 chipset in IT mode, e.g., 9201-8i, 9211-8i, 9207-8i, 9300-8i, 9400-8i, 9500-8i, etc and clones, like the Dell H200/H310 and IBM M1015, these latter ones need to be crossflashed (most of these require a x8 or x16 slot, older models like the 9201-8i and 9211-8i are PCIe 2.0, newer models like the 9207-8i, 9300-8i and newer are PCIe 3.0) For these and when not using a backplane you need SAS to SATA breakout cables, SFF-8087 to SATA for SAS2 models: SFF-8643 to SATA for SAS3 models: Keep in mind that they need to be forward breakout cables (reverse breakout look the same but won't work, as the name implies they work for the reverse, SATA goes on the board/HBA and the miniSAS on a backplane), sometimes they are also called Mini SAS (SFF-8xxx Host) to 4X SATA (Target), this is the same as forward breakout. If more ports are needed you can use multiple controllers, controllers with more ports (there are 16 and 24 port LSI HBAs, like the 9201-16i, 9305-16i, 9305-24i, etc) or use one LSI HBA connected to a SAS expander, like the Intel RES2SV240 or HP SAS expander. P.S. Avoid SATA port multipliers with Unraid, also avoid any Marvell controller. For some performance numbers on most of these see below:

Avoid any SATA port multiplier with Unraid, also avoid any Marvell controller, these are OK: 4 Ports look for Asmedia ASM1064 or ASM1164 5 Ports look for JMB585 6 Ports look for Asmedia ASM1166 8 Ports look for LSI with a SAS2008/2308/3008/3408 chipset in IT mode, e.g., 9201-8i, 9211-8i, 9207-8i, 9300-8i, 9400-8i, etc and clones, like the Dell H200/H310 and IBM M1015, these latter ones need to be crossflashed.

No, they run pretty cool.

One thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Not that one, avoid any Marvell controller and any controller with port multipliers, that one has both. 4 Ports look for Asmedia ASM1064 or ASM1164 5 Ports look for JMB585 6 Ports look for Asmedia ASM1166 8 Ports look for LSI with a SAS2008/2308/3008/3408 chipset in IT mode, e.g., 9201-8i, 9211-8i, 9207-8i, 9300-8i, 9400-8i, etc and clones, like the Dell H200/H310 and IBM M1015, these latter ones need to be crossflashed.

Filesystem corruption on an emulated disk is not that uncommon, sometimes it just happens after the disk gets disable, in this case there might be other reasons, but can't say without the previous syslog files, from when the disk first got disabled, and the ensuing paririty1 issues.

9200-8e, 9207-8e or similar, assuming the cables you have are sff-8088

You only need one cable, but also need a SAS HBA, can't use onboard SATA with a SAS expander, using two cables will double available bandwidth, but it won't be needed for 12 disks, assuming LSI SAS3 HBA with SATA3 disks.

Disk is failing to read: Jan 28 18:12:47 Servme kernel: sd 12:0:0:0: [sdk] tag#434 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Jan 28 18:12:47 Servme kernel: sd 12:0:0:0: [sdk] tag#434 Sense Key : 0x3 [current] [descriptor] Jan 28 18:12:47 Servme kernel: sd 12:0:0:0: [sdk] tag#434 ASC=0x11 ASCQ=0x0 Jan 28 18:12:47 Servme kernel: sd 12:0:0:0: [sdk] tag#434 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 Jan 28 18:12:47 Servme kernel: print_req_error: critical medium error, dev sdk, sector 0 Jan 28 18:12:50 Servme kernel: sd 12:0:0:0: [sdk] tag#434 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Jan 28 18:12:50 Servme kernel: sd 12:0:0:0: [sdk] tag#434 Sense Key : 0x3 [current] [descriptor] Jan 28 18:12:50 Servme kernel: sd 12:0:0:0: [sdk] tag#434 ASC=0x11 ASCQ=0x0 Jan 28 18:12:50 Servme kernel: sd 12:0:0:0: [sdk] tag#434 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 Jan 28 18:12:50 Servme kernel: print_req_error: critical medium error, dev sdk, sector 0 Jan 28 18:12:50 Servme kernel: Buffer I/O error on dev sdk, logical block 0, async page read Jan 28 18:12:50 Servme emhttpd: error: ckmbr, 2030: Input/output error (5): read: /dev/sdk Not sure why, but SMART has a lot of logged issues, possibly a device problem.

As a result of this conversation, the way Main calculates used/free space for btrfs was changed starting with -beta25, IIRC.

I didn't really understand the above using a translator, but there are several ATA errors on multiple devices, suggesting a controller or connection/power problem, just a small example: Jan 28 02:08:39 U3 kernel: ata10.00: exception Emask 0x50 SAct 0x20000000 SErr 0x4890800 action 0xe frozen Jan 28 02:08:39 U3 kernel: ata10.00: irq_stat 0x08400040, interface fatal error, connection status changed Jan 28 02:08:39 U3 kernel: ata10: SError: { HostInt PHYRdyChg 10B8B LinkSeq DevExch } Jan 28 02:08:39 U3 kernel: ata10.00: failed command: READ FPDMA QUEUED Jan 28 02:08:39 U3 kernel: ata10.00: cmd 60/80:e8:d0:13:13/00:00:00:00:00/40 tag 29 ncq dma 65536 in Jan 28 02:08:39 U3 kernel: res 40/00:e8:d0:13:13/00:00:00:00:00/40 Emask 0x50 (ATA bus error) Jan 28 02:08:39 U3 kernel: ata10.00: status: { DRDY } Jan 28 02:08:39 U3 kernel: ata10: hard resetting link Jan 28 02:08:39 U3 kernel: ata9.00: exception Emask 0x10 SAct 0x6000 SErr 0x4090000 action 0xe frozen Jan 28 02:08:39 U3 kernel: ata9.00: irq_stat 0x00400040, connection status changed Jan 28 02:08:39 U3 kernel: ata9: SError: { PHYRdyChg 10B8B DevExch } Jan 28 02:08:39 U3 kernel: ata9.00: failed command: READ FPDMA QUEUED Jan 28 02:08:39 U3 kernel: ata15.00: exception Emask 0x10 SAct 0x3 SErr 0x1990100 action 0xe frozen Jan 28 02:08:39 U3 kernel: ata9.00: cmd 60/40:68:90:0e:13/05:00:00:00:00/40 tag 13 ncq dma 688128 in Jan 28 02:08:39 U3 kernel: res 40/00:70:d0:13:13/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jan 28 02:08:39 U3 kernel: ata15.00: irq_stat 0x08400000, PHY RDY changed Jan 28 02:08:39 U3 kernel: ata9.00: status: { DRDY } Jan 28 02:08:39 U3 kernel: ata15: SError: { UnrecovData PHYRdyChg 10B8B Dispar LinkSeq TrStaTrns } Jan 28 02:08:39 U3 kernel: ata9.00: failed command: READ FPDMA QUEUED Jan 28 02:08:39 U3 kernel: ata15.00: failed command: READ FPDMA QUEUED Jan 28 02:08:39 U3 kernel: ata15.00: cmd 60/40:00:90:0e:13/05:00:00:00:00/40 tag 0 ncq dma 688128 in Jan 28 02:08:39 U3 kernel: res 40/00:08:d0:13:13/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jan 28 02:08:39 U3 kernel: ata9.00: cmd 60/80:70:d0:13:13/00:00:00:00:00/40 tag 14 ncq dma 65536 in Jan 28 02:08:39 U3 kernel: res 40/00:70:d0:13:13/00:00:00:00:00/40 Emask 0x10 (ATA bus error) As for the unmountable disk there's filesystem corruption: Jan 28 01:51:42 U3 kernel: BTRFS warning (device md3): md3 checksum verify failed on 22102016 wanted CB943766 found 565F6860 level 0 Jan 28 01:51:42 U3 kernel: BTRFS warning (device md3): md3 checksum verify failed on 22102016 wanted 7EA49A72 found 222D9443 level 0 and with btrfs not And with btrfs not always so easy to recover, you can see here for some options: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=543490

Jan 28 16:15:23 TheVault kernel: pm80xx 0000:27:00.0: pm80xx: driver version 0.1.40 Jan 28 16:15:23 TheVault kernel: pm80xx0:: pm8001_pci_probe 1111:chip_init failed [ret: -16] Controller problem, it's failing to initialize, older release uses an older driver: Jan 28 16:30:14 TheVault kernel: pm80xx 0000:27:00.0: pm80xx: driver version 0.1.39 You'll need to wait for a new driver, and hope it fixes it, there have been other issues with theses controllers before, if it's an option I would recommend using an LSI instead, you'll need a new cable though.

Rebooting should solve this.

If Parity1 was removed and the array re-started the missing disk would be emulated by parity2, please post current diagnostics.

Looks to me like dashboard is still using old way of calculating the space, should now use the same as main, but @bonienlhas been MIA?

Please use the existing docker support thread.

They're fine for now, keep monitoring.