Sander de Ruiter

That wasn't it...but I did find a container that wrote to /var/lib/folder...Moved that to /mnt/cache/appdata/folder solved this. Thanks!

Hmm, I *do* have a share named appdata... What would be a definitive path to look out for in my container mappings, that certainly will write to RAM?

That was it (and not for the first time in my Unraid History...). I installed Homepage from the apps section, which had a default path to /mnt/user/appdata/homepage, which I think caused this?

How can I check the disk usage? This is the output I can generate: root@NAS:/# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 15347348 15347348 0 100% / tmpfs 32768 636 32132 2% /run /dev/sda1 15614032 913160 14700872 6% /boot overlay 15347348 15347348 0 100% /lib overlay 15347348 15347348 0 100% /usr devtmpfs 8192 0 8192 0% /dev tmpfs 15361108 0 15361108 0% /dev/shm tmpfs 131072 16788 114284 13% /var/log tmpfs 3072220 0 3072220 0% /run/user/0 /dev/md1p1 7811939620 7671723780 140215840 99% /mnt/disk1 /dev/nvme0n1p1 1953513560 190945720 1760428840 10% /mnt/cache shfs 7811939620 7671723780 140215840 99% /mnt/user0 shfs 7811939620 7671723780 140215840 99% /mnt/user /dev/loop2 41943040 8896468 32512092 22% /var/lib/docker

I have 32GB of ram.

Unraid 6.1.49, running stable for 2 weeks or so, and then I can no longer login to the frontend. User/pass are accepted but screen stays on login screen. Running diagnostics over SSH gives the following: Starting diagnostics collection... tail: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device echo: write error: No space left on device tr: write error: No space left on device done. ZIP file '/boot/logs/tower-diagnostics-20231225-1327.zip' created. I'm assuming it's the flash drive that's out of space? If so, that hasn't happened in the last 4 years and it has plenty of space left (from memory). Any hints?

If by server you mean unraid server, then yes, no issue there. Could memory be an issue ? Do docker containers have a memory limit assigned?

It's my own project...

Hi, Tearing my hear out here, I really cannot figure this out. I have a docker image (hosted on Docker Hub), that I have deployed to two separate docker containers in Unraid. NginxProxyManager is the entry point, I have PROD running on port 9999 and dev on 10000. DEV gets a few hits from me and then sits idle. PROD gets 10-20 hits in 5 min or so, and then, without any error, the site will report a 504 Gateway Timeout. There's nothing in the docker log for the PROD container, if I try to go to the console, I can't login (as it's not running/responding). I have a separate service that pings PROD every 5 min (visible in logs), I tried this to make sure the site stays alive, but apparently, it's not this. What would be an appropriate next step to take to try to debug this? is there any log file I can look up that I missed?

Well, another day, another issue, sigh. Array went into parity sync after a clean reboot, took 15 hours or so, finished with no errors. This morning I woke up to another error: 01-08-2021 06:10 Unraid array errors Warning [NAS] - array has errors Array has 1 disk with read errors warning 01-08-2021 06:10 Unraid Disk 2 error Alert [NAS] - Disk 2 in error state (disk dsbl) ST6000DM003-2CY186_ZF2032KA (sde) alert I've attached the logs again. There is a new SATA controller on the way. This seems to be the relevant section: Aug 1 06:00:54 NAS emhttpd: read SMART /dev/sdd Aug 1 06:04:32 NAS kernel: ata6.00: exception Emask 0x0 SAct 0x10000 SErr 0x0 action 0x6 frozen Aug 1 06:04:32 NAS kernel: ata6.00: failed command: READ FPDMA QUEUED Aug 1 06:04:32 NAS kernel: ata6.00: cmd 60/08:80:b8:de:a4/00:00:9b:02:00/40 tag 16 ncq dma 4096 in Aug 1 06:04:32 NAS kernel: res 40/00:00:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout) Aug 1 06:04:32 NAS kernel: ata6.00: status: { DRDY } Aug 1 06:04:32 NAS kernel: ata6: hard resetting link Aug 1 06:04:42 NAS kernel: ata6: softreset failed (1st FIS failed) Aug 1 06:04:42 NAS kernel: ata6: hard resetting link Aug 1 06:04:52 NAS kernel: ata6: softreset failed (1st FIS failed) Aug 1 06:04:52 NAS kernel: ata6: hard resetting link ... Aug 1 06:05:27 NAS kernel: ata6: softreset failed (1st FIS failed) Aug 1 06:08:18 NAS kernel: ata6.00: disabled Aug 1 06:08:28 NAS kernel: ata6: softreset failed (1st FIS failed) ### [PREVIOUS LINE REPEATED 2 TIMES] ### Aug 1 06:09:19 NAS kernel: ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Aug 1 06:09:19 NAS kernel: ata6.00: link online but device misclassified Aug 1 06:09:19 NAS kernel: ata6: EH complete Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#31 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=307s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#31 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00 Aug 1 06:09:19 NAS kernel: blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 a1 68 6e b8 00 00 00 20 00 00 Aug 1 06:09:19 NAS kernel: blk_update_request: I/O error, dev sde, sector 2707975864 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 0 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=2707975800 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=2707975808 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=2707975816 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=2707975824 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#1 CDB: opcode=0x88 88 00 00 00 00 02 84 01 28 98 00 00 00 20 00 00 Aug 1 06:09:19 NAS kernel: blk_update_request: I/O error, dev sde, sector 10804603032 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 0 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=10804602968 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=10804602976 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=10804602984 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=10804602992 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#2 CDB: opcode=0x88 88 00 00 00 00 00 46 c7 18 80 00 00 00 08 00 00 Aug 1 06:09:19 NAS kernel: blk_update_request: I/O error, dev sde, sector 1187453056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=1187452992 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=317s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#3 CDB: opcode=0x88 88 00 00 00 00 02 9b a4 de b8 00 00 00 08 00 00 Aug 1 06:09:19 NAS kernel: blk_update_request: I/O error, dev sde, sector 11201207992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=11201207928 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#4 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00 Aug 1 06:09:19 NAS kernel: blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 ### [PREVIOUS LINE REPEATED 4 TIMES] ### Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#18 CDB: opcode=0x8a 8a 00 00 00 00 01 80 39 4b 90 00 00 00 08 00 00 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=6446205776 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#19 CDB: opcode=0x88 88 00 00 00 00 01 80 39 4b d0 00 00 00 08 00 00 Aug 1 06:09:19 NAS kernel: md: disk2 read error, sector=6446205840 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#21 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#21 CDB: opcode=0x8a 8a 00 00 00 00 01 80 39 4b 98 00 00 00 08 00 00 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=6446205784 Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Aug 1 06:09:19 NAS kernel: sd 6:0:0:0: [sde] tag#22 CDB: opcode=0x8a 8a 00 00 00 00 01 80 39 4b a0 00 00 00 08 00 00 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=6446205792 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=6446205800 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=6446205808 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=6446205816 ... Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=10804602976 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=10804602984 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=10804602992 Aug 1 06:09:19 NAS kernel: md: disk2 write error, sector=11201207928 Is this again related to the controller, or is this a genuine disk error (disk is somewhat old)? nas-diagnostics-20210801-1051.zip

Thanks!

Thank you. Can you hint at where in the logs you would find evidence for this? I'm trying to learn to read the logs better myself.

Out of nowhere (no change to system) these errors popped up on all drives in the array (see attached logs). Jul 31 12:27:31 NAS kernel: BTRFS warning (device sdb1): direct IO failed ino 11586 rw 0,0 sector 0x34d0ca8 len 0 err no 10 Jul 31 12:27:32 NAS kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 4913238, flush 0, corrupt 0, gen 0 Jul 31 12:27:35 NAS kernel: blk_update_request: I/O error, dev sdb, sector 6139136 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0 Any hint what could be a probable cause for this? Should I be worried? nas-diagnostics-20210731-1221.zip

Ok thanks. Two questions: 1. disk2 is disabled and contents are emulated. How can I restore disk2 to a normal state (if that is the right thing to do, given it was a controller problem)? 2. Is there a page with links to add-on controllers? I have no clue what to search for for purchase.

Well, not sure if I'm just unlucky, or something else. Two days ago the replacement 8TB arrived. I've swapped the new one for the faulty one, made sure all connections on the board and drives were sound, and started rebuilding the array. Done after 16 hours, no errors reported. The array has performed without reporting errors for 1 day, and just now I woke up to: I've attached the logs again, but I'm really at a loss here. nas-diagnostics-20210708-0756.zip

Alright, 8TB replacement ordered and the current one RMA'd. Will replace power/connection on disk2 and swap disk1 when the new 8TB arrives. Will report back.

Hi, parity sync finished after nearly 16 hours. I've attached the latest diags. What worries me is the the new drive (Disk 1ST8000VN004-2M2101_WSD0MT7X - 8 TB (sdc)) is already showing errors and that the 6TB had 588 errors during parity sync. Question 1: should I return the new disk and ask for a replacement (hopefully without errors)? Question 2: do these parity sync errors on 588 have any repercussions on the state of the array? Thanks nas-diagnostics-20210704-0620.zip

Thanks both! Will report back tomorrow after replacing the disk and performing the steps outlined.

Agreed. I've now swapped two power cables, but the end result is the same. My conclusion is that disk1 is bad. Now for the solution: I've purchased an additional 8TB drive, which arrives tomorrow. This will replace the 6TB disk1. What's the appropriate procedure to make this work?

Apologies, I did not replace the cable yet. However, I've now replace the SATA data cable for disk1, and attached new logs. My casual inspection of the logs leads me to believe the new cable didn't make a difference? nas-diagnostics-20210702-1704.zip

Array is stopped (I did not have auto start on), disk1 is unmounted (device is missing, disabled). Attached is new diags. nas-diagnostics-20210702-1635.zip

It's running headless, I need to drag it out of it's location to inspect. Back soon.

I did a reboot, but now that server is not responding and/or coming back up. I also cannot ping the Unraid IP (host is down). Any hints?

It’s still doing a read check. Can I do a shutdown now?

Hi, Array with new parity disk (8TB) and 2 6TB chucked disks. Yesterday a parity check led to read errors (2048 errors on disk1) and stopped. Unraid offered to do a read-check, which I did. After 2-3 hours, disk2 is now at 400M+ read errors.... Have I lost all data? nas-diagnostics-20210702-1401.zip