June 8, 20215 yr My log looks like this: Memory->log is 100%. Docker containers have stopped working and container webgui's are not accessible. I don't know what to do. Please help. tower1-diagnostics-20210608-2133.zip Edited June 8, 20215 yr by jj_uk
June 9, 20215 yr Community Expert Corruption was detected on the cache filesystem and it went read-only: Jun 8 07:29:47 tower1 kernel: BTRFS error (device dm-3): block=1041472405504 write time tree block corruption detected Jun 8 07:29:47 tower1 kernel: BTRFS: error (device dm-3) in btrfs_commit_transaction:2377: errno=-5 IO failure (Error while writing out transaction) Jun 8 07:29:47 tower1 kernel: BTRFS info (device dm-3): forced readonly You should backup and restore data, docker image should then be recreated.
June 9, 20215 yr Author I have a backup from 3am on the same date, so i'll try to restore from that. Is it just a case of wiping the Cache drives and restoring them? The cache is 2 SSD drives.
June 9, 20215 yr Community Expert 1 hour ago, jj_uk said: Is it just a case of wiping the Cache drives and restoring them? Yes, if everything important there is backed up.
June 9, 20215 yr Author Looks like one of the SSDs (sdc) has also failed. SMART shows: 202 Percent_Lifetime_Remain ----CK 000 000 001 NOW 100 so i'll have to replace that too. Has data corruption already occurred at this point in time? Do I follow the instructions here: https://wiki.unraid.net/Replace_A_Cache_Drive to get all the data on to the array, then replace the disk? Would it be a good idea to replace the failed SSD with a 1TB drive? The other is 500M, the cache is usually at about 60%.
June 10, 20215 yr Community Expert 12 hours ago, jj_uk said: so i'll have to replace that too. That attribute doesn't mean the SSD failed, just means it's past the expected life, it can still last a long time, my cache NVMe device is way past it's predicted life (127%) and still going strong. 12 hours ago, jj_uk said: Has data corruption already occurred at this point in time? Most if not all data should be time, it went read-only to avoid further filesystem corruption, you can copy the data using your favorite tool, not sure the mover will work correctly because it can't delete source files, never tried.
June 10, 20215 yr Author Mover doesn't seem to be doing anything when i set the shares to 'YES'. I can't use krusader to move the files because docker doesn't work. What other choices are there to get the files off the cache and on to the array?
June 10, 20215 yr 10 minutes ago, jj_uk said: What other choices are there to get the files off the cache and on to the array? mc at a terminal. Midnight commander is a two pane file manager, GUI-ish. Just be sure you stay out of the /mnt/user tree since you need to work with /mnt/cache and /mnt/diskX
June 10, 20215 yr Author Thanks- It's copying to /mnt/disk4/cache-backup/ Once this is done, what do I do next?
June 10, 20215 yr Community Expert Wipe both cache devices, you can use blkdiscard -f /dev/sdX with the array stopped, then start array and format pool.
June 10, 20215 yr Author Cache drives are now formatted. I've started to copy the data back to the cache drive. What do I need to do with docker? it was mentioned at the start of this thread that I have to recreate the docker image, but I'm not sure how.
June 10, 20215 yr Author I've figured it out. Everything now appears to be running. Thanks for all your help guys. I can now stop panicking!
Archived
This topic is now archived and is closed to further replies.