jj_uk Posted June 8, 2021 Share Posted June 8, 2021 (edited) My log looks like this: Memory->log is 100%. Docker containers have stopped working and container webgui's are not accessible. I don't know what to do. Please help. tower1-diagnostics-20210608-2133.zip Edited June 8, 2021 by jj_uk Quote Link to comment
JorgeB Posted June 9, 2021 Share Posted June 9, 2021 Corruption was detected on the cache filesystem and it went read-only: Jun 8 07:29:47 tower1 kernel: BTRFS error (device dm-3): block=1041472405504 write time tree block corruption detected Jun 8 07:29:47 tower1 kernel: BTRFS: error (device dm-3) in btrfs_commit_transaction:2377: errno=-5 IO failure (Error while writing out transaction) Jun 8 07:29:47 tower1 kernel: BTRFS info (device dm-3): forced readonly You should backup and restore data, docker image should then be recreated. Quote Link to comment
jj_uk Posted June 9, 2021 Author Share Posted June 9, 2021 I have a backup from 3am on the same date, so i'll try to restore from that. Is it just a case of wiping the Cache drives and restoring them? The cache is 2 SSD drives. Quote Link to comment
JorgeB Posted June 9, 2021 Share Posted June 9, 2021 1 hour ago, jj_uk said: Is it just a case of wiping the Cache drives and restoring them? Yes, if everything important there is backed up. Quote Link to comment
jj_uk Posted June 9, 2021 Author Share Posted June 9, 2021 Looks like one of the SSDs (sdc) has also failed. SMART shows: 202 Percent_Lifetime_Remain ----CK 000 000 001 NOW 100 so i'll have to replace that too. Has data corruption already occurred at this point in time? Do I follow the instructions here: https://wiki.unraid.net/Replace_A_Cache_Drive to get all the data on to the array, then replace the disk? Would it be a good idea to replace the failed SSD with a 1TB drive? The other is 500M, the cache is usually at about 60%. Quote Link to comment
JorgeB Posted June 10, 2021 Share Posted June 10, 2021 12 hours ago, jj_uk said: so i'll have to replace that too. That attribute doesn't mean the SSD failed, just means it's past the expected life, it can still last a long time, my cache NVMe device is way past it's predicted life (127%) and still going strong. 12 hours ago, jj_uk said: Has data corruption already occurred at this point in time? Most if not all data should be time, it went read-only to avoid further filesystem corruption, you can copy the data using your favorite tool, not sure the mover will work correctly because it can't delete source files, never tried. Quote Link to comment
jj_uk Posted June 10, 2021 Author Share Posted June 10, 2021 Mover doesn't seem to be doing anything when i set the shares to 'YES'. I can't use krusader to move the files because docker doesn't work. What other choices are there to get the files off the cache and on to the array? Quote Link to comment
JonathanM Posted June 10, 2021 Share Posted June 10, 2021 10 minutes ago, jj_uk said: What other choices are there to get the files off the cache and on to the array? mc at a terminal. Midnight commander is a two pane file manager, GUI-ish. Just be sure you stay out of the /mnt/user tree since you need to work with /mnt/cache and /mnt/diskX Quote Link to comment
jj_uk Posted June 10, 2021 Author Share Posted June 10, 2021 Thanks- It's copying to /mnt/disk4/cache-backup/ Once this is done, what do I do next? Quote Link to comment
JorgeB Posted June 10, 2021 Share Posted June 10, 2021 Wipe both cache devices, you can use blkdiscard -f /dev/sdX with the array stopped, then start array and format pool. Quote Link to comment
jj_uk Posted June 10, 2021 Author Share Posted June 10, 2021 Cache drives are now formatted. I've started to copy the data back to the cache drive. What do I need to do with docker? it was mentioned at the start of this thread that I have to recreate the docker image, but I'm not sure how. Quote Link to comment
jj_uk Posted June 10, 2021 Author Share Posted June 10, 2021 I've figured it out. Everything now appears to be running. Thanks for all your help guys. I can now stop panicking! 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.