April 23, 20251 yr I'll just state, I am a novice and in no way suggesting the new Unraid version caused any issues - I am just highlighting that for my specific case I have had something weird go on in the last couple of days. Please be gentle with me. I wanted to share my findings and get some opinions; as I think I have found bad RAM. I recently replaced my cache drive after previous cache drive failure due to using in incorrect controller, that is now fixed and I am using a recommended controller... Potentially unrelated, in all fairness, but I didn't observe these sorts of errors before upgrade, and something strange happened the day after I upgraded after having a stable system for months. I haven't seen this before. I recently upgraded to Unraid 7.0.1. I noticed the following day that I came back to my machine in the morning and it was unresponsive to GUI, SSH, everything, I had to power cycle off manually (and thus lost all logs). This hasn't happened to me before. This morning I installed syslog to write to my cache in case it happened again, I'd set it up the day after the unresponsiveness but it didn't write correctly. However, overall I haven't observed any unresponsiveness since the initial episode (~2 days ago) However, upon using my system as normal today and attempting to install a new docker container (Immich by imagegenius), I started to get these failures when downloading the container: (I had previously installed the other postgres container from SpaceInvaderOne tutorial, which went fine and was running). Error: failed to register layer: archive/tar: invalid tar header I tried to install the docker again, and got this: Pulling image: ghcr.io/imagegenius/immich:latest IMAGE ID [2053444522]: Pulling from imagegenius/immich. IMAGE ID [114bfc582739]: Pulling fs layer. Downloading 100% of 31 MB. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [e1cde46db0e1]: Pulling fs layer. Downloading 100% of 6 KB. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [d403e9fdb1ff]: Pulling fs layer. Downloading 100% of 1 KB. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [fe7b05793469]: Pulling fs layer. Downloading 100% of 700 B. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [d04423d473c1]: Pulling fs layer. Downloading 100% of 427 B. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [ba37c0944114]: Pulling fs layer. Downloading 100% of 276 B. Download complete. Extracting. Pull complete. IMAGE ID [a3a7788d8bcf]: Pulling fs layer. Downloading 100% of 28 MB. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [cc0c773231d1]: Pulling fs layer. Downloading 100% of 4 KB. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [94dab31e49e0]: Pulling fs layer. Downloading 100% of 634 B. Verifying Checksum. Download complete. Extracting. Pull complete. IMAGE ID [8dfb42dbbd38]: Pulling fs layer. Downloading 100% of 475 MB. Verifying Checksum. Download complete. Extracting. IMAGE ID [0e4bb0a352c4]: Pulling fs layer. Downloading 100% of 385 MB. Verifying Checksum. Download complete. IMAGE ID [730e814ed456]: Pulling fs layer. Downloading 100% of 4 KB. Verifying Checksum. Download complete. TOTAL DATA PULLED: 919 MB Error: failed to register layer: read /var/lib/docker/tmp/GetImageBlob2883473593: input/output error I was seeing corruption from my Docker img in my logs: Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386097152 csum 0x5be13239 expected csum 0x5a5261a4 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 47, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386101248 csum 0x1605b97e expected csum 0x08a59cac mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 48, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386105344 csum 0x9f081d89 expected csum 0x60029bd5 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 49, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386109440 csum 0xf7335180 expected csum 0x1ecf0f34 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 50, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386113536 csum 0x09e00baf expected csum 0x352c45d0 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 51, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386117632 csum 0x295c1675 expected csum 0xaeaafcb7 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 52, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386121728 csum 0xccafe50e expected csum 0xa3a38d85 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 53, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386097152 csum 0x5be13239 expected csum 0x5a5261a4 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 54, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386097152 csum 0x5be13239 expected csum 0x5a5261a4 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 55, gen 0 Apr 23 12:45:24 ZOREO kernel: BTRFS warning (device loop2): csum failed root 5 ino 3652289 off 386097152 csum 0x5be13239 expected csum 0x5a5261a4 mirror 1 Apr 23 12:45:24 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 56, gen 0 I tried deleting my docker img and started to reinstall apps from CA Previous Apps section as my AppData backup and restore file would not import the previous AB backup file, stating "invalid source". So I had to do manually via Previous Applications. The data on my cache drive and app data is/ appears otherwise fine and preserved, just missing the containers. I added a couple of containers, (which operate just fine)... Then I started to see the same again in the logs, as below: Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33895 off 0 csum 0xd51a64e5 expected csum 0xb3c3997f mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33895 off 4096 csum 0x3959eaad expected csum 0xe31d428d mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33895 off 0 csum 0xd51a64e5 expected csum 0xb3c3997f mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33895 off 0 csum 0xd51a64e5 expected csum 0xb3c3997f mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33894 off 0 csum 0x6318f719 expected csum 0x0bcf6264 mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33894 off 4096 csum 0xf828e7d9 expected csum 0xf3c70c26 mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33894 off 0 csum 0xe4f2db55 expected csum 0x0bcf6264 mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33894 off 0 csum 0xe4f2db55 expected csum 0x0bcf6264 mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33893 off 0 csum 0x6498a8d9 expected csum 0xb6c1f1f7 mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 23 13:09:45 ZOREO kernel: BTRFS warning (device loop2): csum failed root 264 ino 33893 off 0 csum 0x6498a8d9 expected csum 0xb6c1f1f7 mirror 1 Apr 23 13:09:45 ZOREO kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 The SSD Cache drive looks fine itself after Smart Test: root@ZOREO:~# btrfs device stats /mnt/cache [/dev/mapper/sdf1].write_io_errs 0 [/dev/mapper/sdf1].read_io_errs 0 [/dev/mapper/sdf1].flush_io_errs 0 [/dev/mapper/sdf1].corruption_errs 21 [/dev/mapper/sdf1].generation_errs 0 root@ZOREO:~# smartctl -a /dev/sdf smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.78-Unraid] (local build) Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Samsung based SSDs Device Model: Samsung SSD 870 EVO 500GB Serial Number: S7EWNJ0WC28314V LU WWN Device Id: 5 002538 f33c0b619 Firmware Version: SVT02B6Q User Capacity: 500,107,862,016 bytes [500 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches TRIM Command: Available, deterministic, zeroed Device is: In smartctl database 7.3/5660 ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Apr 23 13:12:54 2025 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 0) seconds. Offline data collection capabilities: (0x53) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 85) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 1 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 4501 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 7 177 Wear_Leveling_Count 0x0013 100 100 000 Pre-fail Always - 0 179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0 181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0 182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0 183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0 187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0032 068 063 000 Old_age Always - 32 195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always - 0 199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 1 241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 901850932 252 Added_Bad_Flash_Blk_Ct 0x0032 100 100 000 Old_age Always - 1 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 3 - # 2 Short offline Completed without error 00% 0 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing 256 0 65535 Read_scanning was never started Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. The above only provides legacy SMART information - try 'smartctl -x' for more root@ZOREO:~# The new CA memtest app is showing me failures across the board after just 1 loop. I stopped it after that. [23.04.2025 13:43:29 BST] /usr/bin/memtester 13G 5 ============================================================================================= memtester version 4.6.0 (64-bit) adapted for use on Unraid installations Copyright (C) 2001-2020 Charles Cazabon Copyright (C) 2024 desertwitch (modifications for Unraid) THIS PROGRAM IS PROVIDED AS IS AND WITHOUT ANY WARRANTIES It is licensed under GNU General Public License Version 2 pagesize is 4096 pagesizemask is 0xfffffffffffff000 want 13312MB (13958643712 bytes) got 13312MB (13958643712 bytes), trying mlock ...locked. Loop 1/5: Stuck Address : Testing... failed Random Value : Testing... failed Compare XOR : Testing... failed Compare SUB : Testing... failed Compare MUL : Testing... failed Compare DIV : Testing... failed Compare OR : Testing... failed Compare AND : Testing... failed Sequential Increment : Testing... failed Solid Bits : Testing... failed Block Sequential : Testing... failed Checkerboard : Testing... failed Bit Spread : Testing... failed Bit Flip : Testing... failed Walking Ones : Testing... failed Walking Zeroes : Testing... failed 8-bit Writes : Testing... failed 16-bit Writes : Testing... failed Loop 2/5: Stuck Address : Testing... failed Random Value : Testing... ============================================================================================= [23.04.2025 13:46:38 BST] The operation was interrupted and has not finished. [23.04.2025 13:46:38 BST] Code: SIGTERM - Program Interrupt. I am guessing I'll have to replace the RAM and/or MoBo? Is there advice or expert opinion on this before I throw my system out the window? Is there anything else I should do or just be done with it and purchase some new RAM and see? The array is currently offline and I think I'll keep it offline for now. Thank you in advance. zoreo-diagnostics-20250423-1358.zip Edited April 23, 20251 yr by Zee-Unraid typo
April 23, 20251 yr Author 23 minutes ago, JorgeB said: Very possibly there is bad RAM, run memtest. After 10min, I'm guessing that's enough to confirm it with this volume of errors. Do you have any guidance for me/ point me to a thread for considerations when purchasing replacement RAM at all? Or does it just need to work with my MoBo (I am thinking about the time I was unaware there were controller specificities for Unraid and don't want to make the same mistake again) I am using: Gigabyte Technology Co., Ltd. B250M-D2VX-SI-CF Intel® Core™ i7-7700K CPU Thanks Edited April 23, 20251 yr by Zee-Unraid
April 23, 20251 yr Community Expert I would first try removing the DRAM, check that all looks undamaged and clean (no huge dust bunnies). Reinstall the DRAM and run Memtest again. You wrote you were recently inside the server, perhaps you disturbed something.
April 23, 20251 yr Author 16 minutes ago, ConnerVT said: I would first try removing the DRAM, check that all looks undamaged and clean (no huge dust bunnies). Reinstall the DRAM and run Memtest again. You wrote you were recently inside the server, perhaps you disturbed something. I took the ram out, inspected, pins all look fine, gave them a blow. Re-inserted into the MoBo, with the clicks so that they went in correctly. Mem test after ~1min... 7k errors...
April 23, 20251 yr Community Expert Solution You need to get a new DIMM, just needs to be compatible with the board/CPU.
April 23, 20251 yr Author Great, thanks for confirming and all your support on the forums. I've ordered Corsair Vengeance LPX 32 GB (2x16 GB) DDR4 2400 MHz CL14 (CMK32GX4M2A2400C14) Best regards, Zee
April 23, 20251 yr Community Expert 5 hours ago, Zee-Unraid said: I took the ram out, inspected, pins all look fine, gave them a blow. Re-inserted into the MoBo, with the clicks so that they went in correctly. Mem test after ~1min... 7k errors... It was worth a try. Zero cost to check that off the list as a suspect. Replacing DRAM would be my next step.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.