July 7, 20242 yr Ok, so I thought I'd fixed this but it seems not!! It's now way beyond frustrating, I've now replaced the NIC that I thought was the cause and replaced all the RAM and still Unraid is unstable! I actually think it's the most unstable system I've had! Even worse than Windows 95/98!! So here are the logs etc. Do I need to remove ZFS? I'm wondering if ZFS was a step too far for Unraid? syslog-10.19.64.2.log tower-diagnostics-20240703-1240.zip
July 7, 20242 yr Check this similar post for some initial steps to take. Patience now, this part sucks but it’s how you find the problem!
July 8, 20242 yr Author On 7/7/2024 at 3:05 PM, Andiroo2 said: Check this similar post for some initial steps to take. Patience now, this part sucks but it’s how you find the problem! Thanks, but nothing new in that thread. It's not quite the same issue as random reboots, which would definitely point at hardware PSU issues, but system crashing is just as likely to be software-related!
July 8, 20242 yr Community Expert Syslog shows multiple call traces, but they don't point to the reason, you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.
July 9, 20242 yr Author 19 hours ago, JorgeB said: Syslog shows multiple call traces, but they don't point to the reason, you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. Ok thanks Jorge, but the system has currently been up for 6 days and couldn't be without it for that long! I would have to leave it at least a couple of weeks to know if it's worked!
July 12, 20241 yr Author Ok, so it stayed up 9 days this time! Logs attached! I'm hoping that eventually, something will show!! syslog-10.19.64.2.log tower-diagnostics-20240712-1126.zip
July 12, 20241 yr Community Expert Server is not running in safe mode as suggested. I do see a couple of zfs related call traces, suggesting there may be an issue with one of those filesystem, or an underlying hardware issue.
July 15, 20241 yr Author How long would I need to leave it in safe mode? As I couldn't for more than 2 days as the server runs my phone system! Ok, I had noticed that ZFS Master often stops displaying anything, usually the day before the system crashes, so I've removed it to see what happens!
August 9, 20241 yr Author Ok, so 23 days this time! Definitely progress, but would really like it not to crash at all! I can see that it's still a zfs issue? Do I need to abandon zfs and go back to xfs? There is no point in these new features if they make the OS unstable! syslog-10.19.64.2.log tower-diagnostics-20240809-1029.zip
August 9, 20241 yr Community Expert 8 minutes ago, MothyTim said: There is no point in these new features if they make the OS unstable! lol, a zfs issue may be the problem for you, but it's not a general problem, lost of users have been using for months without issues, including myself, there can be an issue with that particular filesystem, or a hardware issue causing the problem, try formatting the pool btrfs or xfs and re-test.
August 9, 20241 yr Author Ok, lol, something I have noticed is that it always happens at 01:05 in the morning, I use the Spaceinvaderone script to do snapshots and replication, and that starts for the appdata folder at 01:00 each morning. It always seems to do the snapshots successfully and fails and crashes the system on replication! The cache drive SSD where the appdata folder is is zfs and one of the hard drives in the array is zfs and that is where it replicates too! So not sure what to change to fix/test etc? As I have a feeling that stopping replication will fix the issue, but then I'll have no backup! Or is there now a better way? Thanks for your help as always! 🙂
August 9, 20241 yr Community Expert You could try reformatting both fileystems as zfs and try again, or using btrfs snapshots and replication instead, that would confirm if the problem is with an existing zfs filesystem.
October 11, 20241 yr Author Hi, so I changed the replication destination to the other cache drive instead of the array and it ran for over a month without crashing! Much better! It crashed and corrupted the flash drive though and also blacklisted it as well which was weird! The first syslog is from then, which was a couple of weeks ago! It then crashed again yesterday, the second syslog is from then! I think it's possibly the same old zfs issue? I'm trying to decide whether to stick with zfs and see if V7 brings improvement, but I'm not holding out much hope after reading the release notes for the beta! Or give up and go back to btrfs and xfs?
October 11, 20241 yr Community Expert 7 minutes ago, MothyTim said: the second syslog is from then! Looks like you forgot to post that.
October 11, 20241 yr Community Expert Should be unrelated, but there are issues with one of the devices: Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=2s Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#25 Sense Key : 0x2 [current] Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#25 ASC=0x4 ASCQ=0x0 Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#25 CDB: opcode=0x28 28 00 52 08 48 70 00 00 a8 00 Sep 28 01:00:19 Tower kernel: I/O error, dev sdb, sector 1376274544 op 0x0:(READ) flags 0x700 phys_seg 3 prio class 2 Sep 28 01:00:19 Tower kernel: zio pool=cache_two vdev=/dev/sdb1 error=5 type=1 offset=704651517952 size=86016 flags=180980 Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=2s Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#27 Sense Key : 0x2 [current] Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#27 ASC=0x4 ASCQ=0x0 Sep 28 01:00:19 Tower kernel: sd 2:0:0:0: [sdb] tag#27 CDB: opcode=0x2a 2a 00 a4 00 41 e0 00 00 08 00 Sep 28 01:00:19 Tower kernel: I/O error, dev sdb, sector 2751480288 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 2 Sep 28 01:00:19 Tower kernel: zio pool=cache_two vdev=/dev/sdb1 error=5 type=2 offset=1408756858880 size=4096 flags=184880 Then there are a lot of zfs related crashes, also some NIC related ones, the zfs crashes could be hardware related, or less likely, a compatibility issue with your hardware, but you can try btrfs for example, if there are also issue with that one, it would suggest a likely hardware issue.
January 11, 20251 yr Author Hi, the server started crashing again daily, so I made time to reformat the cache drive to btrfs. Unfortunately, that has made no difference! As I just don't have money to throw away at the moment, so any ideas on what to replace first? Bear in mind that it has recently had new RAM and a new SFP+ NIC. Thanks, Tim tower-diagnostics-20250111-1321.zip syslog-10.19.64.2.log
January 11, 20251 yr Community Expert Solution There are constant call traces logged, start by running memtest, even if the RAM is new, or since memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.
January 12, 20251 yr Author Ok, so memtest won't run!! It just reboots the machine and takes me back to the boot menu?
January 12, 20251 yr Community Expert Are you using the latest version? Which Unraid release? Old memtest won't work with UEFI boot.
January 12, 20251 yr Author With CSM enabled in the BIOS it just reboots, with CSM disabled it just sits there saying "Loading /memtest... ok" and goes no further!
January 12, 20251 yr Community Expert 6.12.14 includes the new memtest, you can try this instead: 23 hours ago, JorgeB said: since memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.
January 12, 20251 yr Author Ok, so I went to memtest website and downloaded latest version put that on a stick and it ran ok, it stopped after 43 errors and said failed! So now running again with one stick! Will keep you posted! Thanks again for your help!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.