Marshalleq

I get this pretty consistently. I don't know if it's an issue or not to be honest, I've done some pretty extensive memory tests, was the only thing I could think of. See screenshot for what I'm talking about, and I've attached logs for those that might be feeling interested or helpful Many thanks, Marshalleq obi-wan-diagnostics-20201203-1833.zip

Updated one box - working great so far thanks @steini84 I'm liking I can be on stable builds again too.

I wish people would look past the words people write and look at the intent of words and what they are trying to say. If it is unclear, ask before reacting. It's advice I haven't always managed to follow myself, but I do try.

Yes exactly this. Thanks.

Normal humans often can't see all the ways messaging can be interpreted (this is why we have comms people) The most offensive sounding things are often not intended in that fashion at all. Written text makes communication harder because there are no facial or audio cues to support the language I expected our community developers (being that they've clearly communicated in text behind screens for many years) would understand that things aren't always intended as they sound. In this regard, I support @limetech wholeheartedly. Nevertheless the only way to fix this is probably for @limetech to privately offer an apology and discuss as a group how to fix, then publish back together that it's resolved. (30 years managing technical teams - seen this a few times and it's usually sorted out with an apology and a conversation).

Did you read the beta 35 notes? It seems that there's some new methods for deploying drivers and modules. They actually include in tree AMD GPU too. At first I was thinking this was going to save the day - then I realised we still have to compile it against the new kernel right? I've found these particular new Unraid beta's to be the worst betas I've ever used for some reason, so I think I'll stay away until they at least put the new linux kernel in.

I've recently been wondering about the speed of rebuild's / checks on USB. I have a 5 bay USB 3 box that came from a QNAP. It's seen in unraid. But I think it will be too slow.

Assuming you're not running any impacted 3rd party plugins like ZFS, you also need to ensure your cache drive is backed up. I'm not sure there's an automated way - I think you have to copy it somewhere else and reformat the drive after you've downgraded, then put the data back in. Could be wrong. It definitely kicks it out of the array though.

I have Roon and it’s not crashing. The other server without Roon is crashing on occasion (not saying it’s same issue but just to say Roon shouldn’t be). Roon does push a disk while it’s scanning and so on so possibly something more down the i/o path given your comments about disks. Could even be sata cable. Sent from my iPhone using Tapatalk

Thanks - I actually installed a new version of 6.8.3 on my USB using the Mac install tool, which in the process wipes the usb stick. I copied over no config and installed ZFS from the App Store and somehow that ended up with the beta4 version of zfs. So I don't know. Anyway a bit of a learning for me! I have to wonder what is on that disk that's so problematic - I only created it recently. I have a suspicion it may have a duplicate mount point or something.

Multiple times today. Never noticed it before. I'd say about three separate times (hours between) today.

I am assuming that the one disk, had some ZFS config on it, that was interfering with the rest. The thing I don't understand is why it was playing havoc with the rest of the whole system. I'm relatively new to ZFS so I guess what do I know, but logically if the system is on a USB in a path that has nothing to do with the ZFS volume, this kind of extenuating circumstance should not happen. Until I know for sure I'll consider this my first ZFS drama. The awesome thing is that throughout, no ZFS data was lost, all pools were active and running. Tested multiple times - even did a scrub. Anyway, I am using kernel helper again as it's easier to keep control of ZFS versions and I did notice the nice ZFS export notice at the bottom of the screen - nice job!

I unplugged one disk and it seems to have brought it back to life. I'm just going to pretend I fixed it for tonight. Too scared to look at it again for now.

Yes, it usually runs on USB3.0, but for this I'm using USB2 to be safe. I half thought it was ZFS again, but known good versions of ZFS and precompiled kernels with ZFS are also triggering the problems. I am unable to provide diagnostics, they don't run, not even in console which is what the attached file above is about. I tried syslog, but nothing is getting logged. It seems the system can't create (or thinks it can't create) files. I can create files in the console though. So I think it's broken too, which is what I was expecting to be honest.

Been getting a lot of these today.

I use /mnt/x y z I've just reformatted the key and this time not copied the config. It was all working fine until I added ZFS. I note that only the beta of ZFS is now being offered not stable, which I think is a huge mistake given it's the stable 6.8.3 I have installed. @steini84 perhaps you could implement something so that stable / next can be chosen for ZFS. Anyway, I'm back to believing it's a ZFS issue. I don't know why it is. I'm going to try the community kernel and see if that sheds any light. Community Kernel (Unraid Kernel Helper) same result (using my normal config though, but I did prove it did this with none of my config also). This is a nightmare. The only thing I can say for sure is it only appears when ZFS is installed. But earlier known good versions of ZFS also exhibit the behaviour.

This really has me stumped. I am unable to run diagnostics, even from the console. The GUI has text strewn through it, docker and vm menus are missing, but oddly still working, actually behind the scenes it is still running but I'm not very happy about it. Background: I moved the server. Oh and I Licenced it, it's a second machine and I was on trial mode. OK that shouldn't be too bad, but I also decided to move my SSD's from a fixed bay to a hot swap bay. That also isn't too bad, and they are still working so I don't think it's that. I've done a memory test, but not an extensive memory test. I've ruled out ZFS I believe and besides the file system is working fine. Symptoms I have seen include mostly USB related / looking challenges. I've reflashed and copied the config quite a few times now. Because it's a new USB I tried the old USB too. That's turned out with the same issue. I've used a config backup I made before this happened. I've downgraded from the beta to stable and it still happens. Check out these screenshots - weird text where the shouldn't be. Even getting told that VM's won't start because my hardware doesn't support VT-d, which it does and the VM's are running in the background. I'm sorry but I can't post diagnostics -see output below. I hope my hardware hasn't died - but would have thought that would show up in either disk or memory related errors, which it hasn't. And not that the docker and virtual machines are showing up as running (and are running fine when I connect from my laptop). Any ideas? I have to admit, I'm stumped. What am I missing?DiagnosticErrors.rtf

I managed to downgrade to beta 3 on 6.8.3. Same things happening so I guess it's co-incidence. Not really what I want to hear, but sadly co-incidence must mean I have bigger issues. No idea what though at this point.

@steini84 Are you sure something in the new build of ZFS beta4 isn't overwriting parts of the system it shouldn't? I've even got it complaining I've got less that 2G of memory in the GUI, yet in the console it's saying I have 96. Check out my main screen. I've done a memory test too just in case. Check out how docker and VM tabs are missing, but actually they're running. Weird scripting / coding language showing up in the GUI.

Upgrade to beta 4 has completely borked my system. That's my current technical assessment anyway. It took me ages to figure that out too. All sorts of errors about missing shared libraries, and also doing odd things like removing the docker and virtual machine menus from the Unraid GUI, also noticed it's telling me docker can't start because VT-d is missing - well it's a dual Xeon I don't think so. So far I don't recommend any of these betas and regret using them at all. I've tried to downgrade to beta 3 using the above method which is saying ZFS modules are not loaded so something is not right. This is all happening at a time when I have no time. But I need it to run. If I had the time I would reformat and downgrade to stable ZFS. ZFS beta HIGHLY NOT RECOMMENDED on UNRAID is my current opinion. The funny thing is that ZFS itself is running in the background - even when the menus are missing docker and vm's are running too (or were at least). It's also causing other issues like randomly not being able to boot from USB stick. My guess is that like the other assumption it's more of an issue to do with the implementation of ZFS as per how it's attached to unraid. But I don't know. Even the nerd pack is throwing up errors with this installed. It's possible it's something else I haven't thought of, but when I disable ZFS everything else starts to run normally again. Painful!

I was 50/50 because of the new FreeNAS Scale coming out with linux kernel, docker and Kubernetes, mainly because I want all the more business like features. But, Unraid have done such a fantastic job and apparently (hopefully) ZFS is coming along with who knows what else. They do a really good job on the whole don't they, so I don't mind giving a bit of extra cash at all. I'm lucky to be recently employed again and I can share that around a little here and there Thanks @limetech for a great product, community and support!

From what I've read, this xattr=sa setting, extensively reduces the array from doing writes to files on disk, instead writing to inodes. If this is always the way it has worked (and I believe it is), changing this is only like to be a workaround of some other thing, impacted by lots of writes such as when creating a new VM image. I had thought this was probably kernel related, but it doesn't happen on 6.8.3 and 6.9 if I recall. Either way, I'm not convinced this is the end of it.

I'm really worried it's ZFS causing all this - I just don't understand why creating a VM would trigger it. I have to say I wish I'd not accepted the new ZFS version and upgraded the pools - it will be challenging to sort out. I think it's enough to upgrade an OS, but keep the filesystem stable @steini84 from the attached syslog, would you agree it's ZFS? If so, I might need some help to log a ticket upstream i.e. around how you've packaged it. Started a ticket here - I'm sure it's ZFS now. I would appreciate if you could take a look and add any commentary - I'm concerned they'll complain about it being unraid and about it being a beta of unraid. Thanks. obi-wan-diagnostics-20201013-1953.zip

Thanks, so it's not specific to ubuntu and not specific to CPU brand. Both my systems are SMP, I assume that's not going to be an issue though.