Menaan

Hello all, I have some questions about ZFS Expansion. Not specific to unraid, although I want to do this on my unraid server and figured people here would be knowledgeable about this sort of stuff. I see that a way to add disks to existing zfs vdevs was added to OpenZFS back at the end of 2023. First think I've got to figure out is if the zfs I have on my unraid is updated enough to have that feature. I'm running ZFS Master to get ZFS support. I've had my NAS running for a while and quite honestly fell behind on updates because for one "if it aint broke don't fix it" lol, but also because I actually ran into a problem on another unraid box I was running after an update to 6.12 and I've just never gotten around to upgrading my main NAS since then cause I didn't want to run into the same issue lol (I know it has long since been fixed and there was a work around even then, I've just been slack). The next and more important question is just along the lines of have people been successfully using the zpool expansion? I've got 40 TB of data on my NAS and while I do have backups (a full backup of everything and 2 more offsite backups of critical data), I'd really rather not hose the system if I can avoid it. But I'm getting low on free space with 5x 16TB drives in RaidZ2 config, and only have 3 available slots to expand. So I can't do the old method of adding another vdev of equal size to make sure the data is balanced properly. I also don't really want to just add another raidz1 3 disk array because I want to keep my 2 disk parity. Also, for those that have experience or knowledge of this expansion, how fast is it usually? Can you use the array while it's expanding? Can you add more than one disk at a time? And one last question not specifically about the zfs expansion, is there any requirement to run standard (non zfs array) in unraid? When I built this nas I had a few extra 8 TB drives so I threw those in the other 3 slots and just used Unraids built in array with a single disk parity on those instead of zfs. I think I remember the reason being something to do with unraid not being able to use the zfs array for things like appdata storage and such? But I really can't remember for sure. I don't really use those drives otherwise though, I have it configured right now to use them for data ingesting and that is about it. But it's really not necessary for me to do that, so I'd be hoping I can give up that array for the full 3 slots, but if it came down to it I can leave one 8TB in there and at least get a bit more with just 2 more 16tb drives. In the past I've just built a new NAS with higher storage capacity when I got low on space lol, but I don't really want to spend that much money right now. So hoping I can do something to just expand the zfs array. Thanks in advance for any help / advice on the subject.

I think I can call this solved now. I've got through multiple reboots now and gone past the monitor going to sleep time period quite a few times now. I was able to get reboots to be reliable by disabling fast boot in the bios. So, I think as of right now I can call this resolved. Will still have to monitor the hardware for a bit because of the flakiness I was seeing, but maybe that was just caused by some bios corruption with all the memory swapping and such. Thanks for the help, if I do have further issues I'll report back.

Interesting, I didn't think I was running 6.12.x prior to the reflash yesterday, but maybe I had upgraded it at some point and forgot. I didn't really pay attention to the version before I reflashed it because I wasn't worried about the data or anything on this system as I was reformatting everything anyway. I applied the fix in this post and so far on the initial test, I've made it to 20 minutes and the monitor turned off like it usually does at the 15 minute mark, but the system is still operating perfectly normal! I'll still have to do some more testing to be sure this one isn't a fluke, and to see if there are hardware issues or not, cause of some of the other flakiness that I've observed during troubleshooting this. Like the VGA light staying on until I reseated the processor yesterday or for example this morning, when starting the system, a few times it posted fine, but the Unraid menu never loaded. Never came up and said that it couldn't find a boot device or anything like that, just sat at a black screen after the POST went away for at least 5 to 10 minutes before I power cycled it. Got to make sure I can get a clean power cycle on this machine, cause it's being repurposed as an offsite backup primarily with some server stuff running on it (why I wanted the additionally memory that started all of this) because the offsite location has better internet that my home system hehe. So I won't have physical access to it easily to make sure it reboots any time I need to reboot it, or there is an extended power outage or something. I'm also just really glad to know about this potential issue with 6.12.x now cause I was about to pull the trigger on upgrading my primary nas from 6.11.x to 6.12.3, and it is also running an 11400 =p I guess I need to get better about reading release notes hehe

No, that plugin is not installed. I did reflash the flash drive today with a fresh install of unraid, and the problem occurred without any additional plugins installed. I didn't even configure the array, just let it sit there for 15 minutes and it started. I just observed some additional behavior since I made the post a few minutes ago that makes me lean more toward some kind of hardware problem. I went through a series of reboots where the system didn't even post, and the VGA light remained lit on the motherboard indicating no graphics card. This makes me lean toward somehow something must've happened to cause the gpu inside the 11400 to start having problems? I dunno, I suppose it could be something on the motherboard in relation to the memory that is causing a false symptom... like in my first post i'm baffled and I'm grasping at straws lol. I just reseated the CPU in the socket just to see if that makes a difference, will be able to tell in 15 minutes I suppose =p Otherwise, unless someone else has ideas, I'm probably going to have to try ordering a new 11400 and see if swapping the cpu fixes it. Or maybe just jump up to a 13400 and replace the mobo too in order to be safe. =/ I dunno, hate just replacing parts when the system was working just fine a couple of days ago before I shut it down for the memory upgrade. It had been running for over 150 days when I shut it down.. who knows, maybe a cap popped somewhere and it was fine until it was powered down =/

Hello all, I've ran into a weird problem with an Unraid server of mine. This server has an i5-11400 CPU on an Asus B560M Plus Wifi motherboard, with 16 or 32 GB of DDR4 ram (more on that in a bit cause it seems to be involved in the problem). The unraid version started off with an older version but was reflashed with the most recent version today and the problem continued. Let me start off by saying this server and hardware was running problem free for over a year. I was repurposing the server a little and adding more memory to it when this problem started occurring. The problem seems to be when some type of suspend even occurs (it's exactly after 15 minutes of the server being turned on, no input on a connected keyboard, and the monitor goes into sleep mode at the time of the event). As soon as this happens, there are tons of different processes that start going to 100% load on the thread they are on. And there ends up being 10+ of them on different threads resulting in a completely maxed out CPU. This is some very odd behavior that I've never seen something quite like it in my 25 years working with computers. The system basically becomes unresponsive making troubleshooting and trying things to fix it very difficult. The oddest thing to me though is the fact that it only does this with 32 GB of memory installed and not with 16. If I just put one of the 16 GB sticks from the 32 GB kit in, it's fine. If I run 2 x 8 GB sticks, it's fine. But 2 x 16 GB and the problem occurs. At first I just assumed I had gotten some bad sticks of memory. So I used memtest86 and sure enough I got hundreds of errors with the 2 x 16 GB sticks. Thinking ok, I've identified the problem I went and exchanged the memory and tried a new set of 2 x 16 GB sticks. This time 0 errors during memtest. But as soon as I go into unraid after exactly 15 minutes same exact problem. I've tried enabling / disabling xmp, asus performance mode, and played around with a few other bios settings. Nothing seems to make a difference. I don't know if it's actually related to the VGA suspending, or if that is just a coincidence of timing with it suspending at the same time as something else. I'm frankly a little baffled and hoping someone else might have some ideas. I don't really think this is an Unraid problem, it really seems like some kind of obscure hardware conflict or something (oh I did update my bios and all to be sure that wasn't the problem). But still posting here just in case someone has seen something like this. Thanks

EDIT: For anyone else that may reference this thread, don't do what I did below Apparently the shfs is supposed to be running haha. Had to reboot my server cause of some docker and vm errors after killing that command. I acted a little hasty there not researching the command before killing it hehe. Everything sems to be fine after rebooting at least. Interesting, thank you for pointing that out. I looked and I did find an shfs command apparently stuck running doing something with writing a file to disk1, not sure exactly what it was doing. The files it had open have already moved to my main zfs storage pool, so I double checked their integrity really quick and they were fine, so I killed the shfs command. Now the parity check speed shot up to 100 MB/s. So I suspect it should get done a lot quicker. So, now I guess I'll have to figure out what that shfs command was actually trying to do, and see if there is either a way to not let it do it during a parity check or something... Not really sure what the solution is but it at least gives me a point to research more from.

I did post them, and subsequently removed them after they were viewed as I said I would. Here they are again if you specifically want to review them, but I will remove them again once they have been reviewed.

I'm aware of wireguard vpn and I do use it for some things. However I'm not really sure how that pertains to my parity check taking a really long time issue? If this is just about the ssh traffic, that isn't really the point of this support ticket. But, also not sure of the relevancy even then because does it really matter which port is open if there is going to be a port open for a direct ssh connection, or a vpn connection? They are both going to get picked up and hammered by script kiddies running port scanners. I'm not going to claim to be a networking expert, but I don't see any security advantage in my use case for connecting the ssh connection through a vpn vs connecting it directly using certificate based authentication only. VPN is going to use more system resources on both ends for the same end result. But, again, not the point of this thread, so that is a topic for another thread. I'd really like to get this parity check thing figured out. t looks like this most recent run is going to take even longer than 3 days this time which is just crazy for such little data to be on this pool to me. Currently at 2 days 7 hours and is only 61% done.

Here is the syslog with all ssh attempts removed. syslogstripped.txt

Yeah, port 22 is open for offsite backup pulling purposes using rsync over ssh (for various reasons I have to pull from the remote server not push from this server). I'm not too concerned with the script kiddies cause password login is disabled, requires a cert to connect. I've seen the attempts in the log but considering the rotated logs go back to the 14th before the parity check started, and it only effects the system.log I assume anything relevant to the parity check would still be in the logs. If you need me to I can easily strip all the failed connection lines out of the logs. I didn't realize that log was singularly important since all the files were needed.

/config/smb-extra.conf is one place where I saw them. In the read and write lists for permissions on the shares. I'm attaching a modified zip of the diagnostics with the information removed that I don't want to share that I don't think will be pertinent for review. I'm going to remove the file after it's been reviewed just in case I missed something somewhere.

VM / docker names are what caught my attention. I have some named in a way for organization purposes that even disclosing that VM name could be considered a violation of NDAs I have with some clients =/ I see those names in a few places, ps.txt, and in the xml and qemu folders. Not sure if they are in other logs that I just didn't spot them. I'm also not thrilled about user names showing up in places, but that at least doesn't violate any NDAs I have with clients lol.

I've pulled a diag since the parity check is currently running. Are there certain files it would be helpful for you to see? I'd rather not upload the entire zip as even though I had anonymize diagnostics selected, I glanced through and there is still some data that I'd consider sensitive in this zip.

Hello all, I wanted to see if anyone had any recommendations on how to deal with some extremely long parity checks I'm seeing on my unraid server. First off for my server, I've got 5 x 16 TB disks that I'm using ZFS plugins to run a raidz2 config on. These are not the issue. I have 3 x 8 TB disks in the default unraid pool with 1 set as a parity drive. I also have a 1 TB Sabrent NVME drive as a cache drive. This is the pool that I'm having issues with. I currently have parity checks to run once a week, and they have been getting slower and slower. To the point now where they are taking upwards of 3 days to run. And I have less that 600 GB of total data on this pool (I don't use this pool for any primary storage, only for VM storage and as a ingest pool for things going to my larger zfs pool).... So I'm a little baffled as to why the parity check would be taking so long. I'm also not sure why the Cache drive gets hammered so hard during a parity check? While the parity check is running I can see it's read and write counts going up just as fast as the pool disks. Again, this is showing my limited knowledge of how this process actually works, but I would think the Cache drive wouldn't be part of the parity check as it should only have data on it that has not yet been written out to the primary disks in the pool. At least I think that is how it works in Unraid? I don't think Unraid does anything with frequently accessed data being moved to the Cache drive does it? This system has a Core i5-11400 CPU, and during the parity check it averages around 25% load. 128 GB of ram, a lot of which is used up by ZFS, but also there are a couple of VMs running on the system. Nothing seems to be maxed out though. Right now, the parity check is running and the Main page is reporting it processing at 7 MB/s. I honestly don't know if that speed is normal, seems pretty slow to me though given that even the 8 TB drives are capable of > 100 MB/s read and write speeds. So, I'm just not sure where to go from here to diagnoses the slowness further. Or if this is normal expected speeds? If this is normal speeds I'm going to have to decrease the frequency of parity checks =/ Thanks

So, I managed to find some settings that seem to work. If anyone else sees this and has a similar problem, in the bios I had to enable the discrete thunderbolt option, and then in the settings for that I had to disable the windows 10 thunderbolt setting. I still have the 99-local.rules implemented so I don't know if that is effecting it as well or not, I may try removing that later and if I do I'll edit this post to update if it still works without that or not. But once I actually got it to show up, passing my 3070ti through to a vm was very easy. I just selected it in the VM settings, didn't have to provide the rom or anything like I've seen in some other pass thru walk throughs.