Menaan

Hello all, I have some questions about ZFS Expansion. Not specific to unraid, although I want to do this on my unraid server and figured people here would be knowledgeable about this sort of stuff. I see that a way to add disks to existing zfs vdevs was added to OpenZFS back at the end of 2023. First think I've got to figure out is if the zfs I have on my unraid is updated enough to have that feature. I'm running ZFS Master to get ZFS support. I've had my NAS running for a while and quite honestly fell behind on updates because for one "if it aint broke don't fix it" lol, but also because I actually ran into a problem on another unraid box I was running after an update to 6.12 and I've just never gotten around to upgrading my main NAS since then cause I didn't want to run into the same issue lol (I know it has long since been fixed and there was a work around even then, I've just been slack). The next and more important question is just along the lines of have people been successfully using the zpool expansion? I've got 40 TB of data on my NAS and while I do have backups (a full backup of everything and 2 more offsite backups of critical data), I'd really rather not hose the system if I can avoid it. But I'm getting low on free space with 5x 16TB drives in RaidZ2 config, and only have 3 available slots to expand. So I can't do the old method of adding another vdev of equal size to make sure the data is balanced properly. I also don't really want to just add another raidz1 3 disk array because I want to keep my 2 disk parity. Also, for those that have experience or knowledge of this expansion, how fast is it usually? Can you use the array while it's expanding? Can you add more than one disk at a time? And one last question not specifically about the zfs expansion, is there any requirement to run standard (non zfs array) in unraid? When I built this nas I had a few extra 8 TB drives so I threw those in the other 3 slots and just used Unraids built in array with a single disk parity on those instead of zfs. I think I remember the reason being something to do with unraid not being able to use the zfs array for things like appdata storage and such? But I really can't remember for sure. I don't really use those drives otherwise though, I have it configured right now to use them for data ingesting and that is about it. But it's really not necessary for me to do that, so I'd be hoping I can give up that array for the full 3 slots, but if it came down to it I can leave one 8TB in there and at least get a bit more with just 2 more 16tb drives. In the past I've just built a new NAS with higher storage capacity when I got low on space lol, but I don't really want to spend that much money right now. So hoping I can do something to just expand the zfs array. Thanks in advance for any help / advice on the subject.

I think I can call this solved now. I've got through multiple reboots now and gone past the monitor going to sleep time period quite a few times now. I was able to get reboots to be reliable by disabling fast boot in the bios. So, I think as of right now I can call this resolved. Will still have to monitor the hardware for a bit because of the flakiness I was seeing, but maybe that was just caused by some bios corruption with all the memory swapping and such. Thanks for the help, if I do have further issues I'll report back.

Interesting, I didn't think I was running 6.12.x prior to the reflash yesterday, but maybe I had upgraded it at some point and forgot. I didn't really pay attention to the version before I reflashed it because I wasn't worried about the data or anything on this system as I was reformatting everything anyway. I applied the fix in this post and so far on the initial test, I've made it to 20 minutes and the monitor turned off like it usually does at the 15 minute mark, but the system is still operating perfectly normal! I'll still have to do some more testing to be sure this one isn't a fluke, and to see if there are hardware issues or not, cause of some of the other flakiness that I've observed during troubleshooting this. Like the VGA light staying on until I reseated the processor yesterday or for example this morning, when starting the system, a few times it posted fine, but the Unraid menu never loaded. Never came up and said that it couldn't find a boot device or anything like that, just sat at a black screen after the POST went away for at least 5 to 10 minutes before I power cycled it. Got to make sure I can get a clean power cycle on this machine, cause it's being repurposed as an offsite backup primarily with some server stuff running on it (why I wanted the additionally memory that started all of this) because the offsite location has better internet that my home system hehe. So I won't have physical access to it easily to make sure it reboots any time I need to reboot it, or there is an extended power outage or something. I'm also just really glad to know about this potential issue with 6.12.x now cause I was about to pull the trigger on upgrading my primary nas from 6.11.x to 6.12.3, and it is also running an 11400 =p I guess I need to get better about reading release notes hehe

No, that plugin is not installed. I did reflash the flash drive today with a fresh install of unraid, and the problem occurred without any additional plugins installed. I didn't even configure the array, just let it sit there for 15 minutes and it started. I just observed some additional behavior since I made the post a few minutes ago that makes me lean more toward some kind of hardware problem. I went through a series of reboots where the system didn't even post, and the VGA light remained lit on the motherboard indicating no graphics card. This makes me lean toward somehow something must've happened to cause the gpu inside the 11400 to start having problems? I dunno, I suppose it could be something on the motherboard in relation to the memory that is causing a false symptom... like in my first post i'm baffled and I'm grasping at straws lol. I just reseated the CPU in the socket just to see if that makes a difference, will be able to tell in 15 minutes I suppose =p Otherwise, unless someone else has ideas, I'm probably going to have to try ordering a new 11400 and see if swapping the cpu fixes it. Or maybe just jump up to a 13400 and replace the mobo too in order to be safe. =/ I dunno, hate just replacing parts when the system was working just fine a couple of days ago before I shut it down for the memory upgrade. It had been running for over 150 days when I shut it down.. who knows, maybe a cap popped somewhere and it was fine until it was powered down =/

Hello all, I've ran into a weird problem with an Unraid server of mine. This server has an i5-11400 CPU on an Asus B560M Plus Wifi motherboard, with 16 or 32 GB of DDR4 ram (more on that in a bit cause it seems to be involved in the problem). The unraid version started off with an older version but was reflashed with the most recent version today and the problem continued. Let me start off by saying this server and hardware was running problem free for over a year. I was repurposing the server a little and adding more memory to it when this problem started occurring. The problem seems to be when some type of suspend even occurs (it's exactly after 15 minutes of the server being turned on, no input on a connected keyboard, and the monitor goes into sleep mode at the time of the event). As soon as this happens, there are tons of different processes that start going to 100% load on the thread they are on. And there ends up being 10+ of them on different threads resulting in a completely maxed out CPU. This is some very odd behavior that I've never seen something quite like it in my 25 years working with computers. The system basically becomes unresponsive making troubleshooting and trying things to fix it very difficult. The oddest thing to me though is the fact that it only does this with 32 GB of memory installed and not with 16. If I just put one of the 16 GB sticks from the 32 GB kit in, it's fine. If I run 2 x 8 GB sticks, it's fine. But 2 x 16 GB and the problem occurs. At first I just assumed I had gotten some bad sticks of memory. So I used memtest86 and sure enough I got hundreds of errors with the 2 x 16 GB sticks. Thinking ok, I've identified the problem I went and exchanged the memory and tried a new set of 2 x 16 GB sticks. This time 0 errors during memtest. But as soon as I go into unraid after exactly 15 minutes same exact problem. I've tried enabling / disabling xmp, asus performance mode, and played around with a few other bios settings. Nothing seems to make a difference. I don't know if it's actually related to the VGA suspending, or if that is just a coincidence of timing with it suspending at the same time as something else. I'm frankly a little baffled and hoping someone else might have some ideas. I don't really think this is an Unraid problem, it really seems like some kind of obscure hardware conflict or something (oh I did update my bios and all to be sure that wasn't the problem). But still posting here just in case someone has seen something like this. Thanks

EDIT: For anyone else that may reference this thread, don't do what I did below Apparently the shfs is supposed to be running haha. Had to reboot my server cause of some docker and vm errors after killing that command. I acted a little hasty there not researching the command before killing it hehe. Everything sems to be fine after rebooting at least. Interesting, thank you for pointing that out. I looked and I did find an shfs command apparently stuck running doing something with writing a file to disk1, not sure exactly what it was doing. The files it had open have already moved to my main zfs storage pool, so I double checked their integrity really quick and they were fine, so I killed the shfs command. Now the parity check speed shot up to 100 MB/s. So I suspect it should get done a lot quicker. So, now I guess I'll have to figure out what that shfs command was actually trying to do, and see if there is either a way to not let it do it during a parity check or something... Not really sure what the solution is but it at least gives me a point to research more from.

I did post them, and subsequently removed them after they were viewed as I said I would. Here they are again if you specifically want to review them, but I will remove them again once they have been reviewed.

I'm aware of wireguard vpn and I do use it for some things. However I'm not really sure how that pertains to my parity check taking a really long time issue? If this is just about the ssh traffic, that isn't really the point of this support ticket. But, also not sure of the relevancy even then because does it really matter which port is open if there is going to be a port open for a direct ssh connection, or a vpn connection? They are both going to get picked up and hammered by script kiddies running port scanners. I'm not going to claim to be a networking expert, but I don't see any security advantage in my use case for connecting the ssh connection through a vpn vs connecting it directly using certificate based authentication only. VPN is going to use more system resources on both ends for the same end result. But, again, not the point of this thread, so that is a topic for another thread. I'd really like to get this parity check thing figured out. t looks like this most recent run is going to take even longer than 3 days this time which is just crazy for such little data to be on this pool to me. Currently at 2 days 7 hours and is only 61% done.

Here is the syslog with all ssh attempts removed. syslogstripped.txt

Yeah, port 22 is open for offsite backup pulling purposes using rsync over ssh (for various reasons I have to pull from the remote server not push from this server). I'm not too concerned with the script kiddies cause password login is disabled, requires a cert to connect. I've seen the attempts in the log but considering the rotated logs go back to the 14th before the parity check started, and it only effects the system.log I assume anything relevant to the parity check would still be in the logs. If you need me to I can easily strip all the failed connection lines out of the logs. I didn't realize that log was singularly important since all the files were needed.

/config/smb-extra.conf is one place where I saw them. In the read and write lists for permissions on the shares. I'm attaching a modified zip of the diagnostics with the information removed that I don't want to share that I don't think will be pertinent for review. I'm going to remove the file after it's been reviewed just in case I missed something somewhere.

VM / docker names are what caught my attention. I have some named in a way for organization purposes that even disclosing that VM name could be considered a violation of NDAs I have with some clients =/ I see those names in a few places, ps.txt, and in the xml and qemu folders. Not sure if they are in other logs that I just didn't spot them. I'm also not thrilled about user names showing up in places, but that at least doesn't violate any NDAs I have with clients lol.

I've pulled a diag since the parity check is currently running. Are there certain files it would be helpful for you to see? I'd rather not upload the entire zip as even though I had anonymize diagnostics selected, I glanced through and there is still some data that I'd consider sensitive in this zip.

Hello all, I wanted to see if anyone had any recommendations on how to deal with some extremely long parity checks I'm seeing on my unraid server. First off for my server, I've got 5 x 16 TB disks that I'm using ZFS plugins to run a raidz2 config on. These are not the issue. I have 3 x 8 TB disks in the default unraid pool with 1 set as a parity drive. I also have a 1 TB Sabrent NVME drive as a cache drive. This is the pool that I'm having issues with. I currently have parity checks to run once a week, and they have been getting slower and slower. To the point now where they are taking upwards of 3 days to run. And I have less that 600 GB of total data on this pool (I don't use this pool for any primary storage, only for VM storage and as a ingest pool for things going to my larger zfs pool).... So I'm a little baffled as to why the parity check would be taking so long. I'm also not sure why the Cache drive gets hammered so hard during a parity check? While the parity check is running I can see it's read and write counts going up just as fast as the pool disks. Again, this is showing my limited knowledge of how this process actually works, but I would think the Cache drive wouldn't be part of the parity check as it should only have data on it that has not yet been written out to the primary disks in the pool. At least I think that is how it works in Unraid? I don't think Unraid does anything with frequently accessed data being moved to the Cache drive does it? This system has a Core i5-11400 CPU, and during the parity check it averages around 25% load. 128 GB of ram, a lot of which is used up by ZFS, but also there are a couple of VMs running on the system. Nothing seems to be maxed out though. Right now, the parity check is running and the Main page is reporting it processing at 7 MB/s. I honestly don't know if that speed is normal, seems pretty slow to me though given that even the 8 TB drives are capable of > 100 MB/s read and write speeds. So, I'm just not sure where to go from here to diagnoses the slowness further. Or if this is normal expected speeds? If this is normal speeds I'm going to have to decrease the frequency of parity checks =/ Thanks

So, I managed to find some settings that seem to work. If anyone else sees this and has a similar problem, in the bios I had to enable the discrete thunderbolt option, and then in the settings for that I had to disable the windows 10 thunderbolt setting. I still have the 99-local.rules implemented so I don't know if that is effecting it as well or not, I may try removing that later and if I do I'll edit this post to update if it still works without that or not. But once I actually got it to show up, passing my 3070ti through to a vm was very easy. I just selected it in the VM settings, didn't have to provide the rom or anything like I've seen in some other pass thru walk throughs.

Hello, I'm attempting to get an egpu working through a thunderbolt 4 ex card on an asus motherboard with unraid. I've gotten this to work under ubuntu, but I can't seem to get unraid to work. If I look at my devices, the thunderbolt card is detected: IOMMU group 18: [8086:1136] 05:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 19: [8086:1136] 06:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 20: [8086:1136] 06:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 21: [8086:1136] 06:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 22: [8086:1136] 06:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 23: [8086:1137] 07:00.0 USB controller: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] IOMMU group 24: [8086:1138] 09:00.0 USB controller: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] Bus 005 Device 001 Port 5-0 ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001 Port 6-0 ID 1d6b:0003 Linux Foundation 3.0 root hub And I know that at least at a some level it knows that a Razor Core X is plugged into this card because if I go to /sys/bus/thunderbolt/devices/ the 0-1 device name is Core X. root@unraid-test:/sys/bus/thunderbolt/devices/0-1# cat device_name Core X But, no matter what I do I can't seem to get unraid to show the 3070 ti that is plugged in to show up anywhere. I've poked around and read a bunch of different things but can't find any solution to this. I've seen posts about thunderbolt security and I've scoured my bios looking for thunderbolt security settings and cannot find any. I found this article: https://www.kernel.org/doc/html/v4.13/admin-guide/thunderbolt.html with a method for trusting all devices using a 99-local.rule but even with implementing that rule that gpu doesn't show up. Also to note I've tried hot plugging the egpu as well as having it pluged in durring boot, neither make a difference. I did find a bios setting for discreet thunderbolt and I tried enabling that but that doesn't make a difference either. I'm testing on Unraid 6.11.5, all firmware is updated. Can anyone help me with what I need to do to get this gpu detected (my ultimate goal being to pass thru to a VM). Thanks

Anyone else have any information on this for me? Is the networking behavior as expected for unraid? I ran into another issue with it today and I'm trying to figure out if these are unraid issues, or there might be an issue with the 4 port NIC card I have or what. The issue I ran into today was after setting my 4 port bond back up with LACP configured. I could tell the unraid server was responding slow but I was trying to install plex. It kept erroring out with "upstream timed out" errors. After a lot of various troubleshooting and searching I figured out it was due to LACP. As soon as I disabled bonding on those 4 ports everything started responding much quicker and I was able to install plex without a problem. I know my switch is working correctly and has the LACP correctly configured (I have other devices that are working with LACP on this same switch as mentioned before with my old server). So that means these problems are either due to the NIC card that I got, or something to do with how unRAID is handling the LACP.

Yeah, if I type in the path, it goes red and won't let it apply as if it's an invalid value. The path entered does exist, triple checked that. EDIT: Ugh, nvm it apparently requires the trailing / to accept it. Thanks again.

Apparently I was wrong about changing the path for docker and vms, apparently those can only be changed to paths in the /mnt/user folder.... that is annoying. Is there some way around that? I want to be sure my dockers and vms are on redundant storage.

Ok, configuring a share in smb-extra seems to work for everything except the appdata, system, domains, and isos folders. But, I think I found where I can change the storage path for docker and vms, so that shouldn't be an issue. I don't see what you mean by export the pool via Unassigned Devices. I probably don't need to do this since the smb-extra.conf works, but I'm just curious what this is / where it is? Thanks!

So, let me start by saying I'm sure I'm doing something here that isn't "supported". That is fine, I'm not asking for it to be supported, I just want to see if anyone has any ideas of how I can get around it. I'm getting the error spamming my log file every second for "error: share_luks_status, 5995: Operation not supported (95): getxattr:". It does this for most of my shares. I fully expect the reason is because I'm using zfs and have symbolic links to the zfs datasets in place of the shares. I have a 1TB drive currently for the unraid array so that docker and vms will work (I'll be swapping this out for a smaller drive at some point probably, it's just a spare I had on hand). The zfs array is made up of 5 x 16 TB ironwolfs in a raidz2 configuration. I'm not sure why these sym links would not support the getxattr because as far as I can tell getxattr being called on a sym link should just call getxattr on the file or folder the link points to. And ZFS says it supports these xattr operations. But anyway, this is the way I'd prefer for things to be set up until official zfs support comes in unraid (which will hopefully make this a non issue). Is there some way for me to disable whatever is generating this error or at least just not log the error maybe? It doesn't seem to be anything important because everything seems to be fully functional. Alternatively, is there a way for me to create smb shares that aren't done through unraids /mnt/user? I know in some cases I could move the sym link to under the /mnt/user/<share> folder (although I find that configuration messy and annoying, guess something about having to use a single folder under a share just sets my ocd off lol)... but that doesn't work for all cases. For example from what I gather (I could be wrong since I'm new to unraid) VMs are stored in the domains share and dockers are stored in the system and appdata shares, I want these things on my redundant disk array not on the dummy 1tb drive. I'm even considering just having a task that runs occasionally and deletes any rotated syslog files to try and keep space free, but this doesn't really seem like a great solution either. It still leaves my syslog file spammed with errors should I need to check it for anything important.

Yes, I could do that, but I'd still like to understand why it does what it does as it is. Why doesn't it balance when using LACP? The old server does that just fine, if I look at it with multiple connections they are spread across 3 ports. And even when LACP is off, I don't understand how it can decide to route traffic through a port that isn't even configured. I'd like to understand these things so I can be sure the networking on unraid is going to be sufficient for my needs before I put this server into a real use case.

And another odd piece of this puzzle... Just trying different things out to see if I can maximize transferring data to this new server. I turned off all bonding. And have 2 transfers successfully going but again not as I would expect based on configuration settings. So currently I have eth0 set with ip 160 (yes there is more but just the last 3 digits for uniqueness), eth1 ip 161, eth2 ip 162. eth3 and eth4 are both not configured. I have one rsync stream running on the new server pulling data from the old server, and one running on the old server pushing data to the IP 161. And it's successfully maxing out 2 x 1 Gb ports. Now, I would think based on my configuration, one stream would be goign to eth1, and one to either eth0 or eth2 depending on which one the pull decided to use. But no... if I look at network traffic on the new server eth0 is running at 1Gb and eth2 is running at 1Gb. And a few minutes ago it was eth2 and eth3. Remember, eth3 not even being configured, yet it was still somehow deciding to use that. This just adds to my confusion on exactly what the heck the networking is even doing in this =/

Ugh, even more annoying now I've observed the server switching streams that were running on separate nics to the same nic mid stream... I had atm 3 rsyncs running, and they were all running on one of the 4 port bond1 ports (still no idea why it won't use more than one port). I had only been able to get them to even use bond1 by unplugging the 2.5 Gb onboard that is currently bond0 (even though it's not actually bonded to anything, I read one post somewhere saying that the stand alone eth0 had to be set to bond for the 802.3ad to work on the others). So anyway, I figured since the server isn't properly utilizing all the ports, I'd start a copy progress over another PC with a 2.5 Gb port to the 2.5 Gb port in the server. Even though I'd only be getting part of the traffic since that computer would have to both pull the data from the old server and send the data to the new server on the same single port in that computer. So, there it was maxing out a single 1 Gb port on the bond1 set, leaving 3 ports idle. And as soon as I start copying data from another device through the 2.5 Gb port, all the transfers jump over to the 2.5 Gb port... and since my 2.5gb switch feeds into my 1 gb switch through a single 1 gb port, it's limited to 1 Gb as well. Ugh... this networking behavior just does not feel right and I have no idea why it's behaving this way or how to fix it. It should not be leaving 3 ports idle in a LAG setup with multiple connections, much less the now 4 ports idle. *sigh*

I've recently built a new NAS server and am testing out unRAID on it. The server has an onboard 2.5Gb NIC, and I added a 4 x 1 Gb PCIx expansion card. The card supports LAG 802.3. My old server also has 4 x 1 Gb ports configured with LAG. I know the LAG configuration on the old server is working because I can initiate transfers on multiple clients and max out multiple 1 Gb ports. The unraid server however is not behaving as I'd expect. I fired off multiple rsync jobs (I know each one will be limited to a single 1 Gb port) but, rather than using multiple ports, they all loaded up onto eth1 and shared the bandwidth of it. I'm not sure why this is? I have confirmed in my switch that the LACP is recognized on the 4 ports, so that part seems to be configured correctly. I then tried just leaving each port individual and assigning them all different static IPs. This seemed to work for 2 rsyncs once, but then the 3rd still went onto the same ethernet port as one of the 2 already running rsyncs. Can someone help me understand what is going on here? I should be able to get at least 3 x 1 Gb streams going on this thing (I know on the old server the 4th port is used for failover so I won't get 4). But I can't figure a reason for why it picks which port it does and when LACP is on, why it doesn't use more than one port. EDIT: Just to add to this, I know the nic grouping isn't an issue on the old server. If I do just 2 x Rsync processes and look at the NICs on the old server, it's using 2 different NICs (just limited to around half the bandwidth each since the new server is only using 1 NIC for some reason).