Ledgem

I know this is everyone's first step and you've probably done it, but since you didn't mention it in your post, did you run at least one full test with Memtest?

I also have this issue. I'm in Hawaii so figured that it's just a slow connection to some servers that don't have geographically closer mirrors or routes, or may be speed limits set by the host servers. Immich is a painful one, though, because it updates so frequently and it's basically a gigabyte of data each time.

For the second time in a row I have to write that I am embarrassed, because when I planned out my expansion cards I referenced the Asus website's product page. For the PCI-E slots they only list four, but after your post I went back to look and you're right that there's a fifth slot at the very top, x1 as you said. The website does make mention of this, but I had skipped down to the PCI-E area, and it's not listed there. So thank you for mentioning this and bringing it to my attention - now I have a PCI-E slot back! If you go the Comet route, let us know how it is. Since I use the IPMI card as my fan controller, as well, I probably won't get rid of it. But I like GL.iNet's stuff, and it looks like a nice solution for people without an IPMI card. I'd consider using it in a dedicated tower system I am building.

I'm slightly embarrassed to say that despite building my system around this board, I didn't look into vPro and was also unaware of it and what it was. After doing a bit of reading, it looks like it's sort of an IPMI alternative, but doesn't cover all IPMI features. IPMI allows for more monitoring features than vPro does, but to quote a random internet comment that I found, "if you just want to be able to reboot your server remotely, vPro is fine." Since vPro is based off of the processor, my concern with relying on it over IPMI would be reliance on the system. The IPMI card kind of stinks in taking up a PCI slot (other motherboards have IPMI functionality baked into the board, itself, although that may shoot the argument I'm about to make), but it's independent. The system can be off, and as long as it's powered, the IPMI card is on. The IPMI card even has its own ethernet port; in a way, it is like putting a computer inside of your computer, albeit one with limited functionality. It's nice to know that even if your main system freezes or somehow becomes unreachable, you can log in to the IPMI card to reboot, access the BIOS, and so on. Granted, while I logged in to the IPMI card heavily during the first two or three weeks with the server, when I was making heavy changes, it has now been months since I touched it. It's there in case of emergencies, and I paid the extra money for that, but I'm half considering if I'd take it out to put in a different expansion card, if I needed it.

For most of what we do the IOPS are more important than raw speed, and IOPS shouldn't be affected by which slot the NVME is in. So for your first question, A or B, I'm not sure that it really matters. What may matter more would be considerations for airflow, since the first slot has a pretty nice heat sink and good exposure, while the second slot does not come with its own heat sink and it's more covered by any PCI cards. For what it's worth, I added a passive heat sink to my NVME in that second slot, both NVMEs are in a mirrored pool, and their temperatures are about the same. For placement of the HBA card, I don't think there's much of a difference in performance. Here my consideration is again based around airflow. In my chassis, the final PCI slot is right near the chassis wall, and there are a lot of cables that I routed down that way. I originally had my HBA card in the third slot (counting up with #1 being closest to the CPU), but found the heat sink was painfully hot to the touch when powering down the system. I added a 20x10mm Noctua fan, which fit well but made me think I was just heating up the next card in line. I moved the HBA card to the first slot and changed the fan for a 20x20mm Noctua, then moved the 20x10mm fan to the heat sink of my SFP+ card (which is in slot #2 - again, has the second most space between the next slot). If not actively cooling the heat sinks then having the HBA card in slot 3 or 4 might result in better airflow from chassis fans, depending on your setup. Adding the fans to the heat sinks wasn't too difficult. For my HBA card (LSI 9200-8i I think) the grooves in the heat sink are not wide enough for the screw-in method I've seen others use, so I got some medium-length twist ties, looped them around the heat sink springs (two), and then looped them through two of the fan's screw holes. The fan is oriented to suck air up from the heat sink and blow it away, so it's not like the securing mechanisms are fighting air flow. For my SFP card (a dual-port Intel SFP+ card) the grooves in the heat sink were wide enough that you just put in a screw (I can't remember if I used a M3 or M4, nor what length) and just twist the screw as if you're screwing something in. The screw gently wedges its way into the groove and secures the fan nicely.

You know, I actually can't find anything specific on whether the W680 (and Intel's processors) support registered DIMMs or only unregistered for ECC. The Asus website does mention unbuffered... you're probably right not to experiment, sorry for that. The IPMI card does give some interesting data but the Asus IPMI interface isn't great. The IPMI data (including fan control) does not carry over to Unraid by default. Maybe there's a way to tweak it. I don't regret getting the IPMI card - it's been useful for me. I just wish it was built-in to the motherboard, as I think they did with the AMD version.

Nemix 4800 MHz ECC DDR5 RAM (link is to a 256 GB set of 4x64 GB sticks currently listed for $1,356, but the same page also lists other multiples of 64 GB sticks) OWC 5600 MHz ECC DDR5 RAM (Nemix also lists a 5600 MHz variant on a different search page, but it's the most expensive of the lot so I'm not linking it) EDIT: these links are for registered (buffered) DIMMs, but the W680 requires unregistered (unbuffered) DIMMs - my mistake, they're not going to be compatible.

Unraid lists this motherboard as being able to max out at 256 GB. I thought that was a glitch, since all of the documentation that I saw indicated 192 GB as the maximum, but when I looked it up a few weeks ago it seems that 256 GB is indeed the maximum. I'm not sure if that was enabled with a firmware update, or if that capacity wasn't available when Asus did the initial testing with this motherboard. I also have 128 GB, and funny enough, also have mismatched speeds (4800 MHz and 5600 MHz). In my case it's because I went with OWC (purchased from Amazon) and had to do an exchange. I initially bought 64 GB (2x32 GB) and then decided to just max out all RAM slots, buying a second pair of 32 GB sticks. My system wouldn't boot at all if a specific two of the RAM sticks were installed. I did an exchange through Amazon and the two sticks that they sent me as replacements were the 5600. It works fine, ECC seems to be registering, so we're good. Two of the RAM sticks are Samsung and two are Micron Technology. My IPMI card reports that they're operating at 4200 MHz... meh. I wanted to go with Kingston RAM, as that's what most people remarked upon, but the only ECC options I could easily find were the OWC or Nemix options. I've used Nemix ECC RAM in my Synology system without problems, but that was my only experience with them. As a primary Mac user, OWC sort of has legendary status and so I went with them, but given my experience (or if I were buying again to upgrade further), I'd probably go with Nemix unless OWC were significantly cheaper. 128 GB is probably more than I need. I have a ZFS pool and the arc cache typically uses 1/8 of RAM; I manually adjusted it to use 50% of the RAM, and there was a very noticeable improvement in performance. (Copying data from my Synology, with both systems running on 10 Gbps SFP+ connections, the speed barely went above 2.5 Gbps burst speeds with only 16 GB of cache, but with 64 GB I'd burst to close to 8 Gbps and the sustained rates were also higher.)

Thanks for that. I tried adjusting the BIOS setting you mentioned and it still doesn't seem to work. Maybe I'll need to go through it again... it's not critical, I suppose, since accessing the BIOS is the key feature I really needed, and I can also still set what mode Unraid loads into. It would be nice to have it working fully in the event of a bigger issue, and it's just puzzling as to why it's having an issue. Other than my IPMI KVM issue (where the IPMI card is optional, anyway), yes, I'd get it again. All of my issues have been with the IPMI card; the motherboard, itself, seems to work just fine. There aren't too many choices for ECC-supporting motherboards on the Intel side, and I think this is one of the better ones. If I were buying again, the only thing I might change would be getting an AMD processor instead of Intel. I went Intel because I wanted to have transcoding capabilities without having to buy a dedicated GPU. But I ended up buying a GPU anyway (Intel A310 - low profile, and quite cheap). There seems to be wider ECC support on the AMD side, and when I briefly looked it over, I think the IPMI functionality might be baked into the motherboard instead of requiring a card that takes up a PCI slot.

Not sure if you're still monitoring this thread, but I've been eying these drives too - any thoughts on it after the past six months? I have a chassis with 36 drive bays and quickly realized that I wouldn't know which drive in Unraid correlated with their physical location. Physical labeling of the drive caddies is possible but would be a bit of a pain (my handwriting isn't the greatest, and it would have to be updated every time I swapped a drive out). My solution was to create an Excel spreadsheet with the grid layout of chassis. I'd copy the drive identifier and put it into the corresponding cell as I added each drive in and it appeared in Unraid. I have another sheet in that file with the fan layout, so that I know which fan corresponds to which fan controller channel, just in case I need to adjust fan curves or if one fan begins to drop its RPMs and needs to be replaced. I'm not sure if your chassis or setup would be conducive to making a spreadsheet with the visual representation of your hard drives, but that would be an easy way to keep track of it, and to know which drive Unraid is referring to if replacement is needed for some reason.

I'll bring this one back up, rather than making a new post. This is about the KVM functionality of the IPMI card. I am running Unraid on an ASUS W680-ACE-IPMI with the included IPMI card. I also have an Intel Arc A310 card installed, if it makes a difference. When I use the virtual KVM I can see and access the BIOS, and I can even see and control the Unraid boot selection screen, but once I choose something that's it - I essentially lose signal. If I plug in a monitor then I can see what's going on (so maybe this isn't a perfect fit for this thread), but the virtual KVM sees nothing. I'm guessing this is an IPMI issue rather than purely an Unraid one, and I say that because when I choose Memtest I also lose signal. Some things I've tried include getting a "dummy" HDMI plug, and plugging it into both the motherboard's monitor output (for iGPU) and then booting with it plugged into the Arc A310. Neither made a difference. The BIOS is also set to have the "in band" be Linux, and not Windows. Firmware for both IPMI and motherboard are the latest as of this posting. A separate question: the IPMI is great for setting fan curves and controlling the fans, but I can't find a way for the IPMI to monitor hard drive temperatures - which is really how I'd like to set the fan speeds. I've just been using the CPU temperature for now, as heavy HDD activity usually does result in higher CPU usage as well... my HDDs are connected over a SAS connection with a HBA card, and Unraid sees the temperatures just fine, but I'm guessing the inability to see the temperatures might have to do with the HBA card? Although the IPMI card shows me that various sensors are disabled and I don't see that HDDs are listed among any of the sensors... I'll set higher fan speeds to account for the chance that a HDD might get too hot, but I'm wondering if anyone has a solution to this one.

Using the original SFP+ card, which now seems to be working fine after the system was powered down over night... I think I have a resolution, but still can't explain it. The solution has to do with assigning a static IP within Unraid, and not just at the router/switch level, and having only one device connected. My network switch was recognizing two unknown MAC addresses as attempting to take the same IP and work on the same port as the actual SFP+ MAC address. Every now and then the switch would throw an error about an IP collision, but for the most part it seemed content to have two or three devices with the same IP - for reasons I can't explain, because normally it would be constantly mentioning the collision. Using a lookup database, the MAC addresses supposedly belong to TP-Link. I don't have any TP-Link devices in my system. As I think I mentioned, Unraid was reporting its IP as being the IP that I made a static assignment to the SFP+ port on, but I couldn't reach it at that IP. I could only reach it from the IP I had assigned to the ethernet port (also set as static within Unraid), and if I unplugged the ethernet cable, the server became unreachable - even though it was showing in the switch as being actively connected. The fix that I found was to set the IP for the SFP+ port as static within Unraid, and then reboot the system, removing the ethernet cable during the reboot process. I've actually only once had success with Unraid dynamically taking an IP - not that an IP isn't assigned (I can see it within my router), but Unraid seems to expect that the IP is declared for it. (The very first time I started Unraid I couldn't access it - I had to follow instructions online to modify the configuration on the USB drive, to assign a static IP.) The very first time I booted with the SFP+ card then Unraid worked with the IP, but ever since then I had issues with the SFP+ card and connection until I set the IP as static within Unraid. (It was always set as fixed within my router.) Plugging in the ethernet cable seems to mess things up, as the connection favors the ethernet cable - even though the SFP+ port is listed as eth0, and even though Unraid reports the SFP+ connection's IP as its own on the main screen, and not the ethernet port's. I've verified again that all ports (two SFP+ ports, and two ethernet ports) are not set to bridge or bond. At this point, with just the SFP+ cable connected, the server has shown stability even after multiple reboots. Realistically I'm not going to use the ethernet port at the same time as the SFP+ connection, so I'm just glad to have it working. All the same, it's been a strange experience. For reference, my Unraid server is using an Asus W680-ACE-IPMI motherboard, with IPMI card connected to the network via ethernet. The SFP+ card is an Intel X520-DA2 (Lenevo-branded). It's all wired to a UniFI Pro Max PoE 48-port network switch, managed by a UniFi Dream Machine router, with a Firewalla transparently connected between the switch and the router. The Firewalla only deals with internet-bound traffic and probably has nothing to do with what was happening here, although as I write that, maybe I should have double-checked what was happening on the Firewalla side of things too.

Hi Jorge, I appreciate the reply! The SFP+ card's working port is set to eth0, with the ethernet port set to eth1. A second ethernet port on the motherboard and the other, non-working port on the SFP+ card are set to eth2 and eth3, respectively, and aren't plugged in. I was initially running them with "port down" but I also tried re-enabling the ports and it didn't seem to make a difference. I'm thinking something is likely going on at the hardware level. I'll replace the SFP+ card and report back (likely later this week). I say that because I kept tinkering after making this post and the rogue MAC address seems to keep showing up even when the ethernet port is plugged in, and I actually could not get any data to transfer over the SFP+ card due to an IP conflict occurring for the SFP+ connection. I can't explain why the connection worked the very first time I powered the system up with the SFP+ card installed, and it was stable enough to work for about 24 hours during which time multiple terabytes were transferred over it (confirmed by looking at eth0 in the dashboard and seeing peak transfer rates of 5 Gbps). The event that triggered this instability was my modifying the ZFS ARC size which necessitated a reboot - all of this happened after the reboot. A subsequent reboot didn't resolve it. Since the card is already suspect (one port not working), hopefully a replacement will resolve it.

I started setting up my Unraid server with the built-in ethernet on my system's motherboard. I later added an SFP+ card (Intel X-520 chipset) and connected it to my switch (a UniFi Pro Max PoE switch). Everything seemed to be going well - whereas file transfers with the ethernet maxed out close to the 2.5 Gbps limit of that port, after adding on the SFP+ card I was getting close to 5 Gbps. I gave the SFP+ connection a fixed IP on the switch side, leaving it set to automatic within Unraid, and figured that was that. However, after rebooting the system it seemed like it was favoring the ethernet port for file transfers again, so I unplugged the ethernet... and lost my connection to the server entirely, even though Unraid reported that its LAN IP was the fixed address assigned to the SFP+ connection. Oddly, UniFi reported that the Unraid server was still online and attached to the correct SFP+ port, but it also showed an unknown MAC address on that port. Plugging the ethernet cable back in gave me access to the server again, and if I forcefully the unknown MAC address through the UniFi Network application, the UniFi Network application would identify the correct Intel SFP+ card's MAC address for the Unraid server on the SFP+ port it is plugged into. I thought maybe it was a fluke, so I removed the ethernet cable and the same thing happened again. Bonding and bridging are both disabled in Unraid. The SFP+ card may be a little faulty - it's a two-port card that I just recently got off of eBay, but only one port seems to be working - but I believe I've already transferred terabytes over it. I'm already toying with contacting the seller to exchange the card; if we can attribute this purely to hardware failure then I definitely will, but the behavior just seems too odd for it be that. If logs would be helpful, please let me know how to generate them, and in what circumstances you'd like them generated (presumably, after the ethernet is disconnected - but I will need to reconnect it to regain control of the server).

I'm in the process of moving from Synology to a custom-built Unraid server, and I think I've finally made it to the funnest part: the software setup. After trying a default Unraid array and determining that the performance wasn't quite what I expected for my main storage share, I decided to go with a ZFS pool (ten HDDs set up as RaidZ2 with two vdevs). I have two 2 TB NVME drives that I had originally intended to use as the cache for the Unraid array and that the system set up as a mirrored pair (raid 1), but since I'll be using a ZFS pool primarily, I figured the drives would house my Docker apps and possibly come into play when I add on a standard Unraid array. This is where I wanted to check in with the community before I begin installing things. I never got on with Docker in Synology, and just installed applications directly; the application data went onto the main volume, and while I had NVME accelerators, Synology's operating system chose how to utilize it. I'd like to give Docker another go, and will probably automate backups of the applications sitting on the NVME. Since the NVME might not be heavily utilized as a cache, I may be better off splitting the two drives such that one is for applications and one is for cache. Docker applications I am looking at include Plex, Homebridge, Immich, and a game server or two (think Minecraft). I guess there are really three questions here: 1) How should I structure the NVME drives? (File system and relation to each other) 2) Is there anything that I should pay attention to when it comes to organizing where things are, or how data is laid out? 3) Is there anything you wish you knew before you started your own Unraid server - things you had to go back and redo, or things that would have made your life easier if you had done it from the start? I have tried to read through a few guides, but I noted that some guides seem to disagree with the way others do things. That said, if you feel that many of these questions would be answered by a particular guide, feel free to link it my way and I will go through it. Thank you in advance for any advice you have to offer!