Wody

I'm not sure if I'm reading things correctly, but if I look at the Idle stats in Powertop, it lists the Pkg as spending a lot of time in C6 and none in C3, and the Core also a lot of time in C6 and none in C3 on the server with the 9600. For the 9500, that server is a lot more busy, so it only reaches C3 in Pkg and none in C6, and the cores run some time both in C3 and C6.

It is plug and play, you just need to connect a cable from the HBA to it, they usually do have firmware, but its more like a switch-device, it just passes information.

Looks good. According to the ad, the 9207 is already running in IT mode and with 20.00.07 firmware which was the last one for that card, so you don't have to do anything for it. But, if you want to, you can of course re-flash the firmware (and bios) yourself. The cables seem okay too, although they don't say forward breakout, the description uses the right words (from 9207). The case comes with some fans, but not all of them that fit, it will be fine with what you have, but when you get more disks, you'll want to add more fans too.

The price is high for it, but that card works great for SATA drives, and with those SFF-8087 connections you'd use what is called a forward breakout cable, that has a SFF-8087 at the end of the card, but 'breaks out' into 4 SATA connectors to connect to drives. (They also make reverse breakout cables where the sata cables connect to the motherboard and the SFF-8087 is at the drive end for servers, because some wires are switched. If you hook such a cable up the other way, no drives would be detected) Of course if you have a drive-cage that uses SFF-8087 then you could use that too. So with that card, you would be able to hook up 8 drives, and when you need more, and get an expander, move one cable to the expander, connect the expander to the HBA, and connect the rest of the drives to the expander. Or get another card, whatever is more convenient at the time.

In your parts list, you have 3 drives, two 16TB ones, and a free 18TB one. Since your parity drive has to be at least as large, or larger than your data drives, that means you'd have to use the 18TB as parity drive, and the other 2 as data drives, meaning you can store 32TB. Since it has 4 SATA slots, I assumed you could use them all, and add another 18TB later as a data drive, but I didn't read the motherboard manual so I wasn't aware that you can add only 3 (which is a very small amount). Anyway, it really depends on how much storage you need. If you're pretty sure that the space you have is enough, or you'll replace disks when needed (since they make them up to 24TB now), you'll probably want to stick with the smaller case and the SATA or SAS card. If you think you'll have enough with about 10 drives or so, get the bigger case, but if you need a lot more, and have space for shelves somewhere, either case for how much you need now, but skip SATA. Alternatively, you could choose a board with 8 SATA slots, but an older chipset like the Z690 for around the same price. I have some define XL cases (or R2), so if you go for big, I'd also recommend one of those stands with wheels, so it is a lot easier to work with if you need to add drives and pull it out because of the weight. I also went with a HBA/SAS (but used cards because cheap), because it has 4 ports per cable, so it is much less likely that you pull some other cable if you are working with the drives. Also you can then buy used SAS drives which some companies sell pretty cheap just because they are a few years old. And you wouldn't even have to worry about internal or external ports much because you can easily add expanders or more cards, I got an Adaptec aec-82885t which I got for about the same price as your SATA card (it is cheaper now) but lets me hook up 24 internal drives, and 2 disk shelves. (and you can stick it in a PCIe slot for power, or just stick it anywhere in the case and use a Molex, the data goes with a SAS cable, and for the external ones, the cable could be up to 10 meter/33feet)

Both boards you picked have 4 SATA connectors, so with your configuration, you currently have 32TB of storage, and can add another drive with 18TB when you need it without having to add anything else. Assuming you don't want to replace drives, you could add a sata card, but an HBA will probably be a lot cheaper, since you can often buy older used models for much cheaper than sata cards, and with less problems and more speed (sata cards often have port multipliers which limit the speed or have very buggy firmware causing weird errors). If you do want to buy a 'drive shelf', mostly it depends on how you connect it. If it has sata or esata, then you'd need to connect to a sata card, but if it is a SAS shelf, it would have something like SFF-8088 and you would need an HBA to communicate with it, and you could connect your other drives to it too, with a forward breakout cable. Unraid won't care how you connect drives though, as long as it can see them.

I had a look at lspci, and my 9600-24i reports ASPM level 1 only with a latency of both L0s and L1 as unlimited, my 9500-16i reports both L0s and L1, with latency for L0s unlimited and for L1 < 64us. Since they are different, i figured I'd mentioned both. Also each card is in a different server, both running Unraid though, and the latest firmware at the moment (Phase 30 for the 9500, Phase 8.8 for the 9600)

I don't know about C-states since I don't really pay attention to that, but its important to note that the 9600 drops support for anything below 6Gb/s, meaning if you have any SATA 1 or 2, or SAS 1 drives, they won't work at all, and simply won't be detected. Also, the 9500 and 9600 no longer have a legacy BIOS, only UEFI, so even though it isn't important for HBA's, there is no information on screen during booting like with older cards, and no key to press to enter their BIOS. Instead, they only show up in the mainboard's BIOS Setup screen (I'm not sure what it is called for UEFI). Sometimes they don't show up there at all though, the 9600 isn't visible for me. But they work fine with Unraid and drive-detection is fine too.

Since heat has to do with how much power something uses, I did some math. The 9300-8i uses 13w, so 1,6w per port (rounded). They made a more efficient version of it, the 9305-16i, which uses 1,01w pet port. The 9400-8i uses 1,25w per port, which seems a lot more, but the 9400-16i only uses 0,75w. The 9500-8i also uses 0,75w, but the 9500-16i uses 0,56w. The 9600-16i uses more again, 1,06w but that's a sas4 (22.5G) card (also drops support for anything below 6Gb/s so old sata 1 & 2 drives won't work. Sata 3 drives will though) In my experience, you'll need some airflow, but if your drives aren't active 24/7, and in server cases with a fan-wall, you don't really need any extra fans or such, not even with a 9300-8i. (I'm currently using a 9500-16i and a 9600-24i but I also have a 9300-8i I use with a SAS expander and a 9400-16i) So, you'll probably want a 9400 or a 9500 card. There is an issue with the 9500 and 9600 though, they use 'new' sff-8654 (slimsas 8i) connectors, and cables for that to sff-8643 work fine (if you have HDD cables, they also have PCIe cables which won't work), but as far as I can tell, all cables to sff-8087 are wired for PCIe even if they specify they are for HDDs (at least the ones I tried).

When sas3 was new, there was an issue with drives not spinning up due to a pin being redefined, but your drives are sas3, so that isn't the issue. Even if the drive was dead, as long as the electronics work, they still power up and get recognized, so that does suggest dead drives. Some controllers aren't compatible with certain drives, but I don't know about those. If you have any other sata or sas drives laying around, I would try those, to see if they are recognized, or get a SFF8643 to SFF8482 breakout cable, that way you'll be able to try any drive individually and see if they spin up, if they do, your backplane may not be compatible, or maybe your drives just aren't making contact properly. (You could also try to insert a drive in the backplane without the sled, and some cardboard under it for support, and see if they work that way).

From your diagnostics, it looks like Unraid finds and uses the controller right, there are lines in the system logs about MPT3SAS being loaded, and starting the 3008 chip (which is your Broadcom 9300-16i), but the controller doesn't detect any drives. You do seem to have one of the latest firmwares (16.00.10, there is a 16.00.12 on the internet) but it doesn't show or list if the BIOS is installed. If the BIOS is installed on your controller, there would be information displayed during booting about the card, and it looking for drives, and a list of the drives detected, together with a message about pressing something like CTRL+R or CTRL+C to enter the BIOS. If the BIOS is not installed, check your firmware package, there should be some files called mpt3sas.rom and mpt3x64.rom which you can install with sas3flash -b (instead of sas3flash -f you use with firmware) The output from the lsscsi command-line command shows the drives attached, and it lists your Samsung 850 and 860, so what I would do, is connect those (temporarily) to the SAS-controller, and see if they are detected that way too. If they aren't, you likely have cable-issues, or backplane-issues (if you use a server-case). If you haven't used the controller and cables before (so you are not sure if they are the problem), and you are using SFF8643 to SAS/SATA breakout cables, you may have reverse breakout cables, which explains why nothing is detected, you need forward breakout cables.

IT/IR mode hasn't been a thing since the 9400. The 9400 has two bios to choose from (legacy and UEFI), and two firmwares, one for SAS/SATA/PCIe support, and one without PCIe support (which means it has room to buffer more commands at once). The 9500 has one bios, and one firmware, and the 9600 only has one file/package to install (which has the one firmware and one bios). You also can't use sas2flash, sas3flash etc, but have to use storcli or storcli2 to flash them. (The reason why they don't have IR/IT mode anymore, is because they added PCIe support, so they had to add that anyway, and they decided to combine everything and do it properly this time. For the HBA-cards that doesn't matter, but for the RAID cards it means you can select 'personalities' to make the card behave differently. How well that works? No idea, I only have HBA cards up to 9500 with a 9600 on the way.) Oh, and if you're going to use PCIe, make sure you have the right cables, there are still different ones for PCIe and SAS/SATA I believe. Although they standardized so you may be able to get away with cheaper generic cables (the 9500 and 9600 have pinouts in the manual. but I haven't studied them to see if the cables are the same pinout or not).

Yes, in the folder config/plugins. Yes, you'd stop the array (if started) and select the new drive as a parity-drive. After starting the array, unraid will tell you it will rebuild the parity-drive, which means reading all the disks and calculating and writing the proper parity. If you want to move data to the cache, you would have to do that manually, and set up the mover to ignore your data. Also, data on cache drives is not protected, so you would lose it if something happened to the cache-drive.

I updated from RC3 to RC4 then told unRAID to reboot, and went to do something else. 10 minutes later, I found the system hung, with no way to access diagnostics. After rebooting manually, everything came back fine, so I guess that is not a real problem. However, since I never saw this before, I thought I'd mention it. Something else though, now unRAID complains ' System notifications are disabled. Click here to change notification settings. '. So, I clicked here, and got to settings, where the help told me ' By default the notifications system is disabled. Enable it here to start receiving notifications.' Why am I getting notifications about a notification system that is disabled as default when it is disabled and so I shouldn't be getting notifications? In other words, why does unRAID complain about its own default settings? tower-diagnostics-20180408-0602.zip

As explanation, the 169.254.something address means that it can't find a network, so it could be a loose cable or something. So if you get network-issues again, try restarting things and re-attaching cables. Glad it is working for you now, and hopefully things will be boring from now on where everything just works, like how its meant.

Glad to hear the USB-thing got resolved. In your previous diagnostics, you were using DHCP and getting an address. Try something like ping google.com and if that doesn't work, try to ping googles nummeric IP-address instead. Since you can reach the server, even if it is just by IP-address, it could be something in the network is confused, so restarting the gateway/router may help. I've seen the not-able-to-reach-by name on my own server with older unRAID versions and older windows, so I've set a static IP-address on my unRAID server and normally connect to it by that, but I noticed with unRAID 6.5 and windows 10 I can use the Tower-name as well. In the past, when I had issues like this, I've also turned off all my devices and computers on my network, and then started first the router, then unraid and then computers to make sure everything started fresh and didn't have old data or something.

If you copy the config-folder with your key, then reformat and reload, and copy the config folder back might work. You'll want to at least save your key, since you might not get another for that USB (I'm not sure). And yes, 3.0 ports or drives can cause issues, so it is possible. So, do try a 2.0 drive and port.

Darn, still same errors. If you're sure you are using an USB 2 port now, what filesystem did you format the Cruzer USB with? On second thought, I also found another error related to USB: Mar 25 20:24:48 Tower kernel: ACPI Error: [\_SB_.PCI0.XHC_.RHUB.HS11] Namespace lookup failure, AE_NOT_FOUND (20170728/dswload-210) From googling, this seems an error in the BIOS, which is fixed by a version 11.1 of ACPICA, I have no idea what is currently used. There is a kernel bug report but I have no idea if it is included already or not. Maybe @limetech can tell us. Also, you are currently using BIOS 2.0b, and since you're having issues, try flashing to BIOS revision 2.0c which is available at the Supermicro site (on the right, Update your BIOS). I have no idea what they fixed in that one, but since you're having issues, it shouldn't hurt. I'm hoping that will fix everything.

Glad that part works now! Any messages? Maybe some new diagnostics would help, to see if the log-file has anything.

Since you have seem to have USB issues, unRAID can't verify your key and doesn't mount or start anything. The relevant section of the logfile is here: Mar 25 14:11:14 Tower emhttpd: shcmd (9): modprobe md-mod super=/boot/config/super.dat Mar 25 14:11:14 Tower kernel: md: unRAID driver 2.9.3 installed Mar 25 14:11:14 Tower kernel: read_file: error 2 opening /boot/config/super.dat Mar 25 14:11:14 Tower kernel: md: could not read superblock from /boot/config/super.dat Mar 25 14:11:14 Tower kernel: md: initializing superblock Mar 25 14:11:14 Tower emhttpd: error: get_key_info, 579: No such file or directory (2): fread: /boot/config/Trial.key Mar 25 14:11:14 Tower emhttpd: Unregistered - unregistered (ENOKEYFILE) So, something happens between 14:11:03 (when it mounts the drive) and 14:11:14 when it can't read it anymore. Which means, try a different port, if that doesn't work, verify the file system, or try to copy the config folder to a different computer, format the drive, use the USB-creator and copy the config folder back.

Like the previous post, you do have key-errors. You can find that from the diagnostics, in the folder logs there is a file syslog.txt that has the system-log. At the end, there are a lot of errors regarding the key-file. At 14:11:16 it read it once properly, but many other times it failed. If I then look at the folder system, in the file lsusb.txt (which lists the USB devices) it seems that there is no name listed for the device 258a:001 but this USB list specifies it as a USB keyboard. Try connecting the unraid USB-key not to your keyboard, but in/on the computer itsself, and not on the USB 3.0 port, which can have issues too. edit: I may have read that wrong, as in you didn't plug it in the keyboard. But try a different port anyway, the one you are using seems to have speed-issues.

Sounds good! And looking at the specifications, I find out it uses the SAS2308 controller, which I happen to have in my server as well, so I can confirm myself that it does work with unRAID! One last safety thing, when you get it, don't connect drives to it, or only drives that are empty so you won't lose any data. Then, start up unRAID, and check the log. It should have a line saying (or something similar so search for LSISAS2308): mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05), BiosVersion(07.39.02.00) Currently, thats the latest one If it has FWversion 16 something it'll work fine (although you can of course also flash it), the older ones from the 20-series of firmware had some bugs, so you'll want to flash the latest firmware and bios to it, which you can download from the download-page on your link.

Both the 2008 and 2308 seem to have the same interfaces with regards to drives, so it is possible you won't notice any differences regarding speed although the 2308 seems faster and newer. As to issues, there won't be any, as long as you're using unRAID 5 or newer, since those track the serials of the drive. So it should be plug and play. Don't forget to flash the 2308's to IT-mode if they aren't on that already.

From what I can read in the manual, is that your chassis uses the standard SFF-8087 connections, which means almost any HBA can be connected to it. If it uses the SFF-8484, which looks like sata-cables then I'm not entirely sure, I found some supermicro-sff-8484 use non-standard cabling, so I hope that isn't the case. Anyway, almost any LSI/Avago/Broadcom HBA adapter should work, (lSI was bought by Avago and then Avago was bought by Broadcom). When you find one, google the name and HBA, and see what it says. Also, for HBA's there is IR and IT firmware. the IR is for raid, and IT is for Initiator-target, which is what you want. So, if your first result says its an HBA, then google the name and IT firmware, and if that also gives a result, it should work. The list is great, but it is far from incomplete, so some extra steps for checking may be required if you do get one from that list. For example, (some of) the supermicro-cards on that list drop drives for some people and are no longer recommended. TL;DR: 1) Make sure it is an LSI/Avago/Broadcom HBA 2) Make sure it has or you can flash IT-firmware to it 3) Make sure your backplane has SFF-8087 and you can find a cable that can connect to your motherboard and/or HBA and that.

unRAID/RAID is not a backup! So yes, you do need a backup. But there is no reason you'd lose any data just because unRAID isn't working, unlike regular RAID where drive-failure or lost controller can mean you lose all of it. So don't worry, only if you have multiple-drive failure, you'd lose the data on those drives. Anything else can be replaced without any loss (but like I said, do make a backup of your unraid USB in case you need the configuration, and having backups of data is the right thing to do as well).