OrdinaryButt

I just got the same problem very recently. My server never sleeps and I don't think it is network related, here is what I changed: I had 1 single NVMe for cache, it was btrfs Added 2x SATA NVME in ZFS mirror for apps/vm cache Added a second NVMe and made a ZFS mirror for storage cache Shortly after adding them I noticed write speeds to cache down to 20mb/s, but writing to array was above 70 and reading was above 100, so it isn't network. T/s a bit to no avail. Rebooted the server - writes to either cache jump back to 110 saturating 1G NIC. Next day, writes to cache were again down to 20mb/s. I am suspecting something with ZFS and/or the mirrors but I don't know what. 6.12.8

Make sure speed stepping is enabled. If you execute this command, you should see individual core frequencies fluctuating. Mine goes down to as low as 800mhz watch grep \"cpu MHz\" /proc/cpuinfo

Dang. ASPM enabled everywhere. Your previous screenshots show high CPU usage to me, or missconfiguration. Your CPU is not going into lower (higher number) C states EDIT - only first core. I am not an expert, but I would start by looking into BIOS settings. Make sure speed step is enabled and stepping are not restricted. I didn't look for your previous posts, but if this is some sort of OEM specialty motherboard, you may be restricted with options in the BIOS. Read to the very first post again and make sure you didn't miss something.

Show us a screenshot of this console command output: lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

My powertop auto tune doesn't survive reboot. Do I just add "powertop --auto-tune &>/dev/null" to go file?

So this seems still unanswered. I've got a container that has available updates. No where I can see what the current installed version is (same request, in case the updates breaks something, want to be able to roll back). Anyone?

That is exactly what I meant when I said "I am stuck at C3". Because the driver doesn't support ASPM, the system (not just the card) uses more power (presumably PCIe lanes). Any chance we can see from you a driver with ASPM enabled, even if it is not published, for testing only? I read somewhere that Realtek has recently updated Linux drivers, but haven't had a chance to dig deeper.

Let us know how it goes and which exact model. Thanks.

I have a ASM1166 Sata expansion with latest firmware from Silverstone. No issues with waking up arrays that I can see (a few other people confirmed supposedly the firmware indeed fixes that). I got the Ableconn version ($59), which seems identical copy to Startech ($66) However, after applying the power top auto, all hosts go GOOD (the card does advertise 38 hosts, probably cause it supports port replicators downstream). But after system reboot, they all go back to BAD. I only have 3 drives and I confirmed their host number is listed as bad. Am I missing something? FYI, I am able to achieve C8 package state with the asm1166 (if I unplug my realtek 8125 2.5g card).

Because ASPM has been disabled for the driver due to some reported issue. Perhaps you can add a comment here to keep the hopes up.

Adding 8125 (Trendnet) adds exactly 4 watts to the kill-a-watt measurement at the tap. The problem is, the card prevents Packaged C-states - with it, I am stuck at C3; with card removed, it goes to C8. 4 watts may not be alot, but put it another way, it is 25% of this system usage (no HDD connected yet). 13 vs 17 watt at idle. If you share how to enable ASPM in the driver (I might find it by the time your respond, still reading), I can test to meassure the actual difference with ASPM on vs off. Thanks

I can confirm that ODD do not count towards the license limit on 6.12. ODD will not show up under list of devices, nor will be discovered by Unassigned device (at least for SATA, didn' try USB). You can check if ODD is detected by sending following console command: lsscsi -g Also, in order to pass ODD to container, you will likely have to adjust permissions. This comment explains how to do it for beginners:

Thanks, this is still required in 6.12 and does work. What is/where is "go file" ? EDIT: Found it. For the inexperienced user: Terminal to server change directory to boot / config ('cd /boot/config') send 'ls' command to confirm 'go' is listed then edit the file, for example 'nano go' Add comment (begins with #) and then parameters as listed above. Exit and save with Ctrl+X Reboot server to test.

I felt like you might be lacking some understanding, so I will explain like I would my kids. Ignore if I felt you wrong. ASM1166 with PCIe x1 will have a max theoretical bandwidth (limited by PCIe) of 8gigaBIT/s or 1gigaBYTE/s (1000megaBYTE/s). 1000/6 and considering overhead, you will get a maximum speed of 150megaBYTE (this is the measurement unit you typically see when you do file transfers in Windows environment). Modern HDDs can go upwards of 250MB/s, so the x1 PCIe slot will be a bottleneck if you have all 6 drives read/write at max capacity at the same time (parity check for example). With SATA SSD, you will reach that limit with 2 units. If you would be getting the ASM1166, it better have x4 PCIe (that would be 1000MB/s x4). This will allow you to have up to 6 SATA SSD on that card with no bandwidth limitation at all (provided your motherboard x4 or larger PCIe is electrical x4 or larger; physical dimension of the slot doesn't always guarantee the speed)

I get that. Have you seen any tests showing that the ASM1164 is indeed faster than 1064? Why would they allow the 1164 to be also x1, if there is 1064 for that? Makes no sense.

Well, in the opening post you do say that you have seen 1164 with x1, and ASM says 1-, 2- (see screenshot) what ever it means. I am looking for actual data that shows - is 1164 faster than 1064 or any other differences. Their web pages shows minimal details as each is listed as 25mhz Crystal https://www.asmedia.com.tw/product/17fYQ85SPeqG8MT8/58dYQ8bxZ4UR9wG5 https://www.asmedia.com.tw/product/A58yQC9Sp5qg6TrF/58dYQ8bxZ4UR9wG5

Does anyone know which ASM supports ASPM? Having hard time finding concrete information except for ASM1166 (positive reports that it does). I only need a controller with 4 SATA, least amount of power draw. EDIT: on ASMedia site, they say all 1064/1164/1166 Support L0s/L1/L2.... This I guess answers my original questions. What I can't find is the difference between 1064 and 1164.

yes, that's the one I meant (forgot the link). I completely missed the fact that the m.2 heat sink/fan is separate from the main sink. In this case, shouldn't adding a M.2 to 5x Sata ports adapters (2) be decent option? The bottom ports can be populated with m.2 for Cache, and you got 10 sata ports from the top 2 m.2. It's enough for me. Everyone focuses on how AMD is more power efficient than intel, However, it seems like AMX AM5 chipsets are twice more power hungry than intel at idle/low CPU tasks (chiplets draw more power). My server is probably at near idle 20 hours a day, so the low load power is more important than full load. EDIT: Personally, the Minis forum approach with regular 20/24 pin power connector fits NAS server better - ATX power supply can be used for the entire machine. Many of the above mentioned boards have only 12/19v DC jack power input, which would require some funky stuff to do for HDD power.

Duh, forgot to add the link https://store.minisforum.com/products/minisforum-ar900i

Did you guys see this? Got exited with the 4x M.2 slots, then I realized 2 are on the bottom and 2 are covered with the heat sink. May be they can be "extended" with cables? A bit steep at $560 but the assumed to be cheaper version with 13650HX isn't available yet (price unknown).

Any updates on this topic?

I run into write permission issues while installing a grafana as part of compos bundle of containers. Was able to "resolve" by adding 'user:"0" ' (root) to compose file. I feel like this is a bad practice, but I don't know if I am actually compromising anything by doing that. Anyone knows if this is a potential week point and if there might a different way to solve it?

Can someone with E3C246D4U2-2L2T confirm that when M.2 is populated with PCIe SSD, all 8 SATA ports are available? The manual only states that when M.2 SATA SSD is populated, one of the SATA ports is disabled (shared with M.2 sata), but there is no chart to show that M.2 PCIe SSD + 8 SATA ports is supported (like the Gigabyte version has). Also, the manual of the smaller ASRock board, C246M WS, does say that even PCIe M.2 disabled one of the SATA ports. Do the 10G ports on the E3C246D4U2-2L2T support 2.5/5G speeds as well? Here is what Intel says for X550-AT2 NIC: Data Rate Per Port 10/5/2.5/1GbE (NBASE-T in Linux Only)

Thanks for the tips!

What if older version is removed from repo? I have trust issues with "the cloud"