May 11, 20251 yr If with the new version of Unraid that comes with zfs 2.3.x and you have upgraded your pools you are having a lower ARC consumption, specially if you have NVMe or fast drives, SSD, this might be the setting that it's causing it. Just for you to know, so you don't waste more time troubleshooting. https://discourse.practicalzfs.com/t/openzfs-2-3-0-release-direct-i-o-question-also-i-dont-understand-how-distros-package-zfs-apparently/2159 https://openzfs.github.io/openzfs-docs/man/v2.3/7/zfsprops.7.html#direct The question could be, even if it's less performant could be interesting to disable it just to not to wear the nvme with readings? It's better in any nvme zpool setup? or it has its use cases? it's worth to enable it with 2 NVMe with zfs mirror? Edited May 11, 20251 yr by L0rdRaiden
May 11, 20251 yr Community Expert 3 minutes ago, L0rdRaiden said: to not to wear the nvme with readings? reads are free on flash based storage, writing is wearing it.
May 11, 20251 yr Community Expert Hmm, still appears to be working as before for me, and this was after only using upgraded NVMe pools:
May 11, 20251 yr Author 7 minutes ago, JorgeB said: Hmm, still appears to be working as before for me, and this was after only using upgraded NVMe pools: How much used storage do you have in your nvme pools?
May 11, 20251 yr Community Expert I have another NVMe pool on this server but had only been using this one since the reboot.
May 16, 20251 yr I am also seeing low arc usage in ram. currently at 5GB even though I set it to 24GB with zfs_arc_max. It is likely caused by this https://openzfs.github.io/openzfs-docs/man/v2.3/7/zfsprops.7.html#direct However the parameter zfs_arc_direct is not found in /sys/module/zfs/parameters/ so I can't test it. zfs_arc_direct was introduced in OpenZFS 2.2 and is not available unless the ZFS kernel module was compiled with it I'm running ZFS 2.3.1 userland, but the kernel module was likely built without full 2.2+ options. Could we get the new parameter in the kernel module to test?
May 16, 20251 yr Community Expert You should be able to disable Direct IO for a pool/dataset by using: zfs set direct=disabled <pool name>/<dataset> After that, everything show go through the ARC like before, and you can confirm if any Direct IO is being made by looking at the output of: cat /proc/spl/kstat/zfs/<pool name>/iostats
May 16, 20251 yr Author I haven't changed direct setting yet but this are my stats after 7 days of uptime. The services pool is made by nvme. Still the direct value is too low to justify that the arc is still in 6GB after a week, when usually it was full at 21GB Maybe L2ARC has something to do? --- Estadísticas del ARC (Cache en RAM) --- Hits (Lecturas desde ARC): 453976458 Misses (Lecturas que no estaban en ARC): 1801262 Hit Ratio del ARC Global: 99% ----------------------------------------- Tamaño actual del ARC: 6.41 GiB Límite mínimo configurado (zfs_arc_min): 953 MiB Límite máximo configurado (zfs_arc_max): 22.35 GiB --- Estadísticas del L2ARC (Cache en SSD/NVMe) --- Hits (Lecturas desde L2ARC): 90766 Misses (Lecturas que no estaban en L2ARC): 60691 Hit Ratio del L2ARC: 59% ----------------------------------------- Espacio en dispositivos L2ARC actualmente usado: 71.65 GiB Espacio en dispositivos L2ARC disponible para cachear: B --- Estadísticas de Lecturas/Escrituras desde ARC y acceso directo por Pool --- Pool: data ARC Read Bytes: 143.31 GiB ARC Write Bytes: 52.35 GiB Direct Read Bytes: 0 B Direct Write Bytes: 0 B Pool: services ARC Read Bytes: 71.20 GiB ARC Write Bytes: 215.34 GiB Direct Read Bytes: 2.04 GiB Direct Write Bytes: 21 MiB The script #!/bin/bash # Script para obtener estadísticas clave de ZFS ARC, L2ARC, acceso directo y compresión. # Compatible con múltiples pools y datasets. # --- Configuración --- # Lista de pools ZFS para extraer estadísticas de /proc/spl/kstat/zfs/<pool name>/iostats POOL_NAMES=("data" "services") # Lista de datasets para verificar la compresión. DATASETS_TO_CHECK_COMPRESSION=( "data/personal" "services/docker" "services/vm" ) # --- Funciones de utilidad --- format_bytes_smart() { local bytes=$1 local gib_threshold=$((1024 * 1024 * 1024)) local mib_threshold=$((1024 * 1024)) local kib_threshold=$((1024)) if (( bytes < kib_threshold )); then echo "${bytes} B" elif (( bytes < mib_threshold )); then echo "$((bytes / 1024)) KiB" elif (( bytes < gib_threshold )); then echo "$((bytes / 1024 / 1024)) MiB" else echo "$bytes" | awk '{printf "%.2f GiB", $1 / 1024 / 1024 / 1024}' fi } # --- Verificación de requisitos --- if [ ! -f /proc/spl/kstat/zfs/arcstats ]; then echo "Error: No se encontró /proc/spl/kstat/zfs/arcstats." echo "Asegúrate de que el módulo ZFS está cargado." exit 1 fi # --- Obtener estadísticas del ARC y L2ARC --- arcstats_output=$(cat /proc/spl/kstat/zfs/arcstats) hits=$(echo "$arcstats_output" | awk '/^hits / {print $NF}') misses=$(echo "$arcstats_output" | awk '/^misses / {print $NF}') arc_current_size=$(echo "$arcstats_output" | awk '/^size / {print $NF}') l2_hits=$(echo "$arcstats_output" | awk '/^l2_hits / {print $NF}') l2_misses=$(echo "$arcstats_output" | awk '/^l2_misses / {print $NF}') l2_size=$(echo "$arcstats_output" | awk '/^l2_size / {print $NF}') l2_free=$(echo "$arcstats_output" | awk '/^l2_free / {print $NF}') if [ -z "$l2_size" ]; then l2arc_present=false l2_hits=0 l2_misses=0 l2_size=0 l2_free=0 else l2arc_present=true fi arc_min_limit_bytes=$(cat /sys/module/zfs/parameters/zfs_arc_min 2>/dev/null || echo "0") arc_max_limit_bytes=$(cat /sys/module/zfs/parameters/zfs_arc_max 2>/dev/null || echo "0") # --- Calcular Hit Ratios --- total_accesses=$((hits + misses)) arc_hit_ratio=$(( total_accesses > 0 ? hits * 100 / total_accesses : 0 )) l2_total_accesses=$((l2_hits + l2_misses)) l2arc_hit_ratio=$(( l2_total_accesses > 0 ? l2_hits * 100 / l2_total_accesses : 0 )) # --- Mostrar Resultados --- echo "--- Estadísticas de ZFS ARC/L2ARC y Compresión ---" echo "" echo "--- Estadísticas del ARC (Cache en RAM) ---" echo " Hits (Lecturas desde ARC): $hits" echo " Misses (Lecturas que no estaban en ARC): $misses" echo " Hit Ratio del ARC Global: ${arc_hit_ratio}%" echo " -----------------------------------------" echo " Tamaño actual del ARC: $(format_bytes_smart $arc_current_size)" echo " Límite mínimo configurado (zfs_arc_min): $(format_bytes_smart $arc_min_limit_bytes)" echo " Límite máximo configurado (zfs_arc_max): $(format_bytes_smart $arc_max_limit_bytes)" echo "" echo "--- Estadísticas del L2ARC (Cache en SSD/NVMe) ---" if [ "$l2arc_present" = true ]; then echo " Hits (Lecturas desde L2ARC): $l2_hits" echo " Misses (Lecturas que no estaban en L2ARC): $l2_misses" echo " Hit Ratio del L2ARC: ${l2arc_hit_ratio}%" echo " -----------------------------------------" echo " Espacio en dispositivos L2ARC actualmente usado: $(format_bytes_smart $l2_size)" echo " Espacio en dispositivos L2ARC disponible para cachear: $(format_bytes_smart $l2_free)" else echo " No se detectó L2ARC configurado." fi echo "" echo "--- Estadísticas de Lecturas/Escrituras desde ARC y acceso directo por Pool ---" for pool in "${POOL_NAMES[@]}"; do iostats_file="/proc/spl/kstat/zfs/$pool/iostats" if [ ! -f "$iostats_file" ]; then echo " Pool '$pool': No se encontró $iostats_file. ¿Es un nombre de pool válido?" continue fi arc_read_bytes=$(awk '$1 == "arc_read_bytes" {print $3}' "$iostats_file") arc_write_bytes=$(awk '$1 == "arc_write_bytes" {print $3}' "$iostats_file") direct_read_bytes=$(awk '$1 == "direct_read_bytes" {print $3}' "$iostats_file") direct_write_bytes=$(awk '$1 == "direct_write_bytes" {print $3}' "$iostats_file") echo " Pool: $pool" echo " ARC Read Bytes: $(format_bytes_smart $arc_read_bytes)" echo " ARC Write Bytes: $(format_bytes_smart $arc_write_bytes)" echo " Direct Read Bytes: $(format_bytes_smart $direct_read_bytes)" echo " Direct Write Bytes: $(format_bytes_smart $direct_write_bytes)" echo "" done echo "--- Estadísticas de Compresión ---" echo "Nota: Mostrando compresión para los datasets listados en el script." echo "" for dataset in "${DATASETS_TO_CHECK_COMPRESSION[@]}"; do echo " Dataset: $dataset" zfs get -H -o value compressratio "$dataset" 2>/dev/null | { read -r compressratio if [ -z "$compressratio" ]; then echo " Ratio de compresión: No encontrado o error al obtener." else echo " Ratio de compresión: $compressratio" fi } done echo "" echo "-------------------------------------------------" Edited May 16, 20251 yr by L0rdRaiden
May 16, 20251 yr Community Expert 9 minutes ago, L0rdRaiden said: Maybe L2ARC has something to do? Possibly, but you can easily disable direct IO and retest
May 16, 20251 yr 5 hours ago, JorgeB said: You should be able to disable Direct IO for a pool/dataset by using: zfs set direct=disabled <pool name>/<dataset> After that, everything show go through the ARC like before, and you can confirm if any Direct IO is being made by looking at the output of: cat /proc/spl/kstat/zfs/<pool name>/iostats I have not yet disabled direct IO yet with zfs set direct=disabled <pool name>/<dataset>. This is my iostats. Surprisingly, no direct IO everything is already going through arc cache. But ZFS cache is only at 5.4GB on the dashboard. root@Tower:~# cat /proc/spl/kstat/zfs/cache/iostats 79 1 0x01 26 7072 104056354597 33592417132851 name type data trim_extents_written 4 0 trim_bytes_written 4 0 trim_extents_skipped 4 0 trim_bytes_skipped 4 0 trim_extents_failed 4 0 trim_bytes_failed 4 0 autotrim_extents_written 4 6815968 autotrim_bytes_written 4 2272647577600 autotrim_extents_skipped 4 4774296 autotrim_bytes_skipped 4 44617293824 autotrim_extents_failed 4 0 autotrim_bytes_failed 4 0 simple_trim_extents_written 4 0 simple_trim_bytes_written 4 0 simple_trim_extents_skipped 4 0 simple_trim_bytes_skipped 4 0 simple_trim_extents_failed 4 0 simple_trim_bytes_failed 4 0 arc_read_count 4 7803763 arc_read_bytes 4 928879902208 arc_write_count 4 65558151 arc_write_bytes 4 923930223048 direct_read_count 4 0 direct_read_bytes 4 0 direct_write_count 4 0 direct_write_bytes 4 0
May 16, 20251 yr Community Expert 19 minutes ago, Sak said: direct_read_count 4 0 direct_read_bytes 4 0 direct_write_count 4 0 direct_write_bytes 4 0 And zero Direct IO, so likely unrelated to that, for me it's all still working as before, I've updated my 2nd main server a couipleof days ago, also upgraded all the pools, and is the same as before, this one has the ARC limited to 64G:
May 17, 20251 yr On the Unraid page about ZFS memory use https://docs.unraid.net/unraid-os/manual/zfs/placeholder/#zfs-pools Quote Future update will include ability to configure the ARC via webGUI, including auto-adjust according to memory pressure, e.g., VM start/stop. Perhaps Unraid is doing something with the auto adjusting mechanism? root@Tower:~# cat /proc/spl/kstat/zfs/arcstats | grep c c 4 8045555296 c_min 4 3140741760 c_max 4 24000000000 'c_max' is the max memory zfs can use while 'c' is the current target. not sure what c was before the update, but the current 'c' is only at 8GB, no wonder my actual usage hovers around 7.5GB at the moment. root@Tower:~# arcstat 1 10 time read ddread ddh% dmread dmh% pread ph% size c avail 11:14:20 0 0 0 0 0 0 0 7.0G 7.0G 52.3G 11:14:21 18K 56 100 18.1K 100 16 100 6.9G 6.9G 52.3G 11:14:22 10K 0 0 10.4K 100 0 0 6.9G 6.9G 52.3G 11:14:23 196 0 0 196 100 0 0 6.9G 6.9G 52.3G 11:14:24 1.5K 737 100 775 100 35 0 6.9G 7.0G 52.3G 11:14:25 2.0K 709 100 1.3K 100 16 100 6.9G 7.0G 52.2G 11:14:26 337 59 100 251 100 27 100 6.9G 7.0G 52.3G 11:14:27 262 56 100 206 100 0 0 6.9G 6.9G 52.3G 11:14:28 6.3K 414 100 5.8K 100 0 0 6.9G 7.0G 52.3G 11:14:29 80K 79.5K 100 271 100 34 0 6.9G 6.9G 52.3G summary of the stats from chatgpt: read peaks at 79K, mostly hitting from ARC (ddh% and dmh% consistently at 100%). Very few prefetch reads (pread), and mostly demand reads (dmread), all hitting. This suggests the data you’re accessing is already cached — good ARC hit rate. size (actual ARC usage): 7.9 GB, very stable. c (target size): floats between 7.9–8.0 GB, nowhere near your c_max (24 GB). avail: plenty of free memory available (~52 GB), so ARC could grow if needed. ARC is working well with 100% cache hit rates — it’s serving your reads efficiently. Current workload only needs ~8 GB of ARC, so there’s no reason for it to grow toward the 24 GB limit. No issue exists unless you expected ZFS to cache more or your workload changes to require it. So I guess it is working as intended? 12 hours ago, JorgeB said: And zero Direct IO, so likely unrelated to that, for me it's all still working as before, I've updated my 2nd main server a couipleof days ago, also upgraded all the pools, and is the same as before, this one has the ARC limited to 64G: Could you run arcstat or cat /proc/spl/kstat/zfs/arcstats | grep c to check what your current target 'c' is?
May 17, 20251 yr Community Expert 2 hours ago, Sak said: Perhaps Unraid is doing something with the auto adjusting mechanism? Not yet, you can only set a limit in zfs.conf
June 22, 20251 yr Have you solved the issue? I encountered this last month but I downgraded to 7.0.1. Now with 7.1.4 available, I wonder if this issue resolved or not?
June 22, 20251 yr Author No, still consuming 4 or 5 gb, in previous versions 21.A lot of people have been reporting this here and in reddit but officially there is no acknowledge officially by unRAID
June 22, 20251 yr 7 hours ago, L0rdRaiden said:No, still consuming 4 or 5 gb, in previous versions 21.A lot of people have been reporting this here and in reddit but officially there is no acknowledge officially by unRAIDThank you for the response! Hope it will be resolved
June 23, 20251 yr supposedly it should support being set separately with the max:options zfs zfs_arc_min=8000000000options zfs zfs_arc_max=16000000000for 8gb min and 16gb max for example, and some have set it to the same as the max which would theoretically fix it to full.but might be somewhat broken in openZFS still? I'll test it next reboot, i'm seeing mixed reports on that but there are a lot of different versions and different platforms.
June 23, 20251 yr Community Expert 23 hours ago, L0rdRaiden said:A lot of people have been reporting this hereI've only seen this thread about this, and I'm unable to reproduce, but if you can give the necessary steps/config to do it I can try again.
June 23, 20251 yr I don't see this as a bug. it's been a feature of OpenZFS for a few versions now but it seems like it has either not worked in the past or the underlying behavior has changed and now more people are noticing it.I remember when I first switched to ZFS for my cache pools in 6.12 the ARC usage sat at the allocated max permanently, but now it does scale up and down with usage.If I fire up something busy hitting my ZFS pools with lots of repeated reads and writes the arc usage quickly maxes out and stays there, but if I let my system idle for a while it dials back down to 4-6gb out of my 16gb allocation and if I stop all containers and really drop usage to zero the arc usage pulls back to around 2gb, which is the currently set zfs_arc_min of 1/32 of system ram (I have 64gb on this system)I think it works well, but I haven't been able to test setting zfs_arc_min which we are supposed to be able to do according to openZFS.
June 27, 20251 yr Looks like setting a minimum works just fine in Unraid 7.1.4options zfs zfs_arc_min=8000000000options zfs zfs_arc_max=16000000000Correctly sets both the minimum and maximum and the system works within those bounds. I haven't tested setting the minimum and max the same yet as this server is my main production server and I cant reboot it willy-nilly, but I cant see why that wouldn't work according to the OpenZFS documentation and some threads on other ZFS forums
June 30, 20251 yr Does this show the cause of the OP's issue :'During system boot, the file /etc/modprobe.d/zfs.conf is auto-generated to limit the ZFS ARC to 1/8 of installed memory. This can be overridden if necessary by creating a custom 'config/modprobe.d/zfs.conf' file. Future update will include ability to configure the ARC via webGUI, including auto-adjust according to memory pressure, e.g., VM start/stop.'About | Unraid DocsAbout | Unraid DocsPlease add additional guides on how to use and configure ZFS in this subfolder
June 30, 20251 yr Community Expert Technically it is adjustable via webgui from the Tools > System Drivers page
July 3, 20251 yr Oh, and just a note, the meter on the dashboard is in GibiBytes, so if for example you set any of these variables to "8000000000" you will actually be setting 7.45GiB so keep that in mind if you need to see exact values in order to sleep soundly.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.