Hey everyone,

 

I updated from 6.9...11(??) a few days ago and now something is reading from every drive in the server every few minutes. Array drives, cache pool drives, unassigned drives (mounted or not), all of them. I spin them down, they spin back up within minutes if not seconds. I disabled Docker, deleted any plugin relating to system stats/monitoring, stopped my VM, still happening. File Activity plugin is catching activity on my VM and a few dockers, that's it. (only by share, nothing at all by disk). These are SAS and I'm running the spin down plugin for them.

 

I'm not sure what to try next, so here I am. 🙂

 

 

 

planetexpress-diagnostics-20221117-2058.zip

Bump :)

20 minutes ago, thestip said:

Bump

Try removing the sas spindown plugin and re install it. I domt see SAS helper messages in the log,

 

try removing the plugin all of the drives seem to be WD ata drives not SAS

Edited November 18, 20223 yr by SimonF

Just did that, drives are still getting spun up/read. Spun them down, spun right back up. Even the unassigned, unmounted, unmountable ones are getting read.

 

The only "new" things are the OS upgrade, the user scripts plugin and a few things from theme park (custom login screen). I've uninstalled the scripts and theme plugins (but the custom login screen remains)

The drives spin down just fine, something just keeps spinning them up again.

Edited November 19, 20223 yr by thestip

I'm using the SAS spin down because the drives are on Dell Perc 310 cards.

Edited November 19, 20223 yr by thestip

7 hours ago, thestip said:

I'm using the SAS spin down because the drives are on Dell Perc 310 cards.

Try running in safe mode to see if the system performs normally. SAS Helper pluging is only required if you have SAS Drives attached as they need to have different commands to spindown/check status etc.

 

Also if you able to update to 6.11.4 it will fix some of the diagnostics and if you could re-post diags.

 

 

I was finally able to mess around and update the server. The drives don't spin up like this in safe mode, but I've uninstalled every plugin aside from CA, and it still happens.

 

New diags attached!

planetexpress-diagnostics-20230121-2206.zip

Edited January 22, 20233 yr by thestip

Moments after spinning down a drive...

Jan 22 18:51:35 PlanetExpress  emhttpd: spinning down /dev/sdl
Jan 22 18:51:51 PlanetExpress  emhttpd: spinning down /dev/sdm
Jan 22 18:51:53 PlanetExpress  emhttpd: spinning down /dev/sdd
Jan 22 18:52:05 PlanetExpress  emhttpd: read SMART /dev/sdd
Jan 22 18:52:10 PlanetExpress  emhttpd: read SMART /dev/sdm
Jan 22 18:52:15 PlanetExpress  emhttpd: read SMART /dev/sdl

planetexpress-diagnostics-20230122-1859.zip

Edited January 23, 20233 yr by thestip
Updated diags

Friendly bump. 🙂

 

The interval between the reads is one minute. I played around with Grafana and Prometheus a while back, but I removed both of them. I don't know what else would be reading the SMART info all the time.

(taps mic) This thing on? 😁

Unraid will regularly read the SMART information if it thinks is drive is NOT spun down.   The strange thing is that you say this does not happen in Safe mode so it has to be something that is only running when not in Safe mode.   Have you identified what that might be?    I  do not see how anyone can guess what might be spinning up the drives without being able to directly examine your system.

 

BTW:   You seem to have a lot of .cfg files in the config/shares folder on the flash drive that relate to shares you no longer seem to have.    It is probably worth deleting them to clean up the diagnostics.

8 hours ago, itimpi said:

Have you identified what that might be?

 

That's the thing. Nothing in the logs says anything about a SMART check or any unexpected drive activity, unless I spin a drive down, yet there's a check every minute. I have no idea how to tell what's checking the disks. Removed the extra cfg files, posted new diagnostics. Since it wasn't any of the plugins, I've reinstalled them. 

 

Recap: uninstalled all plugins aside from CA, disabled docker, disabled VMs. Drives still get read/checked every minute. Prometheus and a few associated apps are the only thing I've messed with that would do that, but they're no longer installed. It's like part of one of those apps didn't get uninstalled.

 

planetexpress-diagnostics-20230129-1930.zip

Your system share has files on the array, and it is set to only have files on the array.

 

The system share normally has docker and libvirt.img on them, but in your case you have put docker.img elsewhere.

 

Disable VM Manager, set system share to cache:prefer to get it moved to cache (or set it to use some other pool and move it there).

3 hours ago, trurl said:

Disable VM Manager, set system share to cache:prefer to get it moved to cache (or set it to use some other pool and move it there).

 

I was just about to post saying it didn't work when I went into safe mode again and spun a drive down. Damned thing spun up. I just didn't wait long enough to see it happen last time. Sorry for the bad info/waste of time.

 

4 hours ago, trurl said:

Your system share has files on the array, and it is set to only have files on the array.

 

Fixed, thanks!

 

Now that you have system share off the array, do your disks stay spun down?

No, but I didn't expect it to since I said every drive was being read, in the array or not. And now I see it's been happening in safe mode as well. The SMART check is set for 1,800 seconds, so that's not it. (I even set it to 0 to make sure)

 

Update: Found the issue. "BTRFS operation is running". 

 

I just happened to be looking at the "STOP" button when the drives all got read. It grayed out for just a moment and some text flashed next to it.  (the button did not gray out every time.) Managed to get a screenshot and could see "BTRFS operation is running". I had an unused drive in a pool that was BTRFS (old drive was going to wipe it). Took the drive out of the pool, deleted the pool. No more checks every minute, drives all spin down and stay down.

 

Shouldn't something like that have been logged?

 

Edited February 4, 20233 yr by thestip
solution