I was hoping this would be fixed on v6.4 but I'm still seeing the same unexplained disk spin ups.

 

This is a good example, I have two scheduled rsync tasks (at 1AM and 1:15AM) everyday to this server, backup share is set to include only disk1 but it shouldn't matter as rsync is done directly to disk share, i.e.,

 

rsync -av --progress --delete -e ssh /source/ [email protected]:/mnt/disk1/Backup/TV/

 

The 2nd rsync task finished at ~1.20AM and disk1 correctly spun down 1 hour later, but all other disks (except disk2 which is set to never spin down) spun up during the copy to disk1.

 

There's nothing using the other disks except a weekly backup task on Saturdays, no dockers on this server, couple of VMs on the cache disk but no mapped server drives.

 

Anything else I can do to help diagnose the problem?

 

 

 

Jun 26 16:35:04 Tower1 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log ipmifan
Jun 27 01:00:00 Tower1 sshd[27089]: Accepted none for root from 192.168.1.27 port 14461 ssh2
Jun 27 01:01:35 Tower1 sshd[27089]: Received disconnect from 192.168.1.27 port 14461:11: disconnected by user
Jun 27 01:01:35 Tower1 sshd[27089]: Disconnected from user root 192.168.1.27 port 14461
Jun 27 01:15:00 Tower1 sshd[32080]: Accepted none for root from 192.168.1.27 port 33791 ssh2
Jun 27 01:19:14 Tower1 sshd[32080]: Received disconnect from 192.168.1.27 port 33791:11: disconnected by user
Jun 27 01:19:14 Tower1 sshd[32080]: Disconnected from user root 192.168.1.27 port 33791
Jun 27 02:02:05 Tower1 kernel: mdcmd (94): spindown 3
Jun 27 02:02:05 Tower1 kernel: mdcmd (95): spindown 4
Jun 27 02:02:06 Tower1 kernel: mdcmd (96): spindown 5
Jun 27 02:02:06 Tower1 kernel: mdcmd (97): spindown 6
Jun 27 02:03:02 Tower1 kernel: mdcmd (98): spindown 7
Jun 27 02:20:15 Tower1 kernel: mdcmd (99): spindown 1
Jun 27 05:00:05 Tower1 root: /mnt/cache: 176.9 GiB (189953007616 bytes) trimmed

tower1-diagnostics-20170627-1440.zip

Edited June 27, 20179 yr by johnnie.black

@johnnie.black

 

Was this working on some prior version of unRAID? Based on first line seems that was an issue in at earlier versions as well.

 

Are there any symlinks on disk1 to the other other disks? Assuming not, but just checking.

 

If you run the command manually, are the other disks spinning up?

 

If not, how about via cron (scheduling it to run when you are sitting there to observe the system)?

 

If you are able to consistently reproduce, you might log htop calls (or maybe there is a better way to dump job executions) to give hints as to what is happening to cause the spinup.

 

Wondering if there is a "sync" somewhere. Sync has a propensity to spin up disks.

20 minutes ago, bjp999 said:

Was this working on some prior version of unRAID? Based on first line seems that was an issue in at earlier versions as well.

 

This has been a problem for some time, at least v6.2, Tom mentioned some time ago that he found a reason for this and together with new mover should fix this (though I'm not using the mover here)

 

20 minutes ago, bjp999 said:

Are there any symlinks on disk1 to the other other disks? Assuming not, but just checking.

 

No symlinks, disk1 is acting as a backup destination only

 

20 minutes ago, bjp999 said:

If you run the command manually, are the other disks spinning up?

 

Command runs from another server, it's only the rsync command, sometimes only a few disks spin up, I ran it manually now a screen shows the result, only disks 1 and 2 should be spinning, disks 3, 6 and 7 woke up this time

Edited June 27, 20179 yr by johnnie.black

Don't have a solution, but do have an idea to perhaps lessen the impact.

 

You can spindown disks with the command:

 

/user/local/sbin/mdcmd #

 

Where # is the disk number. (e.g., parity =0, disk1 = 1, etc.)

 

You could set a cron script to run soon after the earlier script that spins down the unaffected disks.

 

One reason some of the disks are not spinning up is that whatever data was attempted to be read from that disk was already in the disk cache. Seems to imply it was minimal.

8 minutes ago, bjp999 said:

Don't have a solution, but do have an idea to perhaps lessen the impact.

 

My biggest problem with this is not the extra power used by the disks, it's the extra spin ups and the effect they can have on the disks longevity, disks that would only need to spin up once a week will be spinning up once a day (or most days).

Understood and agreed. This does not fix the problem.

 

My understanding, though, is that mechanical spinup is not as much an issue as the thermal impact of repeated spinups and downs. By spinning down almost immediately, the thermal affect would be minimized. 

 

So maybe this is better than nothing while you continue work on (or wait on) the real solution!

Could it be the turbo write feature spinning up all drives?


Sent from my iPhone using Tapatalk

3 minutes ago, miniwalks said:

Could it be the turbo write feature spinning up all drives?


Sent from my iPhone using Tapatalk

 

There's currently no parity on that server.

 
There's currently no parity on that server.

Ok, just throwing it out there

Can you try forcing it to not try turbo write and let us know?

Also backing up to something without parity is asking for trouble


Sent from my iPhone using Tapatalk

34 minutes ago, miniwalks said:

Also backing up to something without parity is asking for trouble

I'm genuinely not being a smart-alec here, why do you say that? If it's a true backup, then the original is elsewhere. It would take 2 (or more) simultaneous errors / failures to lose data, so what difference does adding parity make? Unraid parity can only protect from a clean disk failure, which is pretty rare. Statistically speaking, I mean. I would rather dedicate the lost parity capacity to additional depth of backup recovery, version wise.

 

For a primary unraid server, parity is a must. But a backup server?

Without threadcrapping;
Prevent bitrot
Prevent failure of backup
All the reasons you want your data protected, the cost of a parity disk vs the cost/time of re backing up all that data.

If at the end of the day it's backing up to a single disk, why even use unraid for the secondary? Just attach a HDD via USB caddy then if that's your attitude


Sent from my iPhone using Tapatalk

15 minutes ago, miniwalks said:

Prevent bitrot

Parity does nothing for file integrity, only full disk failures.

16 minutes ago, miniwalks said:

If at the end of the day it's backing up to a single disk, why even use unraid for the secondary? Just attach a HDD via USB caddy then if that's your attitude

Because to backup a full unraid server with many TB's of data takes more than just a HDD with USB. Much more convenient to spin up a second unraid tower.

Because to backup a full unraid server with many TB's of data takes more than just a HDD with USB. Much more convenient to spin up a second unraid tower.

That makes no sense. OP mentioned that it's backing up to a single disk. So you would want to spin up a full OS and Hardware stack for 1 disk as opposed to attaching 1 disk?

My backup NAS isn't an unraid so that I can prevent bitrot, it's like the old idea of having 2 sets of firewalls, you don't use the same ones so the one exploit won't get through both.

With that said, this is getting way off topic. So let's just leave it there.

OP,
Can you try force turbo write to off and report back?


Sent from my iPhone using Tapatalk

I tend to agree with @johnnie.black and @jonathanm -

 

A full backup on top of a parity protected array is a lot of protection.And I think Johnnie has dual parity on his main server.

 

Interestingly - the main thing you loose out on is the parity check - which is a very nice integrity checker. I guess there are alternatives - verifying your checksums for example, but the simplicity and speed of the parity check is pretty darn handy. For that reason my backup array has a parity disk, at least for now. Once it if fully burned in and proven, I may move it to being a data disk. Not decided yet.

 

But I respect @miniwalks perspective - certainly better to be more protected than less. But at some point you have to say enough already. And invest in some fire extinguishers and alarmed vibration sensors on the windows, because risk from fire and theft become more likely than a 3rd level or 4th level failure of the data itself.

7 hours ago, miniwalks said:

OP,
Can you try force turbo write to off and report back?

 

Like I said, the server has no parity, there is no turbo write.

 

7 hours ago, miniwalks said:

That makes no sense. OP mentioned that it's backing up to a single disk. So you would want to spin up a full OS and Hardware stack for 1 disk as opposed to attaching 1 disk?

 

This is of topic, but the reason I used it as a backup server is because it's always on because since it's where my VMs are, disk1 backups a share, and disks 3 to 7 backup a different share, don't need parity for now on this server since data on disk1 is not very important and data on disks 3 to 7 is a second backup.

 

7 hours ago, bjp999 said:

Interestingly - the main thing you loose out on is the parity check - which is a very nice integrity checker.

 

You can still do a read check on all disks, and to my surprise, and because it once had parity and a schedule parity check, it still works on the old schedule:

 

 

Edited June 28, 20179 yr by johnnie.black

Now on topic, I wasn't using cache dirs because in my experience it makes things worse in these situations (when nothing is accessing the disks), but I tried again to confirm and it's worse, disks keep spinning up to feed cache dirs, there was nothing accessing these disks before the backup to disk1 at 1AM

 

Jun 28 00:38:11 Tower1 kernel: mdcmd (123): spindown 5
Jun 28 00:38:12 Tower1 kernel: mdcmd (124): spindown 6
Jun 28 00:39:05 Tower1 kernel: mdcmd (125): spindown 3
Jun 28 00:40:04 Tower1 kernel: mdcmd (126): spindown 7
Jun 28 00:40:13 Tower1 kernel: mdcmd (127): spindown 4
Jun 28 00:57:05 Tower1 kernel: mdcmd (128): spindown 1
Jun 28 01:00:00 Tower1 sshd[1348]: Accepted none for root from 192.168.1.27 port 21112 ssh2
Jun 28 01:00:22 Tower1 sshd[1348]: Received disconnect from 192.168.1.27 port 21112:11: disconnected by user
Jun 28 01:00:22 Tower1 sshd[1348]: Disconnected from user root 192.168.1.27 port 21112
Jun 28 01:15:00 Tower1 sshd[2005]: Accepted none for root from 192.168.1.27 port 55125 ssh2
Jun 28 01:15:30 Tower1 sshd[2005]: Received disconnect from 192.168.1.27 port 55125:11: disconnected by user
Jun 28 01:15:30 Tower1 sshd[2005]: Disconnected from user root 192.168.1.27 port 55125
Jun 28 01:46:34 Tower1 kernel: mdcmd (129): spindown 1
Jun 28 02:11:04 Tower1 kernel: mdcmd (130): spindown 4

 

 

Edited June 28, 20179 yr by johnnie.black

Disks are also spinning up after any activity to other disks or no activity at all, a scripted btrfs send/received to disk2 at 0:00AM made all disks spin up (they are now set to spin down after 30 minutes), then only disk7 spun up after the 01:00AM rsync task to disk1, and finally disk1 at 5.00AM without anything going on at all.

 

 

Jun 29 00:05:33 Tower1 kernel: hrtimer: interrupt took 20701 ns
Jun 29 00:37:02 Tower1 kernel: mdcmd (131): spindown 1
Jun 29 00:39:03 Tower1 kernel: mdcmd (132): spindown 3
Jun 29 00:39:04 Tower1 kernel: mdcmd (133): spindown 4
Jun 29 00:39:04 Tower1 kernel: mdcmd (134): spindown 5
Jun 29 00:39:05 Tower1 kernel: mdcmd (135): spindown 6
Jun 29 00:39:05 Tower1 kernel: mdcmd (136): spindown 7
Jun 29 01:00:00 Tower1 sshd[10579]: Accepted none for root from 192.168.1.27 port 51672 ssh2
Jun 29 01:00:11 Tower1 sshd[10579]: Received disconnect from 192.168.1.27 port 51672:11: disconnected by user
Jun 29 01:00:11 Tower1 sshd[10579]: Disconnected from user root 192.168.1.27 port 51672
Jun 29 01:15:00 Tower1 sshd[15290]: Accepted none for root from 192.168.1.27 port 11016 ssh2
Jun 29 01:18:53 Tower1 sshd[15290]: Received disconnect from 192.168.1.27 port 11016:11: disconnected by user
Jun 29 01:18:53 Tower1 sshd[15290]: Disconnected from user root 192.168.1.27 port 11016
Jun 29 01:49:02 Tower1 kernel: mdcmd (137): spindown 7
Jun 29 01:50:01 Tower1 kernel: mdcmd (138): spindown 1
Jun 29 05:00:05 Tower1 root: /mnt/cache: 176.9 GiB (189953007616 bytes) trimmed
Jun 29 05:49:02 Tower1 kernel: mdcmd (139): spindown 1

@limetechany ideas on this, or it's just something I have to live with?