All disks spinning up for no reason after spin-down

[dgwharrison]

Recently, perhaps since 6.6.7 but I couldn't say exactly, I've noticed that all my disks including the parity are spun up all the time. When you manually press 'spin down' from the main screen they do indeed spin down, but less than 2 minutes later they're back up and running again. I've checked the logs, I can see where I've spun them down which works fine (although it tries to spin down NVME disks which obviously spits an error). But there are no log messages for spinning the disks back up. Log below:

Apr 18 11:05:08 Zeus emhttpd: Spinning down all drives...
Apr 18 11:05:08 Zeus kernel: mdcmd (1050): spindown 0
Apr 18 11:05:08 Zeus kernel: mdcmd (1051): spindown 1
Apr 18 11:05:09 Zeus kernel: mdcmd (1052): spindown 2
Apr 18 11:05:09 Zeus kernel: mdcmd (1053): spindown 3
Apr 18 11:05:10 Zeus kernel: mdcmd (1054): spindown 4
Apr 18 11:05:10 Zeus kernel: mdcmd (1055): spindown 5
Apr 18 11:05:11 Zeus kernel: mdcmd (1056): spindown 6
Apr 18 11:05:12 Zeus kernel: mdcmd (1057): spindown 7
Apr 18 11:05:12 Zeus kernel: mdcmd (1058): spindown 8
Apr 18 11:05:13 Zeus kernel: mdcmd (1059): spindown 9
Apr 18 11:05:13 Zeus emhttpd: shcmd (86844): /usr/sbin/hdparm -y /dev/nvme0n1
Apr 18 11:05:13 Zeus root:  HDIO_DRIVE_CMD(standby) failed: Inappropriate ioctl for device
Apr 18 11:05:13 Zeus root: 
Apr 18 11:05:13 Zeus root: /dev/nvme0n1:
Apr 18 11:05:13 Zeus root:  issuing standby command
Apr 18 11:05:13 Zeus emhttpd: shcmd (86844): exit status: 25
Apr 18 11:05:13 Zeus emhttpd: shcmd (86845): /usr/sbin/hdparm -y /dev/nvme1n1
Apr 18 11:05:13 Zeus root:  HDIO_DRIVE_CMD(standby) failed: Inappropriate ioctl for device
Apr 18 11:05:13 Zeus root: 
Apr 18 11:05:13 Zeus root: /dev/nvme1n1:
Apr 18 11:05:13 Zeus root:  issuing standby command
Apr 18 11:05:13 Zeus emhttpd: shcmd (86845): exit status: 25

Is there some way I can turn on some more logging to resolve this problem?

 

I've attached a diagnostics file generated just now. 

zeus-diagnostics-20190418-1112.zip

[Squid]

disk 6 has corruption on it

Mar 27 01:28:40 Zeus kernel: XFS (md6): Unmount and run xfs_repair

Run the File system checks on it https://wiki.unraid.net/Check_Disk_Filesystems

[dgwharrison]

33 minutes ago, Squid said:

disk 6 has corruption on it

Mar 27 01:28:40 Zeus kernel: XFS (md6): Unmount and run xfs_repair

Run the File system checks on it https://wiki.unraid.net/Check_Disk_Filesystems

Hi, thanks for the prompt reply @Squid. I did that, there were no errors. It did want to run a parity check after being brought out of maintenance mode however I stopped that as no changes were made and a check was completed with 0 errors a week ago and the problem has been going longer than that so parity isn't likely to be the issue.

 

Sure enough, a couple minutes after I spun the disks down, they all spun up again.

 

What to check next?

[dgwharrison]

Hi @Squid, I was running the checks between your first and second post and only just noticed your second post. For disk 6 I ran check file system from the web ui with the array in maintenance mode with the option -v (dropped -n so it would actually repair). The output shows no errors per below. I stopped the array and started it not in maintenance mode. Again it tried the parity check, which I stopped because it takes days and it was done with zero errors a week ago. Spun down all the disks. All disks immediately span back up. I've attached a new diagnostics output. Below is the output from the file system check with -v option only:

Phase 1 - find and verify superblock...
        - block cache size set to 1520352 entries
Phase 2 - using internal log
        - zero log...
zero_log: head block 44836 tail block 44836
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 3
        - agno = 1
Phase 5 - rebuild AG headers and trees...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...

        XFS_REPAIR Summary    Thu Apr 18 13:10:47 2019

Phase		Start		End		Duration
Phase 1:	04/18 13:10:46	04/18 13:10:46
Phase 2:	04/18 13:10:46	04/18 13:10:46
Phase 3:	04/18 13:10:46	04/18 13:10:47	1 second
Phase 4:	04/18 13:10:47	04/18 13:10:47
Phase 5:	04/18 13:10:47	04/18 13:10:47
Phase 6:	04/18 13:10:47	04/18 13:10:47
Phase 7:	04/18 13:10:47	04/18 13:10:47

Total run time: 1 second
done

 

zeus-diagnostics-20190418-1316.zip

[itimpi]

Coming out of Maintenance mode would not trigger a parity check (the whole idea if Maintenance mode is to maintain parity while doing something like a file system repair).     Do you know why the system wants to run a parity check (e.g. did you have an unclean shutdown).

[dgwharrison]

On 4/18/2019 at 1:47 PM, itimpi said:

Coming out of Maintenance mode would not trigger a parity check (the whole idea if Maintenance mode is to maintain parity while doing something like a file system repair).     Do you know why the system wants to run a parity check (e.g. did you have an unclean shutdown).

No, definitely no unclean shutdown in memorable history, certainly not since the last parity check.

 

That being said, it's been a couple days and now the array is spinning down unused disks like it's supposed to. Don't know why it took a couple days to behave though.