Hello all,

 

I am in a bad spot with my server this morning and am concerned about possible data loss.

 

I am running 7.0.1 and have recently added a 3 disk, 2.5" SSD, ZFS pool in RAIDz. I went to work 'moving' (this is my major mistake) more data to the pool this morning and the mc operation hung. Looking at it, the ZFS pool had spun down, which I wish I had turned off, but I didn't realize it was going to do it ahead of time. It seems now that those drives will not spin back up. I have shut down all of my VMs and containers. They all went down gracefully just fine since they have no dependency on this new pool at this point.

 

However, the system will not reboot, I cannot generate diagnostics either, it hangs here:

 

I was able to grab the syslog and also a ps -eaf from the CLI, both attached.

 

I believe this is the applicable snippet from the CLI. Drive sdl, sdm and sdn are the new ZFS pool disks, named photo-video:

Mar  3 10:01:13 RackServer kernel: 
Mar  3 10:13:43 RackServer emhttpd: spinning down /dev/sdi
Mar  3 10:13:45 RackServer emhttpd: read SMART /dev/sdi
Mar  3 10:28:48 RackServer kernel: mptbase: ioc0: mpt_fault_reset_work: IOC is non-operational !!!!
Mar  3 10:28:48 RackServer kernel: mptbase: ioc0: WARNING - mpt_fault_reset_work: Running mpt_dead_ioc thread success !
Mar  3 10:28:48 RackServer kernel: sd 11:0:0:0: [sdl] Synchronizing SCSI cache
Mar  3 10:29:03 RackServer kernel: mptscsih: ioc0: attempting task abort! (sc=00000000ebdea072)
Mar  3 10:29:03 RackServer kernel: sd 11:0:1:0: [sdm] tag#75 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e5 00
Mar  3 10:29:03 RackServer kernel: mptscsih: ioc0: WARNING - TaskMgmt type=1: IOC Not operational (0xffffffff)!
Mar  3 10:29:03 RackServer kernel: mptscsih: ioc0: WARNING - Issuing HardReset from mptscsih_IssueTaskMgmt!!
Mar  3 10:29:03 RackServer kernel: mptbase: ioc0: Initiating recovery
Mar  3 10:29:03 RackServer kernel: mptbase: ioc0: WARNING - Unexpected doorbell active!
Mar  3 10:30:05 RackServer kernel: mptbase: ioc0: ERROR - Failed to come READY after reset! IocState=f0000000
Mar  3 10:30:05 RackServer kernel: mptbase: ioc0: WARNING - ResetHistory bit failed to clear!
Mar  3 10:30:05 RackServer kernel: mptbase: ioc0: ERROR - Diagnostic reset FAILED! (ffffffffh)
Mar  3 10:30:05 RackServer kernel: mptbase: ioc0: WARNING - NOT READY WARNING!
Mar  3 10:30:05 RackServer kernel: mptbase: WARNING - (-1) Cannot recover ioc0, doorbell=0xffffffff
Mar  3 10:30:05 RackServer kernel: mptscsih: ioc0: WARNING - TaskMgmt HardReset FAILED!!
Mar  3 10:30:05 RackServer kernel: mptscsih: ioc0: task abort: SUCCESS (rv=2002) (sc=00000000ebdea072)
Mar  3 10:39:29 RackServer nginx: 2025/03/03 10:39:29 [error] 1582777#1582777: *1436877 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.20.10.193, server: , request: "POST /plugins/unassigned.devices/include/UnassignedDevices.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.sock", host: "rackserver.int.lantard.com", referrer: "https://rackserver.int.lantard.com/Main"
Mar  3 10:40:00 RackServer nginx: 2025/03/03 10:40:00 [error] 1582777#1582777: *1436877 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.20.10.193, server: , request: "POST /plugins/unassigned.devices/include/UnassignedDevices.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.sock", host: "rackserver.int.lantard.com", referrer: "https://rackserver.int.lantard.com/Main"
Mar  3 10:40:38 RackServer kernel: vethb9dc02d: renamed from eth0
Mar  3 10:40:49 RackServer kernel: veth3687d56: renamed from eth0

 

I don't know what else to do besides power it off at this point. Does anyone have any advice before I take that drastic action and cross my fingers it comes back up?

 

I do have backups of much of the data that has been moved to the new pool, but there are some gaps for sure which is the project I am working on now, reorganizing data and backups, but made some foolish decisions along the way removing some safety nets that should have been kept.

 

Anyway, any help you all can provide would be very much appreciated.

 

Thanks!

rackserver-syslog-20250303-1812.zip ps-as-the-server-sits-now.txt

Edited March 3, 20251 yr by harshl

I'm not seeing any actual issues with the pool so far, can you get the output from:

 

zpool status

 

P.S. would recommend reporting the issue with FCP preventing getting the diags, I believe it's the 2nd time I've seen this in the last couple of days.

13 minutes ago, JorgeB said:

I'm not seeing any actual issues with the pool so far, can you get the output from:

 

zpool status

 

zpool status
  pool: photo-video
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
config:

    NAME         STATE     READ WRITE CKSUM
    photo-video  ONLINE       0     0     0
      raidz1-0   ONLINE       0     0     0
        sdl1     ONLINE       0   174     0
        sdm1     ONLINE       0     0     0
        sdn1     ONLINE       0     0     0

errors: No known data errors

11 minutes ago, JorgeB said:

P.S. would recommend reporting the issue with FCP preventing getting the diags, I believe it's the 2nd time I've seen this in the last couple of days.

Will do.

There were some write errors but the pool and data should be fine for now, possibly more a controller issue, type reboot in the CLI, if it doesn't reboot after 5 minutes you will need to force it, then run a scrub on the pool.

Just now, JorgeB said:

There were some write errors but the pool and data should be fine for now, possibly more a controller issue, type reboot in the CLI, if it doesn't reboot after 5 minutes you will need to force it, then run a scrub on the pool.

I tried rebooting it well over an hour ago and it hasn't gone down. Something has it hung. I guess I'll force power it off and maybe replace the HBA and cables?

Just now, harshl said:

I guess I'll force power it off and maybe replace the HBA and cables?

If you have a different one, it would be worth trying, if not, just check/replace cables and see if it happens again, if there's a problem it will likely show up under more under load, like during a scrub or large copy.

9 hours ago, JorgeB said:

If you have a different one, it would be worth trying, if not, just check/replace cables and see if it happens again, if there's a problem it will likely show up under more under load, like during a scrub or large copy.

JorgeB, thank you for the responses and insight. The server came back up and the zpool mounted fine. I have copied the data off to two other locations and will continue troubleshooting the failure. Nice to have a friendly community to lean on when you get into these situations.

 

Thanks again!

As a troubleshooting step, I would like to disable spin down for this zpool. It has the following drives present:

 

If I click on the first drive I see the option to change spin down:

 

However, when I click on the other two disks, they do not have that option:

 

Will the other disks follow the setting of the 1st disk, or is there somewhere else that I need to set this?

 

Thank you!

1 hour ago, harshl said:

Will the other disks follow the setting of the 1st disk

They should.

On 3/4/2025 at 8:59 AM, JorgeB said:

They should.

FYI, the pool is not obeying the setting from the first disk. Is there a way to set it on the other two disks so they are in sync?

That shows the first disk is up and the others down, if it's not obeying it should the other way around, please post the diagnostics after the pool was supposed to spin down.

8 hours ago, JorgeB said:

That shows the first disk is up and the others down, if it's not obeying it should the other way around, please post the diagnostics after the pool was supposed to spin down.

Sorry, I stated in the text what I was doing, but the screenshot shows something different. Sorry for the confusion.

 

It is doing exactly what it has been configured to do. As a troubleshooting step, I wanted to disable spin down. I'm pretty convinced now that it has nothing to do with the issue, but regardless, I don't want this pool to spin down.

 

I have set the first disk not to, but the remaining two disks still spin down and don't have a setting in the GUI I can find to set them not to.

 

It is currently behaving exactly as configured from my perspective.

 

Would you like me to submit a bug somewhere about it either not obeying the config of the first drive, or, the setting missing on subsequent drives?

Thanks!

28 minutes ago, harshl said:

Would you like me to submit a bug somewhere about it either not obeying the config of the first drive, or, the setting missing on subsequent drives?

Thanks!

Let me test and if I find the same thing I will create a bug report.

Yeah, this is a bug, in 6.12 there was a setting per pool device, in 7.0 it's only for the main pool device, maybe the intention was to have that one apply to the remaining pool members, which would make sense, but currently it's not possible to set a custom spin down for the other pool members, I've already reported this to LT, and it should be fixed in an upcoming release.

Thank you so much!

Just to close the loop on this, the issue was either with the cable or the controller, not sure which as I replaced them at the same time. I did not replace the backplane or anything else in the mix.

 

Thanks for the technical and moral support!

Turns out there's still a bug but only for the first time you set a custom spin down for a pool, only the first pool member will spin down, if you reboot, or just re-start the array, all pool members should spin down from then on.

2 hours ago, JorgeB said:

Turns out there's still a bug but only for the first time you set a custom spin down for a pool, only the first pool member will spin down, if you reboot, or just re-start the array, all pool members should spin down from then on.

Confirmed. I am seeing now that after rebooting to replace the controller in the server, none of the drives are spinning down anymore, as configured.

 

Thanks!

This bug has been fixed on 7.1.0-beta.1, not sure yet if it's going to be backported to 7.0.2