I have a recurring issue where Unraid freezes and is completely unresponsive whenever there is a power cut.  UPS has more than  enough power and can run for 30+ minutes on battery.  However, even if there is a 5 second power cut Unraid becomes complete unresponsive and I'm forced to do an unclean shutdown.  When I initially set it up and tested it by unplugging the UPS it shut down fine.  I'm afraid to test it again as it looks like everything is configured correctly.  When the server becomes unresponsive I can't even SSH into it, it's 100% unresponsive.  That being said, even though Unraid is totally unresponsive I can connect to numerous containers and they will run fine.  After the unclean shutdown I have to deal with corrupted files and databases.  Also, even if I turn off all my VMs and containers performance is terrible.  So bad that trying to connect to a share from a Linux or Windows VM just times out.  My CPU utilization is at 50% consistently since the issue and the containers and VMs aren't causing the high utilization and there is only 1 disk being checked or file integrity.  It makes the system unusable as Plex thinks the server is down the performance is so bad,

 

I've scoured the syslog every time to see if I can find any hint of an error but there is nothing.  I can't figure it out and I've run out of ideas of what else I can do to investigate.  Anyone have any ideas what could cause this issue or what else I could do to further investigate?

 

goathead-diagnostics-20241216-1201.zip

Edited December 16, 20241 yr by Bmalone

First to verify does PC unresponsive due to power source switching, disable APCUPSD then try turn off and on main power source.

Do you have a switch or router that is not protected by the UPS?

My UPS has 4 outlets that are powered during an outage, but

2 that lose power during any outage (they are 'convenience' outlets,

meaning 'not-at-all-convienent').  I had my router plugged into

one of those 2 and would lose access to UNRAID (and everything else)

after an outage  until the router was manually reset.

 

-- Tom

On 12/16/2024 at 6:59 PM, Vr2Io said:

First to verify does PC unresponsive due to power source switching, disable APCUPSD then try turn off and on main power source.

Sorry, didn't notice anyone had replied.  Sorry for the delay.  I'll have to test this.  To be clear, you're suggesting that I disable the APC daemon and then just unplug the UPS from the mains?

On 12/16/2024 at 7:31 PM, Tom3 said:

Do you have a switch or router that is not protected by the UPS?

My UPS has 4 outlets that are powered during an outage, but

2 that lose power during any outage (they are 'convenience' outlets,

meaning 'not-at-all-convienent').  I had my router plugged into

one of those 2 and would lose access to UNRAID (and everything else)

after an outage  until the router was manually reset.

 

-- Tom

No, all networking equipment are on UPS'.  This is an enterprise 3000VA UPS and it doesn't have any surge only outlets.  When  Unraid shits the bed I can ping the server and it responds but it refuses any other interaction.

29 minutes ago, Bmalone said:

To be clear, you're suggesting that I disable the APC daemon and then just unplug the UPS from the mains?

 

Yes, this to verify does problem come from power source switching.

Connect up a keyboard and monitor directly to the server.  Now disconnect the mains and watch the screen.  It should tell what is happening.  There are a few MB that do not completely shutdown properly.  Sometimes, there are processes (Dockers, plugins, VMs) that do not shutdown properly which will prevent the computer from shutting down.  See if you can figure what is happening.  IF it hangs during the shutdown, take a picture of the monitor.  Make sure that text on the picture is readable!!!!

16 hours ago, Frank1940 said:

Connect up a keyboard and monitor directly to the server.  Now disconnect the mains and watch the screen.  It should tell what is happening.  There are a few MB that do not completely shutdown properly.  Sometimes, there are processes (Dockers, plugins, VMs) that do not shutdown properly which will prevent the computer from shutting down.  See if you can figure what is happening.  IF it hangs during the shutdown, take a picture of the monitor.  Make sure that text on the picture is readable!!!!

Maybe I can try that once I'm not experiencing issues but since the powrer loss and unclean shutdown I'm having significant performance issues and the CPU is constantly at 40-50% utilization and all my containers are running terribly slow or not fully functional, even after deleting the apopdata and container and restoring.  I'll have to start from scratch recreating them.  This server has 150TB of data so I don't want to cause any further issues until these are resolved.

 

I will say that this has now happened twice, but each time Unraid is completely inaccessible, but IPMI is accessible.  The server is a Supermicro X10DRi-T4+ without any known issues that I can find with regards to powering down.  Is there nothing in the diagnostics or in the syslog (mirrored to USB flash drive) that might indicate what's happening?

6 minutes ago, Bmalone said:

Is there nothing in the diagnostics or in the syslog (mirrored to USB flash drive) that might indicate what's happening?

I see a Diagnostics file but no syslog file from the Syslog Server.   The syslog in the diagnostics is a snapshot in time of everything that happens up to the point when you took it.   It also only records events which take place since the last restart.  To get a continuous syslog, you have to use the syslog server. 

 

Plus, your subject for this thread is somewhat misleading.  Your present problem is not with the UPS and/or the UPS software although the UPS may have triggered what is going on.  Your syslog is being spammed with entries like this:

 

Dec 15 20:02:36 Goathead emhttpd: shcmd (4580): rmdir /mnt/user
Dec 15 20:02:36 Goathead root: rmdir: failed to remove '/mnt/user': Device or resource busy
Dec 15 20:02:36 Goathead emhttpd: shcmd (4580): exit status: 1
Dec 15 20:02:36 Goathead emhttpd: shcmd (4582): rm -f /boot/config/plugins/dynamix/mover.cron
Dec 15 20:02:36 Goathead emhttpd: shcmd (4583): /usr/local/sbin/update_cron
Dec 15 20:02:36 Goathead emhttpd: Retry unmounting user share(s)...
Dec 15 20:02:41 Goathead emhttpd: shcmd (4584): /usr/sbin/zfs unmount -a
Dec 15 20:02:41 Goathead root: cannot unmount '/mnt/containers_cache/appdata': pool or dataset is busy
Dec 15 20:02:41 Goathead root: cannot unmount '/mnt/containers_cache': pool or dataset is busy
Dec 15 20:02:41 Goathead emhttpd: shcmd (4584): exit status: 1
Dec 15 20:02:41 Goathead emhttpd: shcmd (4585): umount /mnt/user
Dec 15 20:02:41 Goathead root: umount: /mnt/user: target is busy.
Dec 15 20:02:41 Goathead emhttpd: shcmd (4585): exit status: 32

 

I am going to ping @JorgeB who is a real Guru at diagnosing system issues and see if he sees anything.

 

PS:  Do you have a recent backup of your flash drive?

6 minutes ago, Frank1940 said:

I see a Diagnostics file but no syslog file from the Syslog Server.   The syslog in the diagnostics is a snapshot in time of everything that happens up to the point when you took it.   It also only records events which take place since the last restart.  To get a continuous syslog, you have to use the syslog server. 

 

Plus, your subject for this thread is somewhat misleading.  Your present problem is not with the UPS and/or the UPS software although the UPS may have triggered what is going on.  Your syslog is being spammed with entries like this:

 

Dec 15 20:02:36 Goathead emhttpd: shcmd (4580): rmdir /mnt/user
Dec 15 20:02:36 Goathead root: rmdir: failed to remove '/mnt/user': Device or resource busy
Dec 15 20:02:36 Goathead emhttpd: shcmd (4580): exit status: 1
Dec 15 20:02:36 Goathead emhttpd: shcmd (4582): rm -f /boot/config/plugins/dynamix/mover.cron
Dec 15 20:02:36 Goathead emhttpd: shcmd (4583): /usr/local/sbin/update_cron
Dec 15 20:02:36 Goathead emhttpd: Retry unmounting user share(s)...
Dec 15 20:02:41 Goathead emhttpd: shcmd (4584): /usr/sbin/zfs unmount -a
Dec 15 20:02:41 Goathead root: cannot unmount '/mnt/containers_cache/appdata': pool or dataset is busy
Dec 15 20:02:41 Goathead root: cannot unmount '/mnt/containers_cache': pool or dataset is busy
Dec 15 20:02:41 Goathead emhttpd: shcmd (4584): exit status: 1
Dec 15 20:02:41 Goathead emhttpd: shcmd (4585): umount /mnt/user
Dec 15 20:02:41 Goathead root: umount: /mnt/user: target is busy.
Dec 15 20:02:41 Goathead emhttpd: shcmd (4585): exit status: 32

 

I am going to ping @JorgeB who is a real Guru at diagnosing system issues and see if he sees anything.

 

PS:  Do you have a recent backup of your flash drive?

Oh no!  I just realized I forgot to save it and when I rebooted yesterday to see if it would alleviate the high CPU usage, I removed the syslog file that was mirrored to flash.  I'vew attached the previous one which is from just after the unclean shutdown on the 14th around 2130 CST.

 

Those errors above have been happening since the power outage.  All the arrs are either failing or very very slow.

 

The issue 

syslog-previous

First thing you need to do is to find out what is keeping the pool from unmounting:

 

Dec 15 19:02:20 Goathead root: cannot unmount '/mnt/containers_cache/appdata': pool or dataset is busy

 

It's something in the appdata share, start by retesting with the docker service disable, if that works, enable the service but try with just half of your containers enabled, if OK try the other half, then keep drilling down.

1 hour ago, JorgeB said:

First thing you need to do is to find out what is keeping the pool from unmounting:

 

Dec 15 19:02:20 Goathead root: cannot unmount '/mnt/containers_cache/appdata': pool or dataset is busy

 

It's something in the appdata share, start by retesting with the docker service disable, if that works, enable the service but try with just half of your containers enabled, if OK try the other half, then keep drilling down.

I paused all containers yesterday and started bringing them online one by one.  Performance improved, but as I would turn on each container performance would get progressively worse and CPU util kept rising.  I'm currently only running 12 our of 27 containers and can't see what is using CPU cycles as when I'm at 50% CPU util, if I total up CPU usage for each container they're only using 2-6% of available CPU on average with the exception of Plex doing a scan where is can take up to 50%, but only temporarily, and CPU usage overall goes to 70-80%.  So something is eating the cycles but there is no single container that gets started that seems to be the issue and I can 't see what is eating the cycles.

One additional point of note is that there does seem to be a slowdown on the HDD array as well.  I have a Plex server on another physical server that accesses files on the array and that also is so slow that most files won't play and it takes a long time to load any title.  I don't think that's touching the appdata share at all so it seems the entire system is slow.

So that is a new problem? The Thread title is not about that.

6 hours ago, JorgeB said:

So that is a new problem? The Thread title is not about that.

It's not new, but rather started exactly when the power cut out, as explained in my original post.

 

"even if I turn off all my VMs and containers performance is terrible"

Personally, I think the OP now has two problems.  The original one is probably in the UPS.  One thing I would be looking at is the actual storage capacity of the battery.  That server has a CPU with a 145W TDP and 19 drives in it.  IF that battery is more than about 18 months old, I suspect it can not power that server as it tries to shutdown and the output of the UPS goes bonky...

 

The second problem is the present running state of the server.  It is possible that the UPS wonky output corrupted something on the boot drive or in the Docker or VM configuration files.  

3 minutes ago, Frank1940 said:

Personally, I think the OP now has two problems.  The original one is probably in the UPS.  One thing I would be looking at is the actual storage capacity of the battery.  That server has a CPU with a 145W TDP and 19 drives in it.  IF that battery is more than about 18 months old, I suspect it can not power that server as it tries to shutdown and the output of the UPS goes bonky...

 

The second problem is the present running state of the server.  It is possible that the UPS wonky output corrupted something on the boot drive or in the Docker or VM configuration files.  

I don't think the UPS is underpowered as is 220v 3000va and even with both my primary and backup server running it's only at 40% load.  Also, the battery is 10 months old.

When I run ps, this is  the output.  the Nvidia driver seems to be pretty busy, but other than that, I can'tr see any immediate culprit causing the performance issues, and I can't understand how running on battery would cause this.

 

 1816 ?        00:00:09 kworker/40:1H-kblockd
 1841 ?        00:00:09 kworker/42:1H-kblockd
 1842 ?        00:00:23 kworker/66:1H-kblockd
 1844 ?        00:00:21 kworker/19:1H-kblockd
 1846 ?        00:00:22 kworker/64:1H-kblockd
 1847 ?        00:00:22 kworker/65:1H-kblockd
 1848 ?        00:00:22 kworker/62:1H-kblockd
 1850 ?        00:00:22 kworker/63:1H-kblockd
 1852 ?        00:00:22 kworker/57:1H-kblockd
 1855 ?        00:00:00 tpm_dev_wq
 1862 ?        00:00:22 kworker/33:1H-kblockd
 1865 ?        00:00:00 ipmi-msghandler
 1868 ?        00:00:22 kworker/61:1H-kblockd
 1871 ?        00:00:22 kworker/35:1H-kblockd
 1873 ?        00:00:00 nvme-wq
 1875 ?        00:00:00 nvme-reset-wq
 1876 ?        00:00:00 nvme-delete-wq
 1893 ?        00:00:11 kworker/50:1H-kblockd
 1896 ?        00:00:00 scsi_eh_1
 1897 ?        00:00:00 scsi_tmf_1
 1898 ?        00:00:00 scsi_eh_2
 1899 ?        00:00:00 scsi_tmf_2
 1904 ?        00:00:00 scsi_eh_3
 1905 ?        00:00:00 scsi_tmf_3
 1912 ?        00:00:00 scsi_eh_4
 1913 ?        00:00:00 scsi_tmf_4
 1921 ?        00:00:11 kworker/52:1H-kblockd
 1923 ?        00:00:16 kworker/45:1H-kblockd
 1940 ?        00:00:21 kworker/58:1H-kblockd
 1944 ?        00:00:01 hwrng
 1945 ?        00:00:00 scsi_eh_5
 1946 ?        00:00:00 scsi_tmf_5
 1947 ?        00:00:00 scsi_eh_6
 1948 ?        00:00:00 scsi_tmf_6
 1950 ?        00:00:00 scsi_eh_7
 1952 ?        00:00:00 scsi_tmf_7
 1953 ?        00:00:00 scsi_eh_8
 1954 ?        00:00:00 scsi_tmf_8
 1955 ?        00:00:00 scsi_eh_9
 1956 ?        00:00:00 scsi_tmf_9
 1957 ?        00:00:00 scsi_eh_10
 1958 ?        00:00:00 scsi_tmf_10
 1959 ?        00:00:00 scsi_eh_11
 1960 ?        00:00:00 scsi_tmf_11
 1969 ?        00:07:24 kipmi0
 1972 ?        00:00:00 cryptd
 1975 ?        00:00:11 kworker/47:1H-kblockd
 2003 ?        00:00:23 kworker/59:1H-kblockd
 2018 ?        00:00:00 spl_system_task
 2019 ?        00:00:00 spl_delay_taskq
 2020 ?        00:00:56 spl_dynamic_tas
 2021 ?        00:00:01 spl_kmem_cache
 2023 ?        00:00:00 nv_queue
 2024 ?        00:00:00 nv_queue
 2025 ?        05:13:54 nv_open_q
 2030 ?        00:00:00 zvol
 2031 ?        00:00:00 arc_prune
 2032 ?        00:00:08 arc_evict
 2033 ?        00:00:02 arc_reap
 2034 ?        00:00:01 dbu_evict
 2035 ?        00:00:33 dbuf_evict
 2036 ?        00:00:00 nvidia-modeset/kthread_q
 2037 ?        00:00:00 nvidia-modeset/deferred_close_kthread_q
 2040 ?        00:00:00 z_vdev_file
 2041 ?        00:00:01 l2arc_feed
 2057 ?        00:00:00 poll_mpt3sas0_s
 2143 ?        00:00:00 kworker/u145:2-btrfs-endio
 2317 ?        00:00:28 kworker/71:0-events
 2495 ?        00:00:00 ixgbe
 2692 ?        00:00:27 kworker/23:0-events
 2777 ?        00:00:23 containerd-shim
 2825 ?        00:00:00 kworker/5:2-events
 2828 ?        00:00:00 dumb-init
 2837 ?        00:00:02 dhcpcd
 2838 ?        00:00:00 dhcpcd
 2839 ?        00:00:00 dhcpcd
 2854 ?        00:00:00 dhcpcd
 2893 ?        00:00:00 kworker/22:0-events
 2905 ?        00:00:00 dhcpcd
 2984 ?        00:00:01 dbus-daemon
 2995 ?        00:00:00 elogind-daemon
 3083 ?        00:00:10 ntpd
 3090 ?        00:00:00 acpid
 3105 ?        00:00:11 supervisord
 3108 ?        00:00:03 crond
 3112 ?        00:00:00 atd
 3203 ?        00:00:00 mcelog
 3298 ?        00:00:04 monitor_nchan
 3536 ?        00:00:00 start.sh
 3538 ?        00:00:00 bash
 3540 ?        00:35:06 Plex Media Serv
 3603 ?        00:00:49 kworker/7:2-events
 4521 ?        00:00:24 kworker/27:2-events
 4639 ?        00:00:00 kworker/33:2-events
 5346 ?        00:00:00 atd
 5348 ?        00:00:00 sh
 5350 ?        00:00:00 sh
 5351 ?        00:00:00 inotifywait
 5500 ?        00:00:00 ipmiseld
 5756 ?        00:00:23 smbd
 5761 ?        00:00:00 smbd-notifyd
 5762 ?        00:00:00 cleanupd
 5769 ?        00:00:00 wsdd2
 5772 ?        00:00:04 winbindd
 5774 ?        00:00:01 winbindd
 5822 ?        00:01:39 avahi-daemon
 5823 ?        00:00:00 avahi-daemon
 5831 ?        00:00:00 avahi-dnsconfd
 6147 ?        00:11:08 smbd
 6148 ?        00:00:00 winbindd
 6216 ?        00:00:00 ipmitail
 6229 ?        00:00:00 tail
 6231 ?        00:00:00 ipmitail
 6326 ?        00:00:32 kworker/9:2-events
 6443 ?        00:00:15 kworker/27:0-events
 6501 ?        00:00:00 kworker/50:1-events
 7607 ?        00:00:00 kworker/u144:2-poll_mpt3sas0_statu
 8755 ?        1-23:07:55 deluged
 8952 ?        00:00:00 kworker/u145:3-events_unbound
 9275 ?        00:00:44 kworker/53:2-events
 9548 tty1     00:00:00 agetty
 9549 tty2     00:00:00 agetty
 9550 tty3     00:00:00 agetty
 9551 tty4     00:00:00 agetty
 9552 tty5     00:00:00 agetty
 9553 tty6     00:00:00 agetty
 9614 ?        00:00:00 sshd
 9622 ?        00:00:00 inetd
 9623 ?        00:18:45 emhttpd
 9632 ?        00:00:00 md
 9633 ?        00:00:00 mdrecoveryd
 9682 ?        00:00:31 apcupsd
 9724 ?        00:00:04 kworker/60:2-events
 9872 ?        00:00:00 rpciod
 9873 ?        00:00:00 xprtiod
 9911 ?        00:00:38 rpcbind
 9915 ?        00:00:00 rpc.statd
 9923 ?        00:00:00 lockd
10074 ?        00:01:10 nfsd
10075 ?        00:01:11 nfsd
10076 ?        00:01:11 nfsd
10077 ?        00:01:11 nfsd
10078 ?        00:01:11 nfsd
10079 ?        00:01:11 nfsd
10080 ?        00:01:12 nfsd
10081 ?        00:01:18 nfsd
10083 ?        00:00:07 rpc.mountd
10157 ?        00:00:14 php-fpm
10258 ?        00:00:00 nginx
10259 ?        00:09:12 nginx
10352 ?        00:48:31 unraid-api
10365 ?        00:00:00 rc.flash_backup
10594 ?        00:00:14 kworker/28:2-events
11384 ?        00:00:00 kworker/54:0-events
13201 ?        00:00:00 sh
13208 ?        00:00:00 php
13379 ?        00:00:00 kworker/6:2-events
13427 ?        00:00:16 kworker/68:1-events
13594 ?        00:00:00 ttyd
13688 pts/1    00:00:00 bash
14027 ?        00:00:00 kworker/30:1-events
14742 ?        00:00:39 kworker/12:2-events
15021 ?        00:00:41 notify_poller
15024 ?        00:00:11 session_check
15026 ?        00:10:37 device_list
15028 ?        00:04:10 disk_load
15031 ?        00:05:00 parity_list
15211 ?        00:01:02 kworker/4:0-events
15284 ?        00:00:00 kworker/44:0-events
15495 ?        00:00:11 kworker/57:2-events
15628 ?        00:00:00 kworker/12:1-events
15862 ?        00:00:00 unraidd0
15863 ?        00:32:45 unraidd1
15864 ?        00:31:00 unraidd2
15865 ?        00:34:01 unraidd3
15866 ?        02:25:04 unraidd4
15867 ?        01:11:17 unraidd5
15868 ?        01:44:30 unraidd6
15869 ?        03:07:50 unraidd7
15870 ?        04:26:48 unraidd8
15871 ?        00:00:00 unraidd9
15929 ?        00:00:00 xfsalloc
15930 ?        00:00:00 xfs_mru_cache
15932 ?        00:00:00 xfs-buf/md1p1
15933 ?        00:00:00 xfs-conv/md1p1
15934 ?        00:00:00 xfs-reclaim/md1
15935 ?        00:00:00 xfs-blockgc/md1
15936 ?        00:00:00 xfs-inodegc/md1
15937 ?        00:00:00 xfs-log/md1p1
15938 ?        00:00:00 xfs-cil/md1p1
15939 ?        00:00:23 xfsaild/md1p1
15950 ?        00:00:00 xfs-buf/md2p1
15951 ?        00:00:00 xfs-conv/md2p1
15952 ?        00:00:00 xfs-reclaim/md2
15953 ?        00:00:00 xfs-blockgc/md2
15954 ?        00:00:00 xfs-inodegc/md2
15955 ?        00:00:00 xfs-log/md2p1
15956 ?        00:00:00 xfs-cil/md2p1
15957 ?        00:00:01 xfsaild/md2p1
16018 ?        00:00:00 xfs-buf/md3p1
16019 ?        00:00:00 xfs-conv/md3p1
16020 ?        00:00:00 xfs-reclaim/md3
16021 ?        00:00:00 xfs-blockgc/md3
16022 ?        00:00:00 xfs-inodegc/md3
16023 ?        00:00:00 xfs-log/md3p1
16024 ?        00:00:00 xfs-cil/md3p1
16025 ?        00:00:21 xfsaild/md3p1
16029 ?        00:00:00 sh
16030 ?        00:05:28 tar
16031 ?        01:06:39 gzip
16039 ?        00:00:00 xfs-buf/md4p1
16040 ?        00:00:00 xfs-conv/md4p1
16041 ?        00:00:00 xfs-reclaim/md4
16042 ?        00:00:00 xfs-blockgc/md4
16043 ?        00:00:00 xfs-inodegc/md4
16044 ?        00:00:00 xfs-log/md4p1
16045 ?        00:00:00 xfs-cil/md4p1
16046 ?        00:00:06 xfsaild/md4p1
16061 ?        00:00:00 xfs-buf/md5p1
16062 ?        00:00:00 xfs-conv/md5p1
16063 ?        00:00:00 xfs-reclaim/md5
16064 ?        00:00:00 xfs-blockgc/md5
16065 ?        00:00:00 xfs-inodegc/md5
16066 ?        00:00:00 xfs-log/md5p1
16067 ?        00:00:00 xfs-cil/md5p1
16068 ?        00:00:00 xfsaild/md5p1
16078 ?        00:00:00 xfs-buf/md6p1
16079 ?        00:00:00 xfs-conv/md6p1
16080 ?        00:00:00 xfs-reclaim/md6
16081 ?        00:00:00 xfs-blockgc/md6
16082 ?        00:00:00 xfs-inodegc/md6
16083 ?        00:00:00 xfs-log/md6p1
16084 ?        00:00:00 xfs-cil/md6p1
16085 ?        00:00:00 xfsaild/md6p1
16095 ?        00:00:00 xfs-buf/md7p1
16096 ?        00:00:00 xfs-conv/md7p1
16097 ?        00:00:00 xfs-reclaim/md7
16098 ?        00:00:00 xfs-blockgc/md7
16099 ?        00:00:00 xfs-inodegc/md7
16100 ?        00:00:00 xfs-log/md7p1
16101 ?        00:00:00 xfs-cil/md7p1
16102 ?        00:00:00 xfsaild/md7p1
16113 ?        00:00:00 xfs-buf/md8p1
16114 ?        00:00:00 xfs-conv/md8p1
16115 ?        00:00:00 xfs-reclaim/md8
16116 ?        00:00:00 xfs-blockgc/md8
16117 ?        00:00:00 xfs-inodegc/md8
16118 ?        00:00:00 xfs-log/md8p1
16119 ?        00:00:00 xfs-cil/md8p1
16120 ?        00:00:37 xfsaild/md8p1
16130 ?        00:00:00 xfs-buf/md9p1
16131 ?        00:00:00 xfs-conv/md9p1
16132 ?        00:00:00 xfs-reclaim/md9
16133 ?        00:00:00 xfs-blockgc/md9
16134 ?        00:00:00 xfs-inodegc/md9
16135 ?        00:00:00 xfs-log/md9p1
16136 ?        00:00:00 xfs-cil/md9p1
16137 ?        00:00:00 xfsaild/md9p1
16347 ?        00:00:05 kworker/34:0-events
16351 ?        00:00:35 kworker/37:2-events
16505 ?        00:00:19 z_null_iss
16506 ?        00:00:04 z_null_int
16507 ?        00:00:00 z_rd_iss
16508 ?        00:00:14 z_rd_int_0
16509 ?        00:00:14 z_rd_int_1
16510 ?        00:00:14 z_rd_int_2
16511 ?        00:00:14 z_rd_int_3
16512 ?        00:00:14 z_rd_int_4
16513 ?        00:00:14 z_rd_int_5
16514 ?        00:00:14 z_rd_int_6
16515 ?        00:01:49 z_wr_iss
16516 ?        00:01:06 z_wr_iss_h
16517 ?        00:01:20 z_wr_int_0
16518 ?        00:01:20 z_wr_int_1
16519 ?        00:01:20 z_wr_int_2
16520 ?        00:01:20 z_wr_int_3
16521 ?        00:01:20 z_wr_int_4
16522 ?        00:01:20 z_wr_int_5
16523 ?        00:01:20 z_wr_int_6
16524 ?        00:00:40 z_wr_int_h
16525 ?        00:00:00 z_fr_iss_0
16526 ?        00:00:00 z_fr_iss_1
16527 ?        00:00:00 z_fr_iss_2
16528 ?        00:00:00 z_fr_iss_3
16529 ?        00:00:00 z_fr_iss_4
16530 ?        00:00:00 z_fr_iss_5
16531 ?        00:00:00 z_fr_iss_6
16532 ?        00:00:00 z_fr_int
16533 ?        00:00:00 z_cl_iss
16534 ?        00:00:00 z_cl_int
16535 ?        00:00:00 z_ioctl_iss
16536 ?        00:00:18 z_ioctl_int
16537 ?        00:00:24 z_trim_iss
16538 ?        00:00:54 z_trim_int
16539 ?        00:00:00 z_zvol
16540 ?        00:00:00 z_prefetch
16541 ?        00:00:00 z_upgrade
16546 ?        00:00:11 dp_sync_taskq
16547 ?        00:00:11 dp_sync_taskq
16548 ?        00:00:10 dp_sync_taskq
16549 ?        00:00:11 dp_sync_taskq
16550 ?        00:00:11 dp_sync_taskq
16551 ?        00:00:10 dp_sync_taskq
16552 ?        00:00:10 dp_sync_taskq
16553 ?        00:00:11 dp_sync_taskq
16554 ?        00:00:11 dp_sync_taskq
16555 ?        00:00:10 dp_sync_taskq
16556 ?        00:00:10 dp_sync_taskq
16557 ?        00:00:11 dp_sync_taskq
16558 ?        00:00:10 dp_sync_taskq
16559 ?        00:00:11 dp_sync_taskq
16560 ?        00:00:11 dp_sync_taskq
16561 ?        00:00:11 dp_sync_taskq
16562 ?        00:00:11 dp_sync_taskq
16563 ?        00:00:10 dp_sync_taskq
16564 ?        00:00:11 dp_sync_taskq
16565 ?        00:00:10 dp_sync_taskq
16566 ?        00:00:11 dp_sync_taskq
16567 ?        00:00:10 dp_sync_taskq
16568 ?        00:00:11 dp_sync_taskq
16569 ?        00:00:11 dp_sync_taskq
16570 ?        00:00:11 dp_sync_taskq
16571 ?        00:00:10 dp_sync_taskq
16572 ?        00:00:11 dp_sync_taskq
16573 ?        00:00:11 dp_sync_taskq
16574 ?        00:00:11 dp_sync_taskq
16575 ?        00:00:10 dp_sync_taskq
16576 ?        00:00:10 dp_sync_taskq
16577 ?        00:00:11 dp_sync_taskq
16578 ?        00:00:11 dp_sync_taskq
16579 ?        00:00:11 dp_sync_taskq
16580 ?        00:00:11 dp_sync_taskq
16581 ?        00:00:10 dp_sync_taskq
16582 ?        00:00:10 dp_sync_taskq
16583 ?        00:00:11 dp_sync_taskq
16584 ?        00:00:10 dp_sync_taskq
16585 ?        00:00:11 dp_sync_taskq
16586 ?        00:00:11 dp_sync_taskq
16587 ?        00:00:10 dp_sync_taskq
16588 ?        00:00:11 dp_sync_taskq
16589 ?        00:00:11 dp_sync_taskq
16590 ?        00:00:10 dp_sync_taskq
16591 ?        00:00:11 dp_sync_taskq
16592 ?        00:00:11 dp_sync_taskq
16593 ?        00:00:11 dp_sync_taskq
16594 ?        00:00:10 dp_sync_taskq
16595 ?        00:00:10 dp_sync_taskq
16596 ?        00:00:10 dp_sync_taskq
16597 ?        00:00:10 dp_sync_taskq
16598 ?        00:00:11 dp_sync_taskq
16599 ?        00:00:11 dp_sync_taskq
16600 ?        00:00:00 dp_zil_clean_ta
16601 ?        00:00:00 dp_zil_clean_ta
16602 ?        00:00:00 dp_zil_clean_ta
16603 ?        00:00:00 dp_zil_clean_ta
16604 ?        00:00:00 dp_zil_clean_ta
16605 ?        00:00:00 dp_zil_clean_ta
16606 ?        00:00:00 dp_zil_clean_ta
16607 ?        00:00:00 dp_zil_clean_ta
16608 ?        00:00:00 dp_zil_clean_ta
16609 ?        00:00:00 dp_zil_clean_ta
16610 ?        00:00:00 dp_zil_clean_ta
16611 ?        00:00:00 dp_zil_clean_ta
16612 ?        00:00:00 dp_zil_clean_ta
16613 ?        00:00:00 dp_zil_clean_ta
16614 ?        00:00:00 dp_zil_clean_ta
16615 ?        00:00:00 dp_zil_clean_ta
16616 ?        00:00:00 dp_zil_clean_ta
16617 ?        00:00:00 dp_zil_clean_ta
16618 ?        00:00:00 dp_zil_clean_ta
16619 ?        00:00:00 dp_zil_clean_ta
16620 ?        00:00:00 dp_zil_clean_ta
16621 ?        00:00:00 dp_zil_clean_ta
16622 ?        00:00:00 dp_zil_clean_ta
16623 ?        00:00:00 dp_zil_clean_ta
16624 ?        00:00:00 dp_zil_clean_ta
16625 ?        00:00:00 dp_zil_clean_ta
16626 ?        00:00:00 dp_zil_clean_ta
16627 ?        00:00:00 dp_zil_clean_ta
16628 ?        00:00:00 dp_zil_clean_ta
16629 ?        00:00:00 dp_zil_clean_ta
16630 ?        00:00:00 dp_zil_clean_ta
16631 ?        00:00:00 dp_zil_clean_ta
16632 ?        00:00:00 dp_zil_clean_ta
16633 ?        00:00:00 dp_zil_clean_ta
16634 ?        00:00:00 dp_zil_clean_ta
16635 ?        00:00:00 dp_zil_clean_ta
16636 ?        00:00:00 dp_zil_clean_ta
16637 ?        00:00:00 dp_zil_clean_ta
16638 ?        00:00:00 dp_zil_clean_ta
16639 ?        00:00:00 dp_zil_clean_ta
16640 ?        00:00:00 dp_zil_clean_ta
16641 ?        00:00:00 dp_zil_clean_ta
16642 ?        00:00:00 dp_zil_clean_ta
16643 ?        00:00:00 dp_zil_clean_ta
16644 ?        00:00:00 dp_zil_clean_ta
16645 ?        00:00:00 dp_zil_clean_ta
16646 ?        00:00:00 dp_zil_clean_ta
16647 ?        00:00:00 dp_zil_clean_ta
16648 ?        00:00:00 dp_zil_clean_ta
16649 ?        00:00:00 dp_zil_clean_ta
16650 ?        00:00:00 dp_zil_clean_ta
16651 ?        00:00:00 dp_zil_clean_ta
16652 ?        00:00:00 dp_zil_clean_ta
16653 ?        00:00:00 dp_zil_clean_ta
16654 ?        00:00:00 dp_zil_clean_ta
16655 ?        00:00:00 dp_zil_clean_ta
16656 ?        00:00:00 dp_zil_clean_ta
16657 ?        00:00:00 dp_zil_clean_ta
16658 ?        00:00:00 dp_zil_clean_ta
16659 ?        00:00:00 dp_zil_clean_ta
16660 ?        00:00:00 dp_zil_clean_ta
16661 ?        00:00:00 dp_zil_clean_ta
16662 ?        00:00:00 dp_zil_clean_ta
16663 ?        00:00:00 dp_zil_clean_ta
16664 ?        00:00:00 dp_zil_clean_ta
16665 ?        00:00:00 dp_zil_clean_ta
16666 ?        00:00:00 dp_zil_clean_ta
16667 ?        00:00:00 dp_zil_clean_ta
16668 ?        00:00:00 dp_zil_clean_ta
16669 ?        00:00:00 dp_zil_clean_ta
16670 ?        00:00:00 dp_zil_clean_ta
16671 ?        00:00:00 dp_zil_clean_ta
16672 ?        00:00:00 z_zrele
16673 ?        00:00:00 z_unlinked_drai
16677 ?        00:00:03 metaslab_group_
16678 ?        00:00:01 metaslab_group_
16757 ?        00:00:07 txg_quiesce
16758 ?        00:06:18 txg_sync
16759 ?        00:00:02 mmp
16835 ?        00:00:00 z_indirect_cond
16836 ?        00:00:00 z_livelist_dest
16837 ?        00:00:00 z_livelist_cond
16838 ?        00:00:00 z_checkpoint_di
16839 ?        00:00:37 vdev_autotrim
17070 ?        00:00:00 kworker/13:1-events
17515 ?        00:00:00 z_null_iss
17516 ?        00:00:00 z_null_int
17517 ?        00:00:00 z_rd_iss
17518 ?        00:00:00 z_rd_int_0
17519 ?        00:00:00 z_rd_int_1
17520 ?        00:00:00 z_rd_int_2
17521 ?        00:00:00 z_rd_int_3
17522 ?        00:00:00 z_rd_int_4
17523 ?        00:00:00 z_rd_int_5
17524 ?        00:00:00 z_rd_int_6
17525 ?        00:00:00 z_wr_iss
17526 ?        00:00:00 z_wr_iss_h
17527 ?        00:00:00 z_wr_int_0
17528 ?        00:00:00 z_wr_int_1
17529 ?        00:00:00 z_wr_int_2
17530 ?        00:00:00 z_wr_int_3
17531 ?        00:00:00 z_wr_int_4
17532 ?        00:00:00 z_wr_int_5
17533 ?        00:00:00 z_wr_int_6
17534 ?        00:00:00 z_wr_int_h
17535 ?        00:00:00 z_fr_iss_0
17536 ?        00:00:00 z_fr_iss_1
17537 ?        00:00:00 z_fr_iss_2
17538 ?        00:00:00 z_fr_iss_3
17539 ?        00:00:00 z_fr_iss_4
17540 ?        00:00:00 z_fr_iss_5
17541 ?        00:00:00 z_fr_iss_6
17542 ?        00:00:00 z_fr_int
17543 ?        00:00:00 z_cl_iss
17544 ?        00:00:00 z_cl_int
17545 ?        00:00:00 z_ioctl_iss
17546 ?        00:00:00 z_ioctl_int
17547 ?        00:00:00 z_trim_iss
17548 ?        00:00:00 z_trim_int
17549 ?        00:00:00 z_zvol
17550 ?        00:00:00 z_prefetch
17551 ?        00:00:00 z_upgrade
17554 ?        00:00:00 dp_sync_taskq
17555 ?        00:00:00 dp_sync_taskq
17556 ?        00:00:00 dp_sync_taskq
17557 ?        00:00:00 dp_sync_taskq
17558 ?        00:00:00 dp_sync_taskq
17559 ?        00:00:00 dp_sync_taskq
17560 ?        00:00:00 dp_sync_taskq
17561 ?        00:00:00 dp_sync_taskq
17562 ?        00:00:00 dp_sync_taskq
17563 ?        00:00:00 dp_sync_taskq
17564 ?        00:00:00 dp_sync_taskq
17565 ?        00:00:00 dp_sync_taskq
17566 ?        00:00:00 dp_sync_taskq
17567 ?        00:00:00 dp_sync_taskq
17568 ?        00:00:00 dp_sync_taskq
17569 ?        00:00:00 dp_sync_taskq
17570 ?        00:00:00 dp_sync_taskq
17571 ?        00:00:00 dp_sync_taskq
17572 ?        00:00:00 dp_sync_taskq
17573 ?        00:00:00 dp_sync_taskq
17574 ?        00:00:00 dp_sync_taskq
17575 ?        00:00:00 dp_sync_taskq
17576 ?        00:00:00 dp_sync_taskq
17577 ?        00:00:00 dp_sync_taskq
17578 ?        00:00:00 dp_sync_taskq
17579 ?        00:00:00 dp_sync_taskq
17580 ?        00:00:00 dp_sync_taskq
17581 ?        00:00:00 dp_sync_taskq
17582 ?        00:00:00 dp_sync_taskq
17583 ?        00:00:00 dp_sync_taskq
17584 ?        00:00:00 dp_sync_taskq
17585 ?        00:00:00 dp_sync_taskq
17586 ?        00:00:00 dp_sync_taskq
17587 ?        00:00:00 dp_sync_taskq
17588 ?        00:00:00 dp_sync_taskq
17589 ?        00:00:00 dp_sync_taskq
17590 ?        00:00:00 dp_sync_taskq
17591 ?        00:00:00 dp_sync_taskq
17592 ?        00:00:00 dp_sync_taskq
17593 ?        00:00:00 dp_sync_taskq
17594 ?        00:00:00 dp_sync_taskq
17595 ?        00:00:00 dp_sync_taskq
17596 ?        00:00:00 dp_sync_taskq
17597 ?        00:00:00 dp_sync_taskq
17598 ?        00:00:00 dp_sync_taskq
17599 ?        00:00:00 dp_sync_taskq
17600 ?        00:00:00 dp_sync_taskq
17601 ?        00:00:00 dp_sync_taskq
17602 ?        00:00:00 dp_sync_taskq
17603 ?        00:00:00 dp_sync_taskq
17604 ?        00:00:00 dp_sync_taskq
17605 ?        00:00:00 dp_sync_taskq
17606 ?        00:00:00 dp_sync_taskq
17607 ?        00:00:00 dp_sync_taskq
17608 ?        00:00:00 dp_zil_clean_ta
17609 ?        00:00:00 dp_zil_clean_ta
17610 ?        00:00:00 dp_zil_clean_ta
17611 ?        00:00:00 dp_zil_clean_ta
17612 ?        00:00:00 dp_zil_clean_ta
17613 ?        00:00:00 dp_zil_clean_ta
17614 ?        00:00:00 dp_zil_clean_ta
17615 ?        00:00:00 dp_zil_clean_ta
17616 ?        00:00:00 dp_zil_clean_ta
17617 ?        00:00:00 dp_zil_clean_ta
17618 ?        00:00:00 dp_zil_clean_ta
17619 ?        00:00:00 dp_zil_clean_ta
17620 ?        00:00:00 dp_zil_clean_ta
17621 ?        00:00:00 dp_zil_clean_ta
17622 ?        00:00:00 dp_zil_clean_ta
17623 ?        00:00:00 dp_zil_clean_ta
17624 ?        00:00:00 dp_zil_clean_ta
17625 ?        00:00:00 dp_zil_clean_ta
17626 ?        00:00:00 dp_zil_clean_ta
17627 ?        00:00:00 dp_zil_clean_ta
17628 ?        00:00:00 dp_zil_clean_ta
17629 ?        00:00:00 dp_zil_clean_ta
17630 ?        00:00:00 dp_zil_clean_ta
17631 ?        00:00:00 dp_zil_clean_ta
17632 ?        00:00:00 dp_zil_clean_ta
17633 ?        00:00:00 dp_zil_clean_ta
17634 ?        00:00:00 dp_zil_clean_ta
17635 ?        00:00:00 dp_zil_clean_ta
17636 ?        00:00:00 dp_zil_clean_ta
17637 ?        00:00:00 dp_zil_clean_ta
17638 ?        00:00:00 dp_zil_clean_ta
17639 ?        00:00:00 dp_zil_clean_ta
17640 ?        00:00:00 dp_zil_clean_ta
17641 ?        00:00:00 dp_zil_clean_ta
17642 ?        00:00:00 dp_zil_clean_ta
17643 ?        00:00:00 dp_zil_clean_ta
17644 ?        00:00:00 dp_zil_clean_ta
17645 ?        00:00:00 dp_zil_clean_ta
17646 ?        00:00:00 dp_zil_clean_ta
17647 ?        00:00:00 dp_zil_clean_ta
17648 ?        00:00:00 dp_zil_clean_ta
17649 ?        00:00:00 dp_zil_clean_ta
17650 ?        00:00:00 dp_zil_clean_ta
17651 ?        00:00:00 dp_zil_clean_ta
17652 ?        00:00:00 dp_zil_clean_ta
17653 ?        00:00:00 dp_zil_clean_ta
17654 ?        00:00:00 dp_zil_clean_ta
17655 ?        00:00:00 dp_zil_clean_ta
17656 ?        00:00:00 dp_zil_clean_ta
17657 ?        00:00:00 dp_zil_clean_ta
17658 ?        00:00:00 dp_zil_clean_ta
17659 ?        00:00:00 dp_zil_clean_ta
17660 ?        00:00:00 dp_zil_clean_ta
17661 ?        00:00:00 dp_zil_clean_ta
17662 ?        00:00:00 dp_zil_clean_ta
17663 ?        00:00:00 dp_zil_clean_ta
17664 ?        00:00:00 dp_zil_clean_ta
17665 ?        00:00:00 dp_zil_clean_ta
17666 ?        00:00:00 dp_zil_clean_ta
17667 ?        00:00:00 dp_zil_clean_ta
17668 ?        00:00:00 dp_zil_clean_ta
17669 ?        00:00:00 dp_zil_clean_ta
17670 ?        00:00:00 dp_zil_clean_ta
17671 ?        00:00:00 dp_zil_clean_ta
17672 ?        00:00:00 dp_zil_clean_ta
17673 ?        00:00:00 dp_zil_clean_ta
17674 ?        00:00:00 dp_zil_clean_ta
17675 ?        00:00:00 dp_zil_clean_ta
17676 ?        00:00:00 dp_zil_clean_ta
17677 ?        00:00:00 dp_zil_clean_ta
17678 ?        00:00:00 dp_zil_clean_ta
17679 ?        00:00:00 dp_zil_clean_ta
17680 ?        00:00:00 z_zrele
17681 ?        00:00:00 z_unlinked_drai
17685 ?        00:00:00 metaslab_group_
17686 ?        00:00:00 metaslab_group_
17706 ?        00:00:01 txg_quiesce
17707 ?        00:00:03 txg_sync
17708 ?        00:00:02 mmp
17709 ?        00:00:00 z_indirect_cond
17710 ?        00:00:00 z_livelist_dest
17711 ?        00:00:00 z_livelist_cond
17712 ?        00:00:00 z_checkpoint_di
17713 ?        00:00:00 vdev_autotrim
17736 ?        00:00:00 metaslab_group_
17737 ?        00:00:00 metaslab_group_
17738 ?        00:00:00 metaslab_group_
17801 ?        00:00:00 metaslab_group_
17823 ?        00:00:00 metaslab_group_
17853 ?        00:00:00 z_upgrade
17890 ?        00:00:01 shfs
17900 ?        1-08:25:59 shfs
17977 ?        00:00:00 kworker/64:2-events
18110 ?        00:03:18 rsyslogd
18597 ?        00:00:01 kworker/63:1-events
19100 ?        00:00:11 kworker/15:1-events
19213 ?        00:00:00 btrfs-worker
19214 ?        00:00:00 kworker/u147:0
19215 ?        00:00:00 kworker/u148:0-btrfs-worker-high
19216 ?        00:00:00 kworker/u149:0-btrfs-worker-high
19218 ?        00:00:00 btrfs-worker-hi
19219 ?        00:00:00 btrfs-delalloc
19220 ?        00:00:00 btrfs-flush_del
19221 ?        00:00:00 btrfs-cache
19222 ?        00:00:00 btrfs-fixup
19223 ?        00:00:00 btrfs-endio
19224 ?        00:00:00 btrfs-endio-met
19225 ?        00:00:00 btrfs-endio-rai
19226 ?        00:00:00 btrfs-rmw
19227 ?        00:00:00 btrfs-endio-wri
19228 ?        00:00:00 btrfs-compresse
19229 ?        00:00:00 btrfs-freespace
19230 ?        00:00:00 btrfs-delayed-m
19231 ?        00:00:00 btrfs-qgroup-re
19233 ?        00:00:00 btrfs-cleaner
19234 ?        00:00:21 btrfs-transaction
19250 pts/2    00:00:00 bash
19276 ?        01:21:22 dockerd
19305 ?        00:00:00 kworker/17:1-events
19494 ?        00:00:00 kworker/42:1-events
19498 ?        00:00:01 kworker/37:1-events
19598 ?        00:00:00 sleep
19830 ?        00:05:36 containerd
19995 ?        00:00:17 kworker/59:0-md
20597 ?        00:00:00 php-fpm
20901 ?        00:00:00 sleep
21039 ?        00:00:06 kworker/14:0-events
21084 ?        00:00:04 kworker/10:2-events
21145 ?        00:00:00 kworker/38:0-events
21223 ?        00:00:00 docker-proxy
21230 ?        00:00:00 btrfs-worker
21231 ?        00:00:00 btrfs-worker-hi
21232 ?        00:00:00 btrfs-delalloc
21233 ?        00:00:00 btrfs-flush_del
21234 ?        00:00:00 btrfs-cache
21235 ?        00:00:00 btrfs-fixup
21236 ?        00:00:00 btrfs-endio
21237 ?        00:00:00 btrfs-endio-met
21238 ?        00:00:00 btrfs-endio-rai
21240 ?        00:00:00 btrfs-rmw
21241 ?        00:00:00 btrfs-endio-wri
21242 ?        00:00:00 btrfs-compresse
21243 ?        00:00:00 btrfs-freespace
21244 ?        00:00:00 btrfs-delayed-m
21245 ?        00:00:00 btrfs-qgroup-re
21246 ?        00:00:00 btrfs-cleaner
21247 ?        00:00:00 btrfs-transaction
21260 ?        00:00:00 virtlockd
21267 ?        00:00:00 docker-proxy
21281 ?        00:00:00 virtlogd
21344 ?        00:00:11 libvirtd
21456 ?        00:00:34 containerd-shim
21478 ?        00:00:03 dnsmasq
21479 ?        00:00:00 dnsmasq
21481 ?        00:00:00 s6-svscan
21483 ?        00:00:00 kworker/18:0-events
21633 ?        00:00:00 s6-supervise
21635 ?        00:00:00 s6-linux-init-s
21717 ?        00:00:00 s6-supervise
21718 ?        00:00:00 s6-supervise
21719 ?        00:00:00 s6-supervise
21720 ?        00:00:00 s6-supervise
21739 ?        00:00:00 s6-ipcserverd
21963 ?        00:00:34 kworker/46:1-events
21966 ?        00:00:24 kworker/45:3-events
22013 ?        00:00:00 busybox
22014 ?        00:00:05 python3
22054 ?        00:00:00 sleep
22179 ?        00:00:00 kworker/53:1-events
22306 ?        00:08:14 python3
22350 ?        00:00:00 UVM global queue
22351 ?        00:00:00 UVM deferred release queue
22352 ?        00:00:00 UVM Tools Event Queue
22376 ?        5-00:54:59 qemu-system-x86
22413 ?        00:00:03 kvm-nx-lpage-recovery-22376
22434 ?        00:00:20 kvm-pit/22376
22455 ?        00:00:22 kworker/48:0-events
23081 ?        00:00:00 docker-proxy
23090 ?        00:00:00 docker-proxy
23105 ?        00:00:00 docker-proxy
23110 ?        00:00:00 docker-proxy
23131 ?        00:00:00 docker-proxy
23136 ?        00:00:00 docker-proxy
23152 ?        00:01:04 kworker/36:1-events
23153 ?        00:00:00 docker-proxy
23160 ?        00:00:00 docker-proxy
23176 ?        00:00:02 docker-proxy
23185 ?        00:00:00 docker-proxy
23226 ?        00:00:35 containerd-shim
23247 ?        00:00:00 dumb-init
23276 ?        00:00:22 supervisord
23313 ?        00:00:00 kworker/9:0-events
23342 ?        00:00:00 php-fpm
23451 ?        00:00:00 docker-proxy
23459 ?        00:00:00 docker-proxy
23474 ?        00:00:00 docker-proxy
23481 ?        00:00:00 docker-proxy
23512 ?        00:00:33 containerd-shim
23536 ?        00:00:26 cinit
23570 ?        00:00:00 nc
23897 ?        00:00:00 kworker/43:2-events
23977 ?        00:00:00 docker-proxy
23988 ?        00:00:00 docker-proxy
24124 ?        00:00:33 containerd-shim
24199 ?        00:00:00 dumb-init
24208 ?        00:00:00 kworker/u144:1
24372 ?        00:00:18 supervisord
24377 ?        00:00:16 kworker/33:1-events
24706 ?        00:00:00 php-fpm
25334 ?        00:00:00 docker-proxy
25340 ?        00:00:00 docker-proxy
25390 ?        00:00:38 containerd-shim
25424 ?        00:00:00 sh
25982 ?        00:00:09 kworker/21:0-events
26095 ?        00:00:00 php-fpm
26889 ?        00:01:59 wg_poller
26891 ?        00:00:33 update_1
26899 ?        00:05:08 update_2
26901 ?        00:07:01 update_3
27064 ?        00:00:37 containerd-shim
27232 ?        00:01:24 jellyfin
27289 ?        00:00:26 kworker/62:1-events
27496 ?        00:01:03 kworker/39:1-events
27817 ?        00:00:00 sleep
28193 ?        00:00:00 z_wr_iss
28227 ?        00:00:00 metaslab_group_
28228 ?        00:00:00 metaslab_group_
28237 ?        00:00:00 php-fpm
28397 ?        00:00:00 start.sh
28399 ?        00:00:00 bash
28404 ?        00:00:14 flaresolverr
28449 ?        00:00:18 kworker/61:1-events
28597 ?        00:00:04 node
28661 ?        00:00:11 next-router-wor
28705 ?        00:00:00 z_wr_iss
28707 ?        00:00:00 z_wr_iss
28710 ?        00:00:00 z_wr_iss
28712 ?        00:00:00 z_wr_iss
28719 ?        00:00:00 z_wr_iss
28724 ?        00:00:00 z_wr_iss
28729 ?        00:00:00 z_wr_iss
28733 ?        00:00:00 z_wr_iss
28734 ?        00:00:00 z_wr_iss
28735 ?        00:00:00 z_wr_iss
28736 ?        00:00:00 z_wr_iss
28737 ?        00:00:00 z_wr_iss
28738 ?        00:00:00 z_wr_iss
28741 ?        00:00:00 z_wr_iss
28743 ?        00:00:00 z_wr_iss
28744 ?        00:00:00 z_wr_iss
28748 ?        00:00:00 z_wr_iss
28752 ?        00:00:00 z_wr_iss
28753 ?        00:00:00 z_wr_iss
28754 ?        00:00:00 z_wr_iss
28755 ?        00:00:00 z_wr_iss
28756 ?        00:00:00 z_wr_iss
28779 ?        00:00:00 start.sh
28780 ?        00:00:00 watchdog.sh
28782 ?        00:00:35 bash
29059 ?        00:00:00 kworker/u146:0
29366 ?        00:00:00 z_wr_iss
29367 ?        00:00:03 start.sh
29371 ?        00:00:00 z_wr_iss
29377 ?        00:00:00 z_wr_iss
29389 ?        00:00:51 kworker/41:1-events
29432 ?        00:00:00 z_wr_int_5
29462 ?        00:00:00 sleep
29483 pts/2    00:00:00 ps
29490 ?        00:00:00 notify
29522 ?        00:00:00 Xvfb
29604 ?        00:00:00 docker-proxy
29612 ?        00:00:00 docker-proxy
29648 ?        00:00:34 containerd-shim
29677 ?        00:00:00 tini
29727 ?        00:00:01 node
29782 ?        00:01:03 next-render-wor
30365 ?        00:00:24 kworker/18:2-events
30641 ?        00:00:11 kworker/60:1-events
30859 ?        00:00:12 node
31025 ?        00:00:00 docker-proxy
31031 ?        00:00:00 docker-proxy
31050 ?        00:00:00 docker-proxy
31057 ?        00:00:00 docker-proxy
31242 ?        00:00:33 containerd-shim
31302 pts/0    00:23:38 Xvnc
31310 ?        00:00:01 portainer
31341 ?        00:00:00 kworker/68:0-events
31344 ?        00:00:00 kworker/3:3-md
31345 ?        00:00:00 kworker/16:2-events
31393 ?        00:00:00 kworker/51:0-events
32040 pts/1    00:00:00 nginx
32081 ?        00:00:00 docker-proxy
32086 ?        00:00:00 kworker/u148:1-btrfs-worker-high
32094 ?        00:00:00 docker-proxy
32095 ?        00:00:00 kworker/u149:1
32123 pts/1    00:00:01 nginx
32124 pts/1    00:00:21 nginx
32151 ?        00:00:33 containerd-shim
32174 ?        00:00:00 postgres
32186 ?        00:00:33 kworker/26:1-events
32330 pts/2    00:00:00 openbox
32391 ?        00:00:15 kworker/64:0-events
32490 ?        00:00:03 docker-proxy
32497 ?        00:00:00 docker-proxy
32570 ?        00:00:00 postgres
32571 ?        00:00:00 postgres
32578 ?        00:00:00 postgres
32580 ?        00:00:00 postgres
32581 ?        00:00:00 postgres
32586 ?        00:00:35 containerd-shim
32683 ?        00:00:00 s6-svscan
32806 ?        00:00:00 kworker/26:0-events
32897 ?        00:00:13 kworker/59:2-events
33594 ?        00:00:00 kworker/67:2-events
33661 ?        00:00:00 kworker/46:0-events
33665 ?        00:00:45 kworker/40:0-events
33675 ?        00:00:06 xcompmgr
33729 ?        00:00:00 s6-supervise
33731 ?        00:00:00 s6-linux-init-s
34068 ?        00:00:00 s6-supervise
34073 ?        00:00:00 s6-supervise
34077 ?        00:00:00 s6-supervise
34078 ?        00:00:00 s6-supervise
34106 ?        00:00:00 s6-ipcserverd
34215 ?        00:00:08 kworker/35:0-events
34301 ?        00:00:16 docker-proxy
34316 ?        00:00:00 docker-proxy
34706 ?        00:00:39 containerd-shim
34743 ?        00:00:34 kworker/22:1-events
34912 ?        00:00:00 s6-svscan
35344 ?        00:00:00 kworker/25:1-events
35560 ?        00:16:12 Prowlarr
35562 ?        00:00:00 busybox
36166 ?        00:00:00 s6-supervise
36178 ?        00:00:00 s6-linux-init-s
36210 ?        00:00:21 kworker/65:0-events
36395 ?        00:00:00 s6-supervise
36397 ?        00:00:00 s6-supervise
36398 ?        00:00:00 s6-supervise
36400 ?        00:00:00 s6-supervise
36415 ?        00:00:00 s6-ipcserverd
36638 ?        00:00:17 kworker/49:2-events
36799 ?        00:00:19 kworker/11:1-events
36980 ?        00:00:05 kworker/56:0-events
36987 ?        00:00:00 docker-proxy
37020 ?        00:00:00 docker-proxy
37487 ?        00:00:37 containerd-shim
37720 ?        00:00:00 s6-svscan
37833 ?        00:01:14 kworker/8:0-events
37883 ?        00:01:14 kworker/1:0-events
38288 ?        00:00:00 kworker/8:1-events
38354 ?        00:00:00 z_rd_iss
38729 ?        00:00:00 kworker/29:2-events
39106 ?        00:00:14 kworker/23:1-events
39167 ?        00:00:00 kworker/20:0-events
39320 ?        00:00:08 kworker/71:1-events
39370 ?        00:02:49 kworker/36:0-events
39429 ?        00:00:20 kworker/69:2-events
39549 ?        00:00:00 s6-supervise
39551 ?        00:00:00 s6-linux-init-s
39553 ?        00:00:10 kworker/52:0-events
39931 ?        00:00:00 s6-supervise
39932 ?        00:00:00 s6-supervise
39933 ?        00:00:00 s6-supervise
39934 ?        00:00:00 s6-supervise
39946 ?        00:00:00 s6-ipcserverd
40075 ?        00:00:33 containerd-shim
40214 ?        00:00:00 dumb-init
40483 ?        00:00:18 supervisord
41669 ?        00:01:06 kworker/2:1-events
42992 ?        00:00:08 smbd
43118 ?        00:01:59 kworker/7:1-events
43153 ?        00:00:34 kworker/21:1-events
43383 ?        00:00:08 kworker/14:2-events
43414 ?        00:00:30 kworker/31:1-events
43457 ?        00:00:00 kworker/49:1-events
43781 ?        00:00:00 docker-proxy
43795 ?        00:00:00 docker-proxy
44216 ?        00:00:33 containerd-shim
44414 ?        00:00:00 dumb-init
44437 ?        00:00:29 kworker/50:0-events
44695 ?        00:00:00 kworker/65:2
45011 ?        00:00:34 node
45630 ?        00:00:00 docker-proxy
45661 ?        00:00:00 docker-proxy
46019 ?        00:00:32 containerd-shim
46215 ?        00:00:08 kworker/51:2-events
46224 ?        00:00:00 s6-svscan
47113 ?        00:00:00 s6-supervise
47115 ?        00:00:00 s6-linux-init-s
47415 ?        00:00:00 s6-supervise
47416 ?        00:00:00 s6-supervise
47418 ?        00:00:00 s6-supervise
47419 ?        00:00:00 s6-supervise
47452 ?        00:00:00 s6-ipcserverd
47866 ?        00:00:05 kworker/55:2-events
47887 ?        00:00:00 start.sh
47948 ?        00:00:00 bash
47999 ?        00:00:08 syncthing
48097 ?        00:00:23 node
48098 ?        00:00:00 busybox
48670 ?        1-15:37:08 syncthing
48882 ?        00:00:25 kworker/20:2-events
49008 ?        00:05:27 java
49554 ?        00:00:00 run
49776 ?        00:00:31 kworker/24:2-events
50475 ?        00:00:37 kworker/13:0-events
50636 ?        00:00:00 kworker/54:2-events
51119 ?        00:03:58 kworker/5:1-events
51283 ?        00:00:00 kworker/u146:5-events_unbound
51369 ?        02:47:23 openvpn
51471 ?        00:00:00 kworker/1:2-events
51479 ?        00:00:00 kworker/52:1-events
51922 ?        00:00:10 kworker/55:0-events
52194 ?        00:00:00 busybox
52195 ?        01:18:13 Sonarr
52565 ?        00:00:00 postgres
52570 ?        00:00:00 postgres
52779 ?        00:00:00 smbd
52786 ?        00:00:00 samba-dcerpcd
52807 ?        00:02:21 smbd
53326 ?        00:00:00 prerunget.sh
53387 ?        00:00:05 kworker/30:2-events
53512 ?        00:00:00 kworker/2:0-events
53708 ?        00:00:00 kworker/41:0-events
53903 ?        00:00:24 kworker/58:1-events
54231 ?        00:00:16 kworker/29:0-events
54265 ?        00:00:04 zenity
54354 ?        00:00:00 smbd
54358 ?        00:00:00 rpcd_lsad
54579 ?        00:00:19 kworker/17:2-events
54818 ?        00:00:39 kworker/48:2-events
55405 ?        00:02:19 EasyAudioEncode
55447 ?        00:00:00 kworker/31:2
56181 ?        00:01:25 kworker/43:1-events
57231 ?        00:00:20 kworker/57:0-events
58955 ?        00:00:00 kworker/u146:3-events_unbound
59159 ?        00:00:13 kworker/45:1-events
59270 ?        00:00:00 kworker/32:2
59773 ?        00:00:26 kworker/47:3-events
59810 ?        00:04:56 deluge-web
59813 ?        00:00:00 privoxy
60306 ?        00:00:07 kworker/35:2-events
60462 ?        00:01:50 kworker/6:0-events
61068 ?        00:00:00 busybox
61069 ?        01:59:29 Radarr
61091 ?        00:00:00 sleep
61827 ?        00:01:47 kworker/0:0-events
62104 ?        00:00:00 php-fpm
62216 ?        00:01:01 kworker/3:0-events
62916 ?        00:00:00 kworker/47:1-events
62931 ?        00:00:00 sleep
63513 ?        00:03:01 kworker/44:3-events
64027 ?        00:00:00 kworker/39:2-events
64523 ?        00:00:10 kworker/69:1-events
64764 ?        00:00:21 kworker/61:0-events
65091 ?        00:07:58 file_manager
65147 ?        00:00:00 kworker/u145:0-events_power_efficient
65175 ?        00:00:17 Plex Transcoder
65291 ?        00:00:00 kworker/62:2-events
65453 ?        00:00:00 kworker/u146:2-flush-btrfs-1
65454 ?        00:00:15 kworker/32:0-events
65774 ?        00:00:11 kworker/40:2-events
66011 ?        00:00:34 kworker/70:2-events
66362 ?        00:00:00 kworker/28:1-events
67098 ?        00:00:07 kworker/15:0-events
67244 ?        00:00:00 php-fpm
67860 ?        00:01:28 kworker/42:0-events
67909 ?        00:00:32 kworker/63:2-events
67954 ?        00:00:42 Plex Script Hos
68013 ?        00:00:00 kworker/0:1
68076 ?        00:00:15 kworker/66:2-events
68358 ?        00:00:02 kworker/11:2-events
68743 ?        00:00:46 Plex Tuner Serv
68842 ?        00:00:00 php-fpm
68843 ?        00:00:00 php-fpm
68844 ?        00:00:00 php-fpm
69734 ?        00:00:00 kworker/54:1-events
70441 ?        00:00:01 kworker/24:0-events
70504 ?        00:00:00 kworker/34:2-events
70532 ?        00:00:00 kworker/70:0-events
70962 ?        00:00:00 kworker/56:1-events
71194 ?        00:00:00 z_wr_iss
71196 ?        00:00:00 z_wr_iss
71199 ?        00:00:00 z_wr_iss
71200 ?        00:00:00 z_wr_iss
71201 ?        00:00:00 z_wr_iss
71202 ?        00:00:00 z_wr_iss
71203 ?        00:00:00 z_wr_iss
71204 ?        00:00:00 z_wr_iss
71337 ?        00:00:15 kworker/67:1-events
72520 ?        00:00:11 Plex Media Scan
72746 ?        00:00:14 kworker/19:0-events
72964 ?        00:00:26 kworker/16:1-events
73228 ?        00:00:00 kworker/4:2-events
73408 ?        00:00:00 kworker/58:0-events
73411 ?        00:00:06 kworker/10:1-events
73501 ?        00:00:00 kworker/u145:1-btrfs-endio
73674 ?        00:01:05 kworker/38:1-events
73676 ?        00:00:00 kworker/u146:1-flush-0:50
73682 ?        00:00:43 kworker/25:2-events
73711 ?        00:00:00 kworker/66:0-events

5 minutes ago, Frank1940 said:

Personally, I think the OP now has two problems.  The original one is probably in the UPS.  One thing I would be looking at is the actual storage capacity of the battery.  That server has a CPU with a 145W TDP and 19 drives in it.  IF that battery is more than about 18 months old, I suspect it can not power that server as it tries to shutdown and the output of the UPS goes bonky...

 

The second problem is the present running state of the server.  It is possible that the UPS wonky output corrupted something on the boot drive or in the Docker or VM configuration files.  

This morning I stopped all containers except Syncthing, as I'm currently syncing to my backup server, and the CPU usage remains unchanged.

Disable the docker service and boot in safe mode, if all looks normal like that, recreate the docker image, and see if performance remains good:

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file
Then:
https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications
Also see below if you have any custom docker networks:
https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

 

 

1 hour ago, JorgeB said:

Disable the docker service and boot in safe mode, if all looks normal like that, recreate the docker image, and see if performance remains good:

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file
Then:
https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications
Also see below if you have any custom docker networks:
https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

 

 

Thanks.  I'll try that.  I kicked off a restore of all my appdata and containers from before the incident so I'll just let that complete first.

On 12/20/2024 at 11:14 AM, JorgeB said:

Disable the docker service and boot in safe mode, if all looks normal like that, recreate the docker image, and see if performance remains good:

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file
Then:
https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications
Also see below if you have any custom docker networks:
https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

 

 

So I tried this and I wasn't able to stop the array to boot into safe mode.  I did everything except booting into safe mode and performance looks good with the Docker service stopped, but deleting the docker image and reinstalling all the apps produces the same issue.  I see lots of these errors in the syslog.

 

Dec 26 11:33:05 Goathead root: rmdir: failed to remove '/mnt/user': Device or resource busy Dec 26 11:33:05 Goathead emhttpd: shcmd (1754): exit status: 1 Dec 26 11:33:05 Goathead emhttpd: shcmd (1756): rm -f /boot/config/plugins/dynamix/mover.cron Dec 26 11:33:05 Goathead emhttpd: shcmd (1757): /usr/local/sbin/update_cron Dec 26 11:33:05 Goathead emhttpd: Retry unmounting user share(s)... Dec 26 11:33:10 Goathead emhttpd: shcmd (1758): /usr/sbin/zfs unmount -a Dec 26 11:33:10 Goathead root: cannot unmount '/mnt/containers_cache/appdata': pool or dataset is busy Dec 26 11:33:10 Goathead root: cannot unmount '/mnt/containers_cache': pool or dataset is busy Dec 26 11:33:10 Goathead emhttpd: shcmd (1758): exit status: 1 Dec 26 11:33:10 Goathead emhttpd: shcmd (1759): umount /mnt/user Dec 26 11:33:10 Goathead root: umount: /mnt/user: target is busy. Dec 26 11:33:10 Goathead emhttpd: shcmd (1759): exit status: 32 Dec 26 11:33:10 Goathead emhttpd: shcmd (1760): rmdir /mnt/user Dec 26 11:33:10 Goathead root: rmdir: failed to remove '/mnt/user': Device or resource busy Dec 26 11:33:10 Goathead emhttpd: shcmd (1760): exit status: 1 Dec 26 11:33:10 Goathead emhttpd: shcmd (1762): rm -f /boot/config/plugins/dynamix/mover.cron Dec 26 11:33:10 Goathead emhttpd: shcmd (1763): /usr/local/sbin/update_cron Dec 26 11:33:10 Goathead emhttpd: Retry unmounting user share(s)... Dec 26 11:33:15 Goathead emhttpd: shcmd (1764): /usr/sbin/zfs unmount -a Dec 26 11:33:15 Goathead root: cannot unmount '/mnt/containers_cache/appdata': pool or dataset is busy Dec 26 11:33:15 Goathead root: cannot unmount '/mnt/containers_cache': pool or dataset is busy Dec 26 11:33:15 Goathead emhttpd: shcmd (1764): exit status: 1 Dec 26 11:33:15 Goathead emhttpd: shcmd (1765): umount /mnt/user Dec 26 11:33:15 Goathead root: umount: /mnt/user: target is busy. Dec 26 11:33:15 Goathead emhttpd: shcmd (1765): exit status: 32 Dec 26 11:33:15 Goathead emhttpd: shcmd (1766): rmdir /mnt/user Dec 26 11:33:15 Goathead root: rmdir: failed to remove '/mnt/user': Device or resource busy Dec 26 11:33:15 Goathead emhttpd: shcmd (1766): exit status: 1 Dec 26 11:33:15 Goathead emhttpd: shcmd (1768): rm -f /boot/config/plugins/dynamix/mover.cron Dec 26 11:33:15 Goathead emhttpd: shcmd (1769): /usr/local/sbin/update_cron Dec 26 11:33:15 Goathead emhttpd: Retry unmounting user share(s)... Dec 26 11:33:20 Goathead emhttpd: shcmd (1770): /usr/sbin/zfs unmount -a Dec 26 11:33:20 Goathead root: cannot unmount '/mnt/containers_cache/appdata': pool or dataset is busy Dec 26 11:33:20 Goathead root: cannot unmount '/mnt/containers_cache': pool or dataset is busy Dec 26 11:33:20 Goathead emhttpd: shcmd (1770): exit status: 1 Dec 26 11:33:20 Goathead emhttpd: shcmd (1771): umount /mnt/user Dec 26 11:33:20 Goathead root: umount: /mnt/user: target is busy. Dec 26 11:33:20 Goathead emhttpd: shcmd (1771): exit status: 32 Dec 26 11:33:20 Goathead emhttpd: shcmd (1772): rmdir /mnt/user Dec 26 11:33:20 Goathead root: rmdir: failed to remove '/mnt/user': Device or resource busy Dec 26 11:33:20 Goathead emhttpd: shcmd (1772): exit status: 1 Dec 26 11:33:20 Goathead emhttpd: shcmd (1774): rm -f /boot/config/plugins/dynamix/mover.cron Dec 26 11:33:20 Goathead emhttpd: shcmd (1775): /usr/local/sbin/update_cron Dec 26 11:33:20 Goathead emhttpd: Retry unmounting user share(s)...

There's something using those mount points preventing them from unmounting, you can start one container at a time and retest to see if it's one of them.

 

Could also be a plugin, so also good to start in safe mode to rule out those.

 

 

 

Performance issues solved, but no root cause of the issue.

 

So I followed the instructions and started one container at a time.  I would check CPU/RAM usage, speed of browsing file system, and then reviewing each line in the logs carefully after starting up the container.  I found more than one container that had a malformed DB or didn't show any clues in the logs but performance was notably impacted.  For those containers I deleted them and the appdata and recreated and reconfigured them.  Took a few days but it worked.  I think the thing that was preventing the array from stopping was an update to FileBot that required an opensubtitles.com login to be added to the container configuration.  Since it was started without this, there was no way to fix it, kill it (CLI or GUI) and so I just needed to do an unclean shutdown.

 

Now, what I still can't understand is how it is possible for a power outage where none of my servers powered off while attached to a fully functioning APC UPS to corrupt my containers.

8 minutes ago, Bmalone said:

I found more than one container that had a malformed DB or didn't show any clues in the logs but performance was notably impacted. 

 

Where are the clients located that write to those databases?   When the powercuts occur, does your UPS continue to power the entire LAN so that these clients are not dropping the connection to the containers? 

 

Is the battery in the UPS capable of powering the full load required?  (Don't tell us that the software 'says' that it will!  Test it using an actual load to determine the run time to UPS shutdown--- Incandescent lamps are good for doing this.)