Fredrick Posted October 17, 2017 Share Posted October 17, 2017 (edited) Hi, I've run into a problem with a process shfs that has a lot of files open, and it wont calm down. There are currently 146 files open, most of them are in /cache/appdata. I had this yesterday aswell, and was unable to kill the process or to shut down the system in any reasonable way. Had to hold the button unfortunately, and the parity check just finished OK. I'm seeing i/o-wait times for my drives skyrocketing and this creates for very poor performance of my server overall. I dont have any Reiser formatted drives. The process goes away if I disable Docker, but comes right back again. No container has access to all the files that the process has open, so I dont know what can cause this. Diagnostics attached. tower-diagnostics-20171017-1538.zip EDIT: The problem got worse until everything stalled, had to pull the plug again. I've stopped containers one by one now, and it continues until all containers are stopped. Plex is the worst (has the biggest appdata folder/most items in appdata). I tried uninstalling Open Files plugin, but still stuck with the same error. Edited October 17, 2017 by Fredrick Quote Link to comment
Fredrick Posted October 18, 2017 Author Share Posted October 18, 2017 Anyone have ideas here? I tested some more with shutting down containers, and the load/activity seems to correspond with how many files are accessed by Docker containers. It doesnt stop until all containers are shut down, but it does get (slightly) better for each container stopped. I've tried with/without cache dir plugin enabled (also tried removing it). The past 12 hours I've had about 1.5million writes to my cache drive, without actually doing much of it intentionally.. Most of my searching on the issue points to some older Unraid versions, so not sure how applicable it is. I was on 6.4 rc9 when I posted the diagnostics, went back to 6.3.5, but still have the same issue. Quote Link to comment
Fredrick Posted October 19, 2017 Author Share Posted October 19, 2017 (edited) Sorry bout this second bump, but I'm still stuck, and cant find much about this at all. There are about 1 million reads and writes to my cache drive per 24h that I cant "account" for. Some is obviously just normal docker traffic, but this is extremely excessive. The load seems to have calmed down after a reboot, but I still suspect this will go to 100% after some uptime.. As it seems the shfs process is associated with user shares, I tried to "move" (just changed the docker settings) docker.img from /mnt/user/system to /mnt/cache/system, and likewise change all container paths to /mnt/cache/appdata instead of the usual /mnt/user/appdata. This did not work, and I saw the same shfs activity. I also tried copying (through ssh with docker stopped) docker.img to an unassigned drive, but the process was unable to complete. I also tried the following fuser command to see the various accesses, and shfs is not the only thing accessing my drives, I've got a lot of nginx for some reason. My /mnt/disks/HP_LOGICAL_VOLUME is my unassigned drive, but why every PID possible is accessing /mnt/disks I dont know.. root@Tower:~# /usr/bin/fuser -mv /mnt/disk* /mnt/user* /mnt/cache* USER PID ACCESS COMMAND /mnt/disk1: root kernel mount /mnt/disk1 /mnt/disk2: root kernel mount /mnt/disk2 /mnt/disk3: root kernel mount /mnt/disk3 /mnt/disk4: root kernel mount /mnt/disk4 root 7664 f.... shfs /mnt/disk5: root kernel mount /mnt/disk5 /mnt/disk6: root kernel mount /mnt/disk6 /mnt/disk7: root kernel mount /mnt/disk7 root 7664 f.... shfs /mnt/disks: root kernel mount / root 1 .rce. init root 2 .rc.. kthreadd root 3 .rc.. ksoftirqd/0 root 5 .rc.. kworker/0:0H root 8 .rc.. rcu_preempt root 9 .rc.. rcu_sched root 10 .rc.. rcu_bh root 11 .rc.. migration/0 root 12 .rc.. lru-add-drain root 13 .rc.. cpuhp/0 root 14 .rc.. cpuhp/1 root 15 .rc.. migration/1 root 16 .rc.. ksoftirqd/1 root 18 .rc.. kworker/1:0H root 20 .rc.. cpuhp/2 root 21 .rc.. migration/2 root 22 .rc.. ksoftirqd/2 root 24 .rc.. kworker/2:0H root 25 .rc.. cpuhp/3 root 26 .rc.. migration/3 root 27 .rc.. ksoftirqd/3 root 29 .rc.. kworker/3:0H root 30 .rc.. cpuhp/4 root 31 .rc.. migration/4 root 32 .rc.. ksoftirqd/4 root 34 .rc.. kworker/4:0H root 35 .rc.. cpuhp/5 root 36 .rc.. migration/5 root 37 .rc.. ksoftirqd/5 root 40 .rc.. cpuhp/6 root 41 .rc.. migration/6 root 42 .rc.. ksoftirqd/6 root 44 .rc.. kworker/6:0H root 45 .rc.. cpuhp/7 root 46 .rc.. migration/7 root 47 .rc.. ksoftirqd/7 root 49 .rc.. kworker/7:0H root 50 .rc.. cpuhp/8 root 51 .rc.. migration/8 root 52 .rc.. ksoftirqd/8 root 54 .rc.. kworker/8:0H root 55 .rc.. cpuhp/9 root 56 .rc.. migration/9 root 57 .rc.. ksoftirqd/9 root 60 .rc.. cpuhp/10 root 61 .rc.. migration/10 root 62 .rc.. ksoftirqd/10 root 64 .rc.. kworker/10:0H root 65 .rc.. cpuhp/11 root 66 .rc.. migration/11 root 67 .rc.. ksoftirqd/11 root 69 .rc.. kworker/11:0H root 70 .rc.. cpuhp/16 root 71 .rc.. migration/16 root 72 .rc.. ksoftirqd/16 root 74 .rc.. kworker/16:0H root 75 .rc.. cpuhp/17 root 76 .rc.. migration/17 root 77 .rc.. ksoftirqd/17 root 79 .rc.. kworker/17:0H root 80 .rc.. cpuhp/18 root 81 .rc.. migration/18 root 82 .rc.. ksoftirqd/18 root 83 .rc.. kworker/18:0 root 84 .rc.. kworker/18:0H root 85 .rc.. cpuhp/19 root 86 .rc.. migration/19 root 87 .rc.. ksoftirqd/19 root 89 .rc.. kworker/19:0H root 90 .rc.. cpuhp/20 root 91 .rc.. migration/20 root 92 .rc.. ksoftirqd/20 root 94 .rc.. kworker/20:0H root 95 .rc.. cpuhp/21 root 96 .rc.. migration/21 root 97 .rc.. ksoftirqd/21 root 100 .rc.. cpuhp/22 root 101 .rc.. migration/22 root 102 .rc.. ksoftirqd/22 root 105 .rc.. cpuhp/23 root 106 .rc.. migration/23 root 107 .rc.. ksoftirqd/23 root 109 .rc.. kworker/23:0H root 110 .rc.. cpuhp/24 root 111 .rc.. migration/24 root 112 .rc.. ksoftirqd/24 root 114 .rc.. kworker/24:0H root 115 .rc.. cpuhp/25 root 116 .rc.. migration/25 root 117 .rc.. ksoftirqd/25 root 120 .rc.. cpuhp/26 root 121 .rc.. migration/26 root 122 .rc.. ksoftirqd/26 root 124 .rc.. kworker/26:0H root 125 .rc.. cpuhp/27 root 126 .rc.. migration/27 root 127 .rc.. ksoftirqd/27 root 129 .rc.. kworker/27:0H root 130 .rc.. kdevtmpfs root 131 .rc.. netns root 407 .rce. docker-proxy root 415 Frce. docker-containe root 416 .rc.. oom_reaper root 417 .rc.. writeback root 419 .rc.. kcompactd0 root 420 .rc.. kcompactd1 root 421 .rc.. ksmd root 422 .rc.. khugepaged root 423 .rc.. crypto root 424 .rc.. kintegrityd root 425 .rc.. bioset root 427 .rc.. kblockd root 444 .r.e. s6-svscan root 467 .rc.. kworker/9:0 root 500 .r.e. s6-supervise root 540 .r.e. s6-supervise root 541 .r.e. s6-supervise root 543 .r.e. s6-supervise (unknown) 544 Fr.e. dbus-daemon (unknown) 545 Frce. avahi-daemon nobody 546 .r.e. Plex Media Serv nobody 593 .rce. Plex Script Hos root 638 .rc.. ata_sff root 656 .rc.. devfreq_wq root 686 .r.e. s6-supervise nobody 689 .r.e. mono root 713 Frce. smbd nobody 790 .r.e. Plex Tuner Serv root 801 .rc.. rpciod root 802 .rc.. xprtiod root 909 .rc.. kswapd0 root 910 .rc.. kswapd1 root 911 .rc.. vmstat root 1007 .rc.. nfsiod root 1010 .rc.. cifsiod root 1011 .rc.. cifsoplockd root 1019 .rc.. bioset root 1028 .rc.. xfsalloc root 1029 .rc.. xfs_mru_cache root 1060 .rc.. kthrotld root 1089 .rc.. kworker/u65:3 root 1101 .rc.. acpi_thermal_pm root 1157 .rc.. bioset root 1163 .rc.. bioset root 1169 .rc.. bioset root 1175 .rc.. bioset root 1178 .rc.. bioset root 1181 .rc.. bioset root 1184 .rc.. bioset root 1187 .rc.. bioset root 1190 .rc.. bioset root 1191 .rc.. bioset root 1192 .rc.. bioset root 1193 .rc.. bioset root 1194 .rc.. bioset root 1195 .rc.. bioset root 1196 .rc.. bioset root 1197 .rc.. bioset root 1198 .rc.. bioset root 1199 .rc.. bioset root 1200 .rc.. bioset root 1201 .rc.. bioset root 1202 .rc.. bioset root 1203 .rc.. bioset root 1204 .rc.. bioset root 1205 .rc.. bioset root 1249 .rc.. vfio-irqfd-clea root 1371 .rc.. bioset root 1424 .rc.. scsi_eh_0 root 1425 .rc.. scsi_tmf_0 root 1426 .rc.. usb-storage root 1430 .rc.. bioset root 1543 .rc.. kworker/3:0 root 1545 .rc.. kworker/25:0 root 1562 .rce. docker-proxy root 1566 frce. udevd root 1571 Frce. docker-containe root 1589 .rce. s6-svscan root 1592 .rc.. scsi_eh_1 root 1593 .rc.. scsi_tmf_1 root 1594 .rc.. fw_event_mpt2sa root 1611 .rc.. scsi_eh_2 root 1612 .rc.. scsi_tmf_2 root 1631 .rc.. bioset root 1645 .rc.. bioset root 1646 .rc.. bioset root 1647 .rc.. scsi_eh_3 root 1648 .rc.. scsi_tmf_3 root 1649 .rc.. scsi_eh_4 root 1650 .rc.. scsi_tmf_4 root 1651 .rc.. bioset root 1652 .rc.. bioset root 1653 .rc.. bioset root 1654 .rc.. bioset root 1655 .rc.. bioset root 1656 .rc.. bioset root 1657 .rc.. bioset root 1658 .rc.. bioset root 1659 .rc.. bioset root 1660 .rc.. bioset root 1661 .rc.. bioset root 1662 .rc.. bioset root 1663 .rc.. bioset root 1664 .rc.. bioset root 1665 .rc.. bioset root 1666 .rc.. bioset root 1667 .rc.. bioset root 1668 .rc.. bioset root 1669 .rc.. bioset root 1670 .rc.. bioset root 1671 .rc.. bioset root 1672 .rc.. bioset root 1673 .rc.. bioset root 1674 .rc.. bioset root 1676 .rc.. bioset root 1677 .rc.. bioset root 1679 .rc.. bioset root 1680 .rc.. bioset root 1681 .rc.. bioset root 1683 .rc.. bioset root 1684 .rc.. bioset root 1686 .rc.. bioset root 1687 .rc.. bioset root 1688 .rc.. bioset root 1689 .rc.. bioset root 1708 .rc.. bioset root 1709 .rc.. bioset root 1710 .rc.. bioset root 1711 .rc.. bioset root 1712 .rc.. bioset root 1713 .rc.. bioset root 1714 .rc.. bioset root 1715 .rc.. bioset root 1716 .rc.. bioset root 1717 .rc.. bioset root 1718 .rc.. bioset root 1719 .rc.. bioset root 1728 .rc.. poll_mpt2sas0_s root 1732 .rce. s6-supervise root 1883 .rc.. kworker/9:1H root 1884 .rc.. kworker/11:1H root 1890 .rce. rsyslogd root 1924 .rc.. kworker/3:1H root 1925 .rc.. kworker/19:1H root 1926 .rc.. kworker/23:1H root 1927 .rc.. kworker/17:1H root 1929 .rc.. kworker/21:1H root 1931 .rc.. kworker/7:1H root 1932 .rc.. kworker/27:1H root 1933 .rc.. kworker/16:1H root 1934 .rc.. kworker/22:1H root 1935 .rc.. kworker/0:1H root 1936 .rc.. kworker/20:1H root 1937 .rc.. kworker/4:1H root 1938 .rc.. kworker/2:1H root 1939 .rc.. kworker/18:1H root 1940 .rc.. kworker/8:1H root 1941 .rc.. kworker/6:1H root 1942 .rc.. kworker/10:1H root 1943 .rc.. kworker/26:1H root 1944 .rc.. kworker/24:1H messagebus 2029 Frce. dbus-daemon bin 2037 Frce. rpcbind rpc 2042 Frce. rpc.statd root 2061 .rce. sshd root 2063 .rce. sshd ntp 2075 .rce. ntpd root 2082 Frce. acpid root 2091 .rce. crond daemon 2093 Frce. atd root 2099 Frce. nmbd root 2101 Frce. smbd root 2103 Frce. smbd-notifyd root 2104 Frce. cleanupd root 2108 Frce. winbindd root 2110 Frce. winbindd root 2120 .rce. cpuload root 2201 .rce. s6-supervise root 2204 .rce. s6-supervise root 2205 .rce. s6-supervise root 2209 .rce. nginx root 2210 .rce. crond root 2212 .rce. php-fpm7 nobody 2234 .rce. nginx nobody 2235 .rce. nginx nobody 2236 .rce. nginx nobody 2237 .rce. nginx root 2378 .rce. bash root 2418 frce. sleep root 2426 .rce. sleep root 2441 .rc.. kworker/25:0H root 2631 .rc.. loop1 root 2634 .rc.. btrfs-worker root 2635 .rc.. btrfs-worker-hi root 2636 .rc.. btrfs-delalloc root 2637 .rc.. btrfs-flush_del root 2638 .rc.. btrfs-cache root 2639 .rc.. btrfs-submit root 2640 .rc.. btrfs-fixup root 2641 .rc.. btrfs-endio root 2642 .rc.. btrfs-endio-met root 2643 .rc.. btrfs-endio-met root 2644 .rc.. btrfs-endio-rai root 2645 .rc.. btrfs-endio-rep root 2646 .rc.. btrfs-rmw root 2647 .rc.. btrfs-endio-wri root 2648 .rc.. btrfs-freespace root 2649 .rc.. btrfs-delayed-m root 2650 .rc.. btrfs-readahead root 2651 .rc.. btrfs-qgroup-re root 2652 .rc.. btrfs-extent-re root 2653 .rc.. btrfs-cleaner root 2654 .rc.. btrfs-transacti root 2671 Frce. virtlockd root 2677 Frce. virtlogd root 2692 Frce. libvirtd root 2757 .rc.. kworker/22:0 root 2759 .rc.. kworker/24:2 nobody 2779 .rce. dnsmasq root 2780 .rce. dnsmasq root 3133 .rc.. kworker/u69:0 root 3304 .rc.. kworker/u68:0 root 3324 .rce. docker-proxy root 3331 Frce. docker-containe root 3350 .rce. s6-svscan root 3400 .rce. s6-supervise root 3583 .rce. s6-supervise nobody 3586 .rce. python root 3621 .rc.. kworker/11:0 root 4440 .rc.. kworker/8:0 root 4515 .rce. emhttp root 4516 .rce. agetty root 4517 .rce. agetty root 4518 .rce. agetty root 4519 .rce. agetty root 4520 .rce. agetty root 4521 .rce. agetty root 4586 .rce. netatalk root 4591 .rce. afpd root 4592 .rce. cnid_metad root 4594 .rce. docker-proxy avahi 4598 Frce. avahi-daemon avahi 4599 .rce. avahi-daemon root 4605 Frce. docker-containe root 4608 .rce. avahi-dnsconfd root 4627 .r.e. s6-svscan root 4677 .r.e. s6-supervise root 5045 .r.e. s6-supervise nobody 5048 .rce. mono root 5078 .rc.. kworker/u65:7 root 5193 .rc.. kworker/2:2 root 5391 .rc.. kworker/u66:8 root 5420 .rc.. kworker/1:4 root 5612 .rc.. kworker/1:1 root 5613 .rc.. kworker/4:1 root 5625 .rc.. kworker/20:0 root 5639 .rce. docker-proxy root 5647 Frce. docker-containe root 5666 .r.e. s6-svscan root 5670 .rc.. kworker/17:3 root 5694 .rc.. kworker/23:0 root 5723 .r.e. s6-supervise root 5865 .rc.. kworker/u66:1 root 5907 .r.e. s6-supervise nobody 5910 .rce. mono root 5985 .rc.. kworker/6:0 fredrick 6003 Fr.e. smbd root 6211 .rc.. kworker/10:0 root 6508 .rc.. kworker/5:2H root 6776 .rc.. kworker/26:2 root 6787 .rc.. kworker/5:0 root 6836 .rce. docker-proxy root 6844 Frce. docker-containe root 6863 .r.e. s6-svscan root 6911 .r.e. s6-supervise root 7094 .r.e. s6-supervise nobody 7097 .rce. mono root 7182 .rc.. md root 7183 .rc.. mdrecoveryd root 7191 .rc.. spinupd root 7192 .rc.. spinupd root 7193 .rc.. spinupd root 7194 .rc.. spinupd root 7195 .rc.. spinupd root 7196 .rc.. spinupd root 7197 .rc.. spinupd root 7198 .rc.. spinupd root 7199 .rc.. spinupd root 7367 .rc.. unraidd root 7369 .rc.. bioset root 7370 .rc.. bioset root 7372 .rc.. bioset root 7374 .rc.. bioset root 7375 .rc.. bioset root 7376 .rc.. bioset root 7377 .rc.. bioset root 7475 frce. inetd root 7494 .rc.. xfs-buf/md1 root 7495 .rc.. xfs-data/md1 root 7496 .rc.. xfs-conv/md1 root 7497 .rc.. xfs-cil/md1 root 7498 .rc.. xfs-reclaim/md1 root 7499 .rc.. xfs-log/md1 root 7500 .rc.. xfs-eofblocks/m root 7501 .rc.. xfsaild/md1 root 7515 .rc.. xfs-buf/md2 root 7516 .rc.. xfs-data/md2 root 7517 .rc.. xfs-conv/md2 root 7518 .rc.. xfs-cil/md2 root 7519 .rc.. xfs-reclaim/md2 root 7520 .rc.. xfs-log/md2 root 7521 .rc.. xfs-eofblocks/m root 7522 .rc.. xfsaild/md2 root 7532 .rc.. xfs-buf/md3 root 7533 .rc.. xfs-data/md3 root 7534 .rc.. xfs-conv/md3 root 7535 .rc.. xfs-cil/md3 root 7536 .rc.. xfs-reclaim/md3 root 7537 .rc.. xfs-log/md3 root 7538 .rc.. xfs-eofblocks/m root 7539 .rc.. xfsaild/md3 root 7549 .rc.. xfs-buf/md4 root 7550 .rc.. xfs-data/md4 root 7551 .rc.. xfs-conv/md4 root 7552 .rc.. xfs-cil/md4 root 7553 .rc.. xfs-reclaim/md4 root 7554 .rc.. xfs-log/md4 root 7555 .rc.. xfs-eofblocks/m root 7556 .rc.. xfsaild/md4 root 7566 .rc.. xfs-buf/md5 root 7567 .rc.. xfs-data/md5 root 7568 .rc.. xfs-conv/md5 root 7569 .rc.. xfs-cil/md5 root 7570 .rc.. xfs-reclaim/md5 root 7571 .rc.. xfs-log/md5 root 7572 .rc.. xfs-eofblocks/m root 7573 .rc.. xfsaild/md5 root 7587 .rc.. xfs-buf/md6 root 7588 .rc.. xfs-data/md6 root 7589 .rc.. xfs-conv/md6 root 7590 .rc.. xfs-cil/md6 root 7591 .rc.. xfs-reclaim/md6 root 7592 .rc.. xfs-log/md6 root 7593 .rc.. xfs-eofblocks/m root 7594 .rc.. xfsaild/md6 root 7604 .rc.. xfs-buf/md7 root 7605 .rc.. xfs-data/md7 root 7606 .rc.. xfs-conv/md7 root 7607 .rc.. xfs-cil/md7 root 7608 .rc.. xfs-reclaim/md7 root 7609 .rc.. xfs-log/md7 root 7610 .rc.. xfs-eofblocks/m root 7611 .rc.. xfsaild/md7 root 7622 .rc.. btrfs-worker root 7623 .rc.. kworker/u67:0 root 7626 .rc.. btrfs-worker-hi root 7627 .rc.. btrfs-delalloc root 7628 .rc.. btrfs-flush_del root 7629 .rc.. btrfs-cache root 7630 .rc.. btrfs-submit root 7631 .rc.. btrfs-fixup root 7632 .rc.. btrfs-endio root 7633 .rc.. btrfs-endio-met root 7634 .rc.. btrfs-endio-met root 7635 .rc.. btrfs-endio-rai root 7636 .rc.. btrfs-endio-rep root 7637 .rc.. btrfs-rmw root 7638 .rc.. btrfs-endio-wri root 7639 .rc.. btrfs-freespace root 7640 .rc.. btrfs-delayed-m root 7641 .rc.. btrfs-readahead root 7642 .rc.. btrfs-qgroup-re root 7643 .rc.. btrfs-extent-re root 7644 .rc.. btrfs-cleaner root 7645 .rc.. btrfs-transacti root 7654 frce. shfs root 7664 frce. shfs root 7729 frce. diskload root 7844 .rc.. xfs-buf/sdb1 root 7845 .rc.. xfs-data/sdb1 root 7846 .rc.. xfs-conv/sdb1 root 7847 .rc.. xfs-cil/sdb1 root 7848 .rc.. xfs-reclaim/sdb root 7849 .rc.. xfs-log/sdb1 root 7850 .rc.. xfs-eofblocks/s root 7851 .rc.. xfsaild/sdb1 root 7887 .rce. docker-proxy root 7898 Frce. docker-containe root 7916 .r.e. s6-svscan root 7966 .r.e. s6-supervise root 8130 .rc.. kworker/27:1 root 8148 .r.e. s6-supervise nobody 8151 .rce. mono root 8706 .rc.. kworker/25:3 root 8993 .rc.. kworker/u65:11 root 9006 .rc.. kworker/21:1 root 9032 .rc.. kworker/u64:1 root 9282 .rc.. kworker/0:1 root 9992 .rc.. kworker/10:2 root 10010 .rc.. kworker/7:1 root 10027 .rc.. kworker/3:1 root 10078 .rc.. kworker/23:2 root 10171 .rc.. kworker/4:0 root 10494 .rc.. kworker/u66:5 root 10500 Frce. docker-containe root 10510 .rc.. loop0 root 10513 .rc.. btrfs-worker root 10514 .rc.. btrfs-worker-hi root 10515 .rc.. btrfs-delalloc root 10516 .rc.. btrfs-flush_del root 10517 .rc.. btrfs-cache root 10518 .rc.. btrfs-submit root 10519 .rc.. btrfs-fixup root 10520 .rc.. btrfs-endio root 10521 .rc.. btrfs-endio-met root 10522 .rc.. btrfs-endio-met root 10523 .rc.. btrfs-endio-rai root 10524 .rc.. btrfs-endio-rep root 10525 .rc.. btrfs-rmw root 10526 .rc.. btrfs-endio-wri root 10527 .rc.. btrfs-freespace root 10528 .rc.. btrfs-delayed-m root 10529 .rc.. btrfs-readahead root 10530 .rc.. btrfs-qgroup-re root 10531 .rc.. btrfs-extent-re root 10532 .rc.. btrfs-cleaner root 10533 .rc.. btrfs-transacti root 10541 .r.e. s6-svscan root 10544 Frce. dockerd root 10564 Frce. docker-containe root 10608 .r.e. s6-supervise root 10824 .r.e. s6-supervise nobody 10827 .rce. java root 10882 .rc.. kworker/19:0 root 10923 .rc.. kworker/u64:2 root 11151 .rc.. kworker/u69:2 nobody 11856 .rce. mongod root 12068 .rc.. kworker/21:2H root 12699 .rc.. kworker/16:0 root 12810 .rc.. kworker/26:0 nobody 13511 .rce. php-fpm7 root 14511 .rc.. kworker/16:2 nobody 14966 .rce. php-fpm7 root 15123 .rc.. kworker/5:2 nobody 15482 .rce. php-fpm7 root 16462 .rc.. kworker/u65:4 root 16769 .rc.. kworker/0:0 root 17316 .rc.. kworker/2:0 root 17364 .rc.. kworker/u65:6 root 18086 .rc.. kworker/1:1H root 18095 .rc.. kworker/17:0 root 18137 .rc.. kworker/u65:5 root 18845 .rc.. kworker/9:2 root 20058 Frce. winbindd root 20325 .rce. qemu-system-x86 root 20330 .rc.. vhost-20325 root 20335 .rc.. kvm-pit/20325 root 22021 .rce. docker-proxy root 22029 Frce. docker-containe (unknown) 22047 .rce. grafana-server root 22357 .rc.. bond0 root 22408 .rc.. kworker/22:2 root 23302 .rc.. kworker/u66:3 root 23474 .rc.. kworker/9:2H root 23504 .rce. docker-proxy root 23511 Frce. docker-containe root 23529 .rce. s6-svscan root 23581 .rce. s6-supervise root 23752 .rce. s6-supervise nobody 23756 .rce. python root 23951 .rc.. kworker/25:2H root 24987 .rc.. kworker/7:0 root 25171 .rce. docker-proxy root 25182 .rce. docker-proxy root 25192 .rce. docker-proxy root 25200 Frce. docker-containe root 25219 .rce. tini root 25243 .rce. supervisord root 25324 .rc.. kworker/21:0 root 25358 .rc.. kworker/5:0H root 25378 .rc.. kworker/24:0 root 25392 .rc.. kworker/8:1 nobody 25481 .rce. deluge.sh nobody 25482 .rce. deluge-web.sh nobody 25484 .rce. deluged nobody 25972 .rce. deluge-web root 26342 .rc.. kworker/u65:2 root 26515 .rc.. kworker/18:1 root 26520 .rc.. kworker/6:2 root 26542 .rce. docker-proxy root 26553 .rce. docker-proxy root 26562 Frce. docker-containe root 26581 .rce. influxd root 27889 .rc.. kworker/19:1 root 27915 .rce. docker-proxy root 27922 Frce. docker-containe root 27940 .r.e. s6-svscan root 27990 .r.e. s6-supervise root 28279 .r.e. s6-supervise nobody 28282 .r.e. mono root 29340 .rc.. kworker/27:0 root 29481 .rce. docker-proxy root 29492 .rce. docker-proxy root 29500 Frce. docker-containe root 29518 .rce. s6-svscan root 29570 .rce. s6-supervise root 30033 .rce. s6-supervise root 30034 .rce. s6-supervise root 30035 .rce. s6-supervise root 30036 .rce. s6-supervise root 30038 .rce. s6-supervise root 30041 .rce. nginx root 30043 .rce. php-fpm7 root 30044 .rce. fail2ban-client root 30045 .rce. crond (unknown) 30046 .rce. memcached root 30069 .rc.. kworker/u66:0 nobody 30075 .rce. php-fpm7 nobody 30076 .rce. php-fpm7 nobody 30081 .rce. nginx nobody 30082 .rce. nginx nobody 30083 .rce. nginx nobody 30084 .rce. nginx root 30616 .rc.. kworker/22:0H root 30624 .rc.. kworker/11:1 root 30645 .rce. docker-proxy root 30655 .rce. docker-proxy root 30663 Frce. docker-containe (unknown) 30681 .rce. mosquitto root 30851 .rc.. kworker/20:1 root 30852 Frce. docker-containe root 30870 .rce. telegraf root 31295 .rc.. kworker/u65:0 root 31642 .rce. docker-proxy root 31649 Frce. docker-containe root 31667 .rce. s6-svscan root 31717 .rce. s6-supervise root 31854 .rc.. kworker/u65:9 root 31885 .rce. s6-supervise nobody 31888 .rce. nzbget root 32348 .rc.. kworker/u65:1 root 32482 .rc.. kworker/u68:1 root 32491 Frce. docker-containe root 32510 .r.e. s6-svscan root 32561 .r.e. s6-supervise /mnt/user: root kernel mount /mnt/user nobody 546 F...m Plex Media Serv nobody 593 F.... Plex Script Hos nobody 689 F.... mono root 713 f.... smbd nobody 790 F.... Plex Tuner Serv root 2209 F.... nginx root 2212 F.... php-fpm7 nobody 2234 F.... nginx nobody 2235 F.... nginx nobody 2236 F.... nginx nobody 2237 F.... nginx nobody 3586 F.... python nobody 5048 F...m mono nobody 5910 F...m mono fredrick 6003 f.c.. smbd nobody 7097 F...m mono nobody 8151 F...m mono nobody 10827 F.... java nobody 11856 F...m mongod (unknown) 22047 F.... grafana-server nobody 23756 F...m python root 25243 F.... supervisord nobody 25484 F.... deluged root 26581 F...m influxd root 30041 F.... nginx root 30043 F.... php-fpm7 root 30044 F.... fail2ban-client nobody 30081 F.... nginx nobody 30082 F.... nginx nobody 30083 F.... nginx nobody 30084 F.... nginx /mnt/user0: root kernel mount /mnt/user0 /mnt/cache: root kernel mount /mnt/cache root 7664 F.... shfs Edited October 19, 2017 by Fredrick Quote Link to comment
kennelm Posted November 27, 2017 Share Posted November 27, 2017 (edited) I'm having similar issues when I run Docker on unraid 6.3.5. Shfs is 100% cpu and the only solution is a hard reboot which triggers an 8-hour parity check. I've had 3 of these in the last 4 days. Edited November 27, 2017 by kennelm Quote Link to comment
Squid Posted November 27, 2017 Share Posted November 27, 2017 Change the references on all of your containers for the /config folder from /mnt/user/.... to /mnt/cache/... 2 Quote Link to comment
mbezzo Posted August 14, 2018 Share Posted August 14, 2018 @SquidI know this is a really old post, but I'm super curious as to why you say to move /config from /mnt/user to /mnt/cache/? What changes when we do this? I'm new to unRaid and trying to figure it all out still. Secondarily, (and this isn't directed at you!) if we should point there, why does unRaid point to /mnt/user by default? I'm still randomly seeing this shfs process go nuts. It's pretty random for me, but definitely happens every few days. If I just leave it alone, it seems to recover so not a huge deal... but all my dockers freeze up when this happens, so have been searching for solutions and came across this. Thanks! Quote Link to comment
Vr2Io Posted August 14, 2018 Share Posted August 14, 2018 (edited) 27 minutes ago, mbezzo said: if we should point there, why does unRaid point to /mnt/user by default? Because not everyone have cache drive setup. For my understanding, some layers i.e. SHFS ( FUSE ) was in disk pool and always perform not expect. Cache pool have better a lot performance. Edited August 14, 2018 by Benson Quote Link to comment
mbezzo Posted August 14, 2018 Share Posted August 14, 2018 ah, okay that might make more sense then. I was thinking that since I do have a cache drive, and appdata is set to cache only, that even though my paths are /mnt/user/appdata - it still is on the cache drive. aWhich is exactly why I didn't understand Squids suggestion. Hopefully that makes sense! That being said, since mine are already on the cache drive - changing to /mnt/cache won't do much or me, will it? Thanks! Quote Link to comment
John_M Posted August 14, 2018 Share Posted August 14, 2018 12 hours ago, mbezzo said: changing to /mnt/cache won't do much or me, will it? There is a subtle difference between /mnt/user/appdata and /mnt/cache/appdata even when the share is set to cache only. The former is handled by shfs while the latter isn't. Hence Squid's suggestion to bypass shfs. I thought this issue had been fixed in the meantime, though. Quote Link to comment
mbezzo Posted August 14, 2018 Share Posted August 14, 2018 3 minutes ago, John_M said: There is a subtle difference between /mnt/user/appdata and /mnt/cache/appdata even when the share is set to cache only. The former is handled by shfs while the latter isn't. Hence Squid's suggestion to bypass shfs. I thought this issue had been fixed in the meantime, though. Thanks, John - that's exactly the info I was looking for. The issue definitely seems to be much better in my case, but I've seen it several times over the past 1.5 months. It always recovers and usually never last for more than 10 mins. I'm going to try moving things to /mnt/cache and see how it goes. Thanks again! Quote Link to comment
JonathanM Posted August 14, 2018 Share Posted August 14, 2018 10 minutes ago, John_M said: I thought this issue had been fixed in the meantime, though. For the most part it has been fixed. It's broken for sure if you enable the global share settings tunable direct I/O Quote Link to comment
AgentXXL Posted June 20, 2019 Share Posted June 20, 2019 On 11/27/2017 at 5:49 AM, Squid said: Change the references on all of your containers for the /config folder from /mnt/user/.... to /mnt/cache/... @Squid I just stumbled across this gem of a tip while researching my own high CPU usage by the SHFS process.... this is a game changer for performance if your cache drive is an SSD. Made my system ultimately more useable by switching the/config folder location from /mnt/user/... to /mnt/cache/... for each of my Docker containers. I'm sure it'll work the same for any VMs I add down the road. If there's only one tip I could give to new unRAID users, this one is right at the top of the list! Thank you again! Your donation is coming shortly. This tip is especially useful for those using systems with say an older 4 core (8 threads with HT) CPU, like I am, but with the caveat that your cache drive/pool is SSD based for the best results. Quote Link to comment
unw1red Posted December 8, 2019 Share Posted December 8, 2019 I just found this thread as well. I having been dealing with the SHFS problem for the last several weeks about every other day. It required a hard reset to clear. My server is currently in this state and I was trying to troubleshoot the situation. I found a couple of things: 1. The CPU load on my dual Xeon 2.0GHz 12 core/24 thread is climbing steadily and at about >=30.00, certain Docker containers become unusable 2. If I SSH into the server and try to do any dick I/O in the /mnt disk structure (except for /mnt/cache/appdata), the terminal session locks up and has to be killed. 3. The CPU usage in UNRAID is still within normal levels (<=25% CPU) while there are a few individual cores that are peaked at 100% and do not go down. I am going to hard reboot the server and make this change in all of my Docker containers to see if this alleviates my issues. Thank you for the recommendation. Quote Link to comment
unw1red Posted December 8, 2019 Share Posted December 8, 2019 Additional question on this topic. After making the above changes, should you then set your appdata share to Cache: ONLY? Quote Link to comment
Vattar Posted March 31, 2023 Share Posted March 31, 2023 (edited) On 11/27/2017 at 10:49 AM, Squid said: Altere as referências em todos os seus contêineres para a pasta /config de /mnt/user/.... para /mnt/cache/... It works for me! Tks ------------x----------- Should I change the path from mnt/user > mnt/cache inside each container or directly in docker settings? Edited March 31, 2023 by Vattar Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.