• [6.11.0] No space left on device


    nexusmaniac
    • Minor

     

    Dockers seem to be working, I can access SMB shares but disks vanish from the Dashboard & Main tab and browsing files through the Unraid UI brings up errors but appears to mostly work (I think)

     

    Warning: syntax error, unexpected end of file, expecting TC_DOLLAR_CURLY or TC_QUOTED_STRING or '"' in state/disks.ini on line 240 in /usr/local/emhttp/plugins/dynamix/template.php on line 36
    
    Warning: syntax error, unexpected end of file, expecting TC_DOLLAR_CURLY or TC_QUOTED_STRING or '"' in state/disks.ini on line 240 in /usr/local/emhttp/plugins/dynamix/include/Browse.php on line 44

     

    Sadly, I'm unable to get a copy of the syslog or diagnostics – they both 404 😔

     

    image.thumb.png.6417dead91f4c12cca6c7d6f80e0f8f2.png

     

    And here's what the dashboard looks like:

     

    image.thumb.png.736b51057250fce0a5f8ae74319a6ea7.png

     

    Sep 28 01:16:49 Raptor CA Backup/Restore: #######################
    Sep 28 01:16:49 Raptor CA Backup/Restore: appData Backup complete
    Sep 28 01:16:49 Raptor CA Backup/Restore: #######################
    Sep 28 01:16:49 Raptor CA Backup/Restore: Deleting /mnt/user/Backups/unRAID/Appdata/[email protected]
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: Joining mDNS multicast group on interface vethb85f084.IPv6 with address fe80::8c65:47ff:fe95:b3f.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: New relevant interface vethb85f084.IPv6 for mDNS.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: Registering new address record for fe80::8c65:47ff:fe95:b3f on vethb85f084.*.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: Joining mDNS multicast group on interface veth2debf98.IPv6 with address fe80::e87a:86ff:fe47:5f0b.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: New relevant interface veth2debf98.IPv6 for mDNS.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: Registering new address record for fe80::e87a:86ff:fe47:5f0b on veth2debf98.*.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: Joining mDNS multicast group on interface veth715e3c6.IPv6 with address fe80::f441:50ff:fe0d:8aec.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: New relevant interface veth715e3c6.IPv6 for mDNS.
    Sep 28 01:16:49 Raptor  avahi-daemon[3790]: Registering new address record for fe80::f441:50ff:fe0d:8aec on veth715e3c6.*.
    Sep 28 01:16:50 Raptor  avahi-daemon[3790]: Joining mDNS multicast group on interface veth0c8a49f.IPv6 with address fe80::30ce:daff:fe4e:4dcd.
    Sep 28 01:16:50 Raptor  avahi-daemon[3790]: New relevant interface veth0c8a49f.IPv6 for mDNS.
    Sep 28 01:16:50 Raptor  avahi-daemon[3790]: Registering new address record for fe80::30ce:daff:fe4e:4dcd on veth0c8a49f.*.
    Sep 28 01:16:50 Raptor  avahi-daemon[3790]: Joining mDNS multicast group on interface veth9de59c5.IPv6 with address fe80::fc99:8dff:fe8e:aa1d.
    Sep 28 01:16:50 Raptor  avahi-daemon[3790]: New relevant interface veth9de59c5.IPv6 for mDNS.
    Sep 28 01:16:50 Raptor  avahi-daemon[3790]: Registering new address record for fe80::fc99:8dff:fe8e:aa1d on veth9de59c5.*.
    Sep 28 01:17:01 Raptor CA Backup/Restore: Backup / Restore Completed
    Sep 28 01:17:01 Raptor  emhttpd: read SMART /dev/sdb
    Sep 28 01:17:01 Raptor  emhttpd: read SMART /dev/sdc
    Sep 28 01:17:02 Raptor  emhttpd: read SMART /dev/sdk
    Sep 28 01:17:02 Raptor  emhttpd: read SMART /dev/sdi
    Sep 28 01:17:09 Raptor  emhttpd: read SMART /dev/sdh
    Sep 28 01:17:16 Raptor  emhttpd: read SMART /dev/sdg
    Sep 28 01:17:24 Raptor  emhttpd: read SMART /dev/sdl
    Sep 28 01:17:35 Raptor  emhttpd: read SMART /dev/sde
    Sep 28 01:17:47 Raptor  emhttpd: read SMART /dev/sdd
    Sep 28 01:46:46 Raptor  emhttpd: spinning down /dev/sdj
    Sep 28 01:47:03 Raptor  emhttpd: spinning down /dev/sdk
    Sep 28 01:47:50 Raptor  emhttpd: spinning down /dev/sdd
    Sep 28 01:47:50 Raptor  emhttpd: spinning down /dev/sdc
    Sep 28 01:47:53 Raptor  emhttpd: spinning down /dev/sdi
    Sep 28 01:47:55 Raptor  emhttpd: spinning down /dev/sdh
    Sep 28 01:47:56 Raptor  emhttpd: spinning down /dev/sdg
    Sep 28 01:47:56 Raptor  emhttpd: spinning down /dev/sdf
    Sep 28 01:47:56 Raptor  emhttpd: spinning down /dev/sdl
    Sep 28 01:48:18 Raptor  emhttpd: spinning down /dev/sde
    Sep 28 01:48:18 Raptor  emhttpd: spinning down /dev/sdb
    Sep 28 02:04:18 Raptor  emhttpd: read SMART /dev/sdl
    Sep 28 02:04:30 Raptor  emhttpd: read SMART /dev/sdb
    Sep 28 02:04:42 Raptor  emhttpd: read SMART /dev/sdd
    Sep 28 02:04:48 Raptor  emhttpd: read SMART /dev/sdh
    Sep 28 02:04:56 Raptor  emhttpd: read SMART /dev/sdc
    Sep 28 02:05:03 Raptor  emhttpd: read SMART /dev/sdf
    Sep 28 02:05:16 Raptor  emhttpd: read SMART /dev/sdg
    Sep 28 02:05:24 Raptor  emhttpd: read SMART /dev/sdi
    Sep 28 02:56:49 Raptor  emhttpd: spinning down /dev/sdc
    Sep 28 02:56:59 Raptor  emhttpd: spinning down /dev/sdh
    Sep 28 02:56:59 Raptor  emhttpd: spinning down /dev/sdg
    Sep 28 02:57:00 Raptor  emhttpd: spinning down /dev/sdb
    Sep 28 02:57:00 Raptor  emhttpd: spinning down /dev/sdl
    Sep 28 02:57:00 Raptor  emhttpd: spinning down /dev/sdi
    Sep 28 02:57:02 Raptor  emhttpd: spinning down /dev/sdd
    Sep 28 02:57:49 Raptor  emhttpd: spinning down /dev/sdf
    Sep 28 03:39:12 Raptor  emhttpd: read SMART /dev/sdl
    Sep 28 03:40:12 Raptor  emhttpd: read SMART /dev/sdb
    Sep 28 03:40:29 Raptor  emhttpd: read SMART /dev/sdh
    Sep 28 03:40:37 Raptor  emhttpd: read SMART /dev/sdc
    Sep 28 03:40:46 Raptor  emhttpd: read SMART /dev/sdi
    Sep 28 03:41:19 Raptor  emhttpd: read SMART /dev/sdg
    Sep 28 03:41:41 Raptor  emhttpd: read SMART /dev/sdf
    Sep 28 04:11:06 Raptor  emhttpd: spinning down /dev/sdc
    Sep 28 04:11:11 Raptor  emhttpd: spinning down /dev/sdl
    Sep 28 04:11:13 Raptor  emhttpd: spinning down /dev/sdi
    Sep 28 04:11:33 Raptor  emhttpd: spinning down /dev/sdg
    Sep 28 04:11:36 Raptor  emhttpd: spinning down /dev/sdh
    Sep 28 04:11:58 Raptor  emhttpd: spinning down /dev/sdf
    Sep 28 04:12:00 Raptor  emhttpd: spinning down /dev/sdb
    Sep 28 08:00:01 Raptor  emhttpd: read SMART /dev/sdk
    Sep 28 08:00:02 Raptor  emhttpd: read SMART /dev/sdj
    Sep 28 08:00:20 Raptor root: /mnt/staging: 5.7 GiB (6158344192 bytes) trimmed on /dev/sdk1
    Sep 28 08:00:20 Raptor root: /mnt/scratch: 11.1 GiB (11889082368 bytes) trimmed on /dev/sdj1
    Sep 28 08:00:20 Raptor root: /mnt/cache: 8 GiB (8593104896 bytes) trimmed on /dev/nvme0n1p1
    Sep 28 08:27:02 Raptor  emhttpd: read SMART /dev/sdf
    Sep 28 08:27:14 Raptor  emhttpd: read SMART /dev/sdb
    Sep 28 08:30:03 Raptor  emhttpd: spinning down /dev/sdj
    Sep 28 08:30:03 Raptor  emhttpd: spinning down /dev/sdk
    Sep 28 08:57:15 Raptor  emhttpd: spinning down /dev/sdb
    Sep 28 08:57:27 Raptor  emhttpd: spinning down /dev/sdf
    Sep 28 08:59:02 Raptor kernel: vethff49b5f: renamed from eth0
    Sep 28 09:29:01 Raptor  emhttpd: read SMART /dev/sdb
    Sep 28 09:29:13 Raptor  emhttpd: read SMART /dev/sdh
    Sep 28 09:29:23 Raptor  emhttpd: read SMART /dev/sdi
    Sep 28 09:29:31 Raptor  emhttpd: read SMART /dev/sdd
    Sep 28 09:29:31 Raptor  emhttpd: read SMART /dev/sdc
    Sep 28 09:29:46 Raptor  emhttpd: read SMART /dev/sdg
    Sep 28 09:30:08 Raptor  emhttpd: read SMART /dev/sdf
    Sep 28 09:30:27 Raptor  emhttpd: read SMART /dev/sdl
    Sep 28 10:00:56 Raptor  emhttpd: spinning down /dev/sdh
    Sep 28 10:01:04 Raptor  emhttpd: spinning down /dev/sdf
    Sep 28 10:01:04 Raptor  emhttpd: spinning down /dev/sdl
    Sep 28 10:01:06 Raptor  emhttpd: spinning down /dev/sdg
    Sep 28 10:01:06 Raptor  emhttpd: spinning down /dev/sdd
    Sep 28 10:01:06 Raptor  emhttpd: spinning down /dev/sdb
    Sep 28 10:01:06 Raptor  emhttpd: spinning down /dev/sdi
    Sep 28 10:01:10 Raptor  emhttpd: spinning down /dev/sdc
    Sep 28 11:38:24 Raptor kernel: eth0: renamed from vethc7d238e
    Sep 28 12:14:00 Raptor kernel: docker0: port 2(veth3105c7e) entered disabled state
    Sep 28 12:14:00 Raptor kernel: vethf8bf137: renamed from eth0
    Sep 28 12:14:00 Raptor  avahi-daemon[3790]: Interface veth3105c7e.IPv6 no longer relevant for mDNS.
    Sep 28 12:14:00 Raptor  avahi-daemon[3790]: Leaving mDNS multicast group on interface veth3105c7e.IPv6 with address fe80::1812:b1ff:fefc:2a1e.
    Sep 28 12:14:00 Raptor kernel: docker0: port 2(veth3105c7e) entered disabled state
    Sep 28 12:14:00 Raptor kernel: device veth3105c7e left promiscuous mode
    Sep 28 12:14:00 Raptor kernel: docker0: port 2(veth3105c7e) entered disabled state
    Sep 28 12:14:00 Raptor  avahi-daemon[3790]: Withdrawing address record for fe80::1812:b1ff:fefc:2a1e on veth3105c7e.
    Sep 28 12:14:00 Raptor kernel: docker0: port 2(veth387de0f) entered blocking state
    Sep 28 12:14:00 Raptor kernel: docker0: port 2(veth387de0f) entered disabled state
    Sep 28 12:14:00 Raptor kernel: device veth387de0f entered promiscuous mode
    Sep 28 12:14:00 Raptor kernel: docker0: port 2(veth387de0f) entered blocking state
    Sep 28 12:14:00 Raptor kernel: docker0: port 2(veth387de0f) entered forwarding state
    Sep 28 12:14:00 Raptor kernel: eth0: renamed from vethc4c3331
    Sep 28 12:14:00 Raptor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth387de0f: link becomes ready
    Sep 28 12:14:02 Raptor  avahi-daemon[3790]: Joining mDNS multicast group on interface veth387de0f.IPv6 with address fe80::6c52:b2ff:fe8a:632b.
    Sep 28 12:14:02 Raptor  avahi-daemon[3790]: New relevant interface veth387de0f.IPv6 for mDNS.
    Sep 28 12:14:02 Raptor  avahi-daemon[3790]: Registering new address record for fe80::6c52:b2ff:fe8a:632b on veth387de0f.*.
    Sep 28 12:14:05 Raptor kernel: docker0: port 2(veth387de0f) entered disabled state
    Sep 28 12:14:05 Raptor kernel: vethc4c3331: renamed from eth0
    Sep 28 12:14:05 Raptor  avahi-daemon[3790]: Interface veth387de0f.IPv6 no longer relevant for mDNS.
    Sep 28 12:14:05 Raptor  avahi-daemon[3790]: Leaving mDNS multicast group on interface veth387de0f.IPv6 with address fe80::6c52:b2ff:fe8a:632b.
    Sep 28 12:14:05 Raptor kernel: docker0: port 2(veth387de0f) entered disabled state
    Sep 28 12:14:05 Raptor kernel: device veth387de0f left promiscuous mode
    Sep 28 12:14:05 Raptor kernel: docker0: port 2(veth387de0f) entered disabled state
    Sep 28 12:14:06 Raptor  avahi-daemon[3790]: Withdrawing address record for fe80::6c52:b2ff:fe8a:632b on veth387de0f.
    Sep 28 12:14:06 Raptor kernel: docker0: port 2(veth0a58b51) entered blocking state
    Sep 28 12:14:06 Raptor kernel: docker0: port 2(veth0a58b51) entered disabled state
    Sep 28 12:14:06 Raptor kernel: device veth0a58b51 entered promiscuous mode
    Sep 28 12:14:06 Raptor kernel: docker0: port 2(veth0a58b51) entered blocking state
    Sep 28 12:14:06 Raptor kernel: docker0: port 2(veth0a58b51) entered forwarding state
    Sep 28 12:14:06 Raptor kernel: eth0: renamed from veth1e8f41d
    Sep 28 12:14:06 Raptor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0a58b51: link becomes ready
    Sep 28 12:14:08 Raptor  avahi-daemon[3790]: Joining mDNS multicast group on interface veth0a58b51.IPv6 with address fe80::c0f7:85ff:fea7:7ab1.
    Sep 28 12:14:08 Raptor  avahi-daemon[3790]: New relevant interface veth0a58b51.IPv6 for mDNS.
    Sep 28 12:14:08 Raptor  avahi-daemon[3790]: Registering new address record for fe80::c0f7:85ff:fea7:7ab1 on veth0a58b51.*.
    Sep 28 13:06:02 Raptor  emhttpd: read SMART /dev/sdl
    Sep 28 13:16:53 Raptor  emhttpd: read SMART /dev/sdf
    Sep 28 13:17:05 Raptor  emhttpd: read SMART /dev/sdb
    Sep 28 13:17:18 Raptor  emhttpd: read SMART /dev/sdg
    Sep 28 13:17:25 Raptor  emhttpd: read SMART /dev/sdh
    Sep 28 13:17:34 Raptor  emhttpd: read SMART /dev/sdi
    Sep 28 13:18:13 Raptor  emhttpd: read SMART /dev/sdc
    Sep 28 13:47:18 Raptor  emhttpd: spinning down /dev/sdb
    Sep 28 13:47:36 Raptor  emhttpd: spinning down /dev/sdh
    Sep 28 13:47:36 Raptor  emhttpd: spinning down /dev/sdi
    Sep 28 13:47:43 Raptor  emhttpd: spinning down /dev/sdg
    Sep 28 13:48:14 Raptor  emhttpd: spinning down /dev/sdc
    Sep 28 14:10:11 Raptor  emhttpd: spinning down /dev/sdf
    Sep 28 14:32:26 Raptor  emhttpd: read SMART /dev/sdk
    Sep 28 15:02:46 Raptor  emhttpd: spinning down /dev/sdk
    Sep 28 16:22:20 Raptor  emhttpd: read SMART /dev/sdb
    Sep 28 16:26:12 Raptor nginx: 2022/09/28 16:26:12 [crit] 16469#16469: *2270300 pwritev() "/var/lib/nginx/client_body/0000223547" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:26:27 Raptor nginx: 2022/09/28 16:26:27 [crit] 16469#16469: *2270445 pwritev() "/var/lib/nginx/client_body/0000223555" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:27:01 Raptor nginx: 2022/09/28 16:27:01 [crit] 16469#16469: *2270781 pwrite() "/var/lib/nginx/client_body/0000223571" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:27:22 Raptor nginx: 2022/09/28 16:27:22 [crit] 16469#16469: *2270944 pwritev() "/var/lib/nginx/client_body/0000223572" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:27:33 Raptor nginx: 2022/09/28 16:27:33 [crit] 16469#16469: *2271044 pwritev() "/var/lib/nginx/client_body/0000223573" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:30:09 Raptor nginx: 2022/09/28 16:30:09 [crit] 16469#16469: *2272423 pwritev() "/var/lib/nginx/client_body/0000223574" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:30:11 Raptor unraid-api[7523]: ⚠️ Caught exception: Maximum call stack size exceeded
    Sep 28 16:30:11 Raptor unraid-api[7523]: ⚠️ UNRAID API crashed with exit code 1
    Sep 28 16:30:51 Raptor nginx: 2022/09/28 16:30:51 [crit] 16469#16469: *2272850 pwritev() "/var/lib/nginx/client_body/0000223575" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:30:52 Raptor nginx: 2022/09/28 16:30:52 [crit] 16469#16469: *2272855 pwritev() "/var/lib/nginx/client_body/0000223576" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:31:01 Raptor  winbindd[3762]: [2022/09/28 16:31:01.482777,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:31:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:32:01 Raptor  winbindd[3762]: [2022/09/28 16:32:01.565444,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:32:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:33:01 Raptor  winbindd[3762]: [2022/09/28 16:33:01.646888,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:33:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:34:01 Raptor  winbindd[3762]: [2022/09/28 16:34:01.703275,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:34:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:34:19 Raptor nginx: 2022/09/28 16:34:19 [crit] 16469#16469: *2275193 pwritev() "/var/lib/nginx/client_body/0000223577" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:35:01 Raptor  winbindd[3762]: [2022/09/28 16:35:01.785317,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:35:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:36:01 Raptor  winbindd[3762]: [2022/09/28 16:36:01.872562,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:36:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:36:56 Raptor  winbindd[3762]: [2022/09/28 16:36:56.363827,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:36:56 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:37:01 Raptor  winbindd[3762]: [2022/09/28 16:37:01.948671,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:37:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:37:13 Raptor nginx: 2022/09/28 16:37:13 [crit] 16469#16469: *2277199 pwritev() "/var/lib/nginx/client_body/0000223578" has written only 8192 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:38:01 Raptor  winbindd[3762]: [2022/09/28 16:38:01.034477,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:38:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:38:14 Raptor nginx: 2022/09/28 16:38:14 [crit] 16469#16469: *2277811 pwritev() "/var/lib/nginx/client_body/0000223583" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:38:16 Raptor nginx: 2022/09/28 16:38:16 [crit] 16469#16469: *2277832 pwrite() "/var/lib/nginx/client_body/0000223584" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:38:38 Raptor nginx: 2022/09/28 16:38:38 [crit] 16469#16469: *2277999 pwritev() "/var/lib/nginx/client_body/0000223585" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:39:01 Raptor  winbindd[3762]: [2022/09/28 16:39:01.123008,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:39:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:40:01 Raptor  winbindd[3762]: [2022/09/28 16:40:01.210119,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:40:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:40:02 Raptor nginx: 2022/09/28 16:40:02 [crit] 16469#16469: *2278742 pwritev() "/var/lib/nginx/client_body/0000223586" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:40:18 Raptor nginx: 2022/09/28 16:40:18 [crit] 16469#16469: *2278886 pwritev() "/var/lib/nginx/client_body/0000223587" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:41:01 Raptor  winbindd[3762]: [2022/09/28 16:41:01.284950,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:41:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:41:14 Raptor nginx: 2022/09/28 16:41:14 [crit] 16469#16469: *2279381 pwritev() "/var/lib/nginx/client_body/0000223588" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:42:01 Raptor  winbindd[3762]: [2022/09/28 16:42:01.356158,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:42:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:43:01 Raptor  winbindd[3762]: [2022/09/28 16:43:01.440045,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:43:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:44:01 Raptor  winbindd[3762]: [2022/09/28 16:44:01.514730,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:44:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:45:01 Raptor  winbindd[3762]: [2022/09/28 16:45:01.599798,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:45:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:46:01 Raptor  winbindd[3762]: [2022/09/28 16:46:01.687302,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:46:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:46:19 Raptor nginx: 2022/09/28 16:46:19 [crit] 16469#16469: *2282696 pwritev() "/var/lib/nginx/client_body/0000223589" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:46:29 Raptor root: error: /plugins/community.applications/scripts/notices.php: wrong csrf_token
    Sep 28 16:46:29 Raptor root: error: /webGui/include/Report.php: wrong csrf_token
    Sep 28 16:46:29 Raptor root: error: /webGui/include/FileTree.php: wrong csrf_token
    Sep 28 16:46:59 Raptor  winbindd[3762]: [2022/09/28 16:46:59.387775,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:46:59 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:47:00 Raptor unraid-api[22543]: ✔️ UNRAID API started successfully!
    Sep 28 16:47:01 Raptor  winbindd[3762]: [2022/09/28 16:47:01.771620,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:47:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:47:01 Raptor  winbindd[3762]: [2022/09/28 16:47:01.794217,  0] ../../source3/winbindd/winbindd_samr.c:71(open_internal_samr_conn)
    Sep 28 16:47:01 Raptor  winbindd[3762]:   open_internal_samr_conn: Could not connect to samr pipe: NT_STATUS_CONNECTION_DISCONNECTED
    Sep 28 16:47:01 Raptor nginx: 2022/09/28 16:47:01 [crit] 16469#16469: *2283241 pwritev() "/var/lib/nginx/client_body/0000223620" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:47:10 Raptor nginx: 2022/09/28 16:47:10 [crit] 16469#16469: *2283344 pwritev() "/var/lib/nginx/client_body/0000223625" has written only 4096 of 9068, client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
    Sep 28 16:47:12 Raptor nginx: 2022/09/28 16:47:12 [crit] 16469#16469: *2283370 pwrite() "/var/lib/nginx/client_body/0000223626" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"



    User Feedback

    Recommended Comments

    The unraid-api crashing is another symptom of the problems, not the cause.

     

    The "wrong csrf_token" messages in the log mean you have another window/tab open somewhere that needs to be closed.

     

    Open a web terminal (or use SSH, or a local keyboard/mouse) and type "diagnostics".  Note the path on the flash drive where it writes the zip file, you can either access it over the network or you can shutdown and put the flash drive in another computer to upload the diagnostics zip file here.

    Link to comment

    By that point in the logs (the first occurrence of csrf token), unraid had already caught fire, so to speak. I had a couple of tabs open on the same machine after seeing all of the issues. Closing all of them and starting afresh stopped any further csrf warnings but didn't predate the 'undetermined' state of the array, nor the vanishing disks on the dashboard/main tabs.

     

    I'll have to wait for it to happen again, unfortunately 😭 I rebooted so that I could be sure all of my overnight actions would run successfully (appdata & usb backup, mover, trim, etc.) Completely forgot about manual diagnostics via SSH 🤦‍♀️ it's been a long while since anything has gone horribly wrong 😅 which is a silver lining I suppose!

     

    • Like 1
    Link to comment

    It's happened again 👀

     

    We can rule out csrf_token as the cause too, that's another symptom of the problem, it would seem 😰

     

    • Typing diagnostics command runs but it's very upset (output & ZIP attached... It's just empty files because nothing can be written to the destination).
    • Lots of mentions of smbd & NT_STATUS_DISK_FULL
      • The boot USB has 745MB used, so that's not full
      • Memory is being heavily used but it's cached memory rather than processes so that's not actually 'full'
    • Dockers are still mostly up - they just struggle to write to anything
    • Disks are all detected, accessible and have 10's of GB of free space (50GB+ on all disks with ~87% usage on the entire array)

     

    I've attached some files with various outputs but I can't get diagnostics to run or save so that's a no-go unfortunately 😔

    diagnostics.txt free-m.txt syslog.txt diagnostics.zip

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.