Howdy folks,

I am downloading my entire dropbox (~15TB) about 1TB at a time.  Process:

- used Rclone to mount my dropbox

- using Krusader to copy about 1TB at a time from dropbox to my array.  

Getting crashes irregularly--sometimes nothing for four hours of copying, sometimes crashes after like 15 hours.  It's pretty annoying, because it's hard to tell where I left off. 

I'm not seeing anything interesting in syslog.  The last crash there was nothing at all logged within hours of the crash.

I have run memtest.  The build is really barebones, and I have tested all of my drives and replaced all sata cables.

 

Hardware:

i7 2700k

32GB RAM

18TB parity, 18+4 TB storage

250 GB SSD Cache

 

Any thoughts? Could this be a result of using Krusader?

 

thanks!

Edited October 11, 20232 yr by able-resume8851

If it helps at all:

1) When it crashes, it seems the machine is usually still powered on with fans spinning.  But it's not accessible on the network, and nothing shows up when I plug in a monitor directly to it.  So I end up just powering it down.

 

 

2) here was the last stuff in my syslog.  Nothing was written around when it crashed as far as i can tell.

 

Oct 10 17:45:09 JCbackup23 emhttpd: Starting services...
Oct 10 17:45:09 JCbackup23 emhttpd: shcmd (71): /etc/rc.d/rc.samba restart
Oct 10 17:45:09 JCbackup23 wsdd2[2651]: 'Terminated' signal received.
Oct 10 17:45:09 JCbackup23 winbindd[2654]: [2023/10/10 17:45:09.627843,  0] ../../source3/winbindd/winbindd_dual.c:1957(winbindd_sig_term_handler)
Oct 10 17:45:09 JCbackup23 winbindd[2654]:   Got sig[15] terminate (is_parent=1)
Oct 10 17:45:09 JCbackup23 winbindd[2660]: [2023/10/10 17:45:09.627844,  0] ../../source3/winbindd/winbindd_dual.c:1957(winbindd_sig_term_handler)
Oct 10 17:45:09 JCbackup23 winbindd[2660]:   Got sig[15] terminate (is_parent=0)
Oct 10 17:45:09 JCbackup23 wsdd2[2651]: terminating.
Oct 10 17:45:09 JCbackup23 winbindd[2829]: [2023/10/10 17:45:09.627965,  0] ../../source3/winbindd/winbindd_dual.c:1957(winbindd_sig_term_handler)
Oct 10 17:45:09 JCbackup23 winbindd[2829]:   Got sig[15] terminate (is_parent=0)
Oct 10 17:45:10 JCbackup23 rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="3782" x-info="https://www.rsyslog.com"] start
Oct 10 17:45:11 JCbackup23 root: Starting Samba:  /usr/sbin/smbd -D
Oct 10 17:45:11 JCbackup23 smbd[3886]: [2023/10/10 17:45:11.822430,  0] ../../source3/smbd/server.c:1741(main)
Oct 10 17:45:11 JCbackup23 smbd[3886]:   smbd version 4.17.7 started.
Oct 10 17:45:11 JCbackup23 smbd[3886]:   Copyright Andrew Tridgell and the Samba Team 1992-2022
Oct 10 17:45:11 JCbackup23 root:                  /usr/sbin/wsdd2 -d -4
Oct 10 17:45:11 JCbackup23 wsdd2[3900]: starting.
Oct 10 17:45:11 JCbackup23 root:                  /usr/sbin/winbindd -D
Oct 10 17:45:11 JCbackup23 winbindd[3901]: [2023/10/10 17:45:11.913502,  0] ../../source3/winbindd/winbindd.c:1440(main)
Oct 10 17:45:11 JCbackup23 winbindd[3901]:   winbindd version 4.17.7 started.
Oct 10 17:45:11 JCbackup23 winbindd[3901]:   Copyright Andrew Tridgell and the Samba Team 1992-2022
Oct 10 17:45:11 JCbackup23 winbindd[3903]: [2023/10/10 17:45:11.917535,  0] ../../source3/winbindd/winbindd_cache.c:3116(initialize_winbindd_cache)
Oct 10 17:45:11 JCbackup23 winbindd[3903]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2
Oct 10 17:45:11 JCbackup23 emhttpd: shcmd (75): /etc/rc.d/rc.avahidaemon restart
Oct 10 17:45:11 JCbackup23 root: Stopping Avahi mDNS/DNS-SD Daemon: stopped
Oct 10 17:45:11 JCbackup23 avahi-dnsconfd[2716]: read(): EOF
Oct 10 17:45:12 JCbackup23 root: Starting Avahi mDNS/DNS-SD Daemon: /usr/sbin/avahi-daemon -D
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Found user 'avahi' (UID 61) and group 'avahi' (GID 214).
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Successfully dropped root privileges.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: avahi-daemon 0.8 starting up.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Successfully called chroot().
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Successfully dropped remaining capabilities.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Loading service file /services/sftp-ssh.service.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Loading service file /services/smb.service.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Loading service file /services/ssh.service.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.0.249.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: New relevant interface br0.IPv4 for mDNS.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Network interface enumeration completed.
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Registering new address record for 192.168.0.249 on br0.IPv4.
Oct 10 17:45:12 JCbackup23 emhttpd: shcmd (76): /etc/rc.d/rc.avahidnsconfd restart
Oct 10 17:45:12 JCbackup23 root: Stopping Avahi mDNS/DNS-SD DNS Server Configuration Daemon: stopped
Oct 10 17:45:12 JCbackup23 root: Starting Avahi mDNS/DNS-SD DNS Server Configuration Daemon:  /usr/sbin/avahi-dnsconfd -D
Oct 10 17:45:12 JCbackup23 avahi-dnsconfd[3949]: Successfully connected to Avahi daemon.
Oct 10 17:45:12 JCbackup23 emhttpd: shcmd (86): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 20
Oct 10 17:45:12 JCbackup23 kernel: loop2: detected capacity change from 0 to 41943040
Oct 10 17:45:12 JCbackup23 kernel: BTRFS: device fsid f382a1a6-a5a8-41e4-8674-ec5ebcddd754 devid 1 transid 2032 /dev/loop2 scanned by mount (3983)
Oct 10 17:45:12 JCbackup23 kernel: BTRFS info (device loop2): using crc32c (crc32c-intel) checksum algorithm
Oct 10 17:45:12 JCbackup23 kernel: BTRFS info (device loop2): using free space tree
Oct 10 17:45:12 JCbackup23 kernel: BTRFS info (device loop2): enabling ssd optimizations
Oct 10 17:45:12 JCbackup23 root: Resize device id 1 (/dev/loop2) from 20.00GiB to max
Oct 10 17:45:12 JCbackup23 emhttpd: shcmd (88): /etc/rc.d/rc.docker start
Oct 10 17:45:12 JCbackup23 root: starting dockerd ...
Oct 10 17:45:12 JCbackup23 avahi-daemon[3940]: Server startup complete. Host name is JCbackup23.local. Local service cookie is 417680418.
Oct 10 17:45:13 JCbackup23 avahi-daemon[3940]: Service "JCbackup23" (/services/ssh.service) successfully established.
Oct 10 17:45:13 JCbackup23 avahi-daemon[3940]: Service "JCbackup23" (/services/smb.service) successfully established.
Oct 10 17:45:13 JCbackup23 avahi-daemon[3940]: Service "JCbackup23" (/services/sftp-ssh.service) successfully established.
Oct 10 17:45:14 JCbackup23 kernel: Bridge firewalling registered
Oct 10 17:45:14 JCbackup23 kernel: Initializing XFRM netlink socket
Oct 10 17:45:15 JCbackup23 emhttpd: shcmd (103): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1
Oct 10 17:45:15 JCbackup23 kernel: loop3: detected capacity change from 0 to 2097152
Oct 10 17:45:15 JCbackup23 kernel: BTRFS: device fsid 8261f252-709d-488d-be30-6e2835dca049 devid 1 transid 44 /dev/loop3 scanned by mount (4514)
Oct 10 17:45:15 JCbackup23 kernel: BTRFS info (device loop3): using crc32c (crc32c-intel) checksum algorithm
Oct 10 17:45:15 JCbackup23 kernel: BTRFS info (device loop3): using free space tree
Oct 10 17:45:15 JCbackup23 kernel: BTRFS info (device loop3): enabling ssd optimizations
Oct 10 17:45:15 JCbackup23 root: Resize device id 1 (/dev/loop3) from 1.00GiB to max
Oct 10 17:45:15 JCbackup23 emhttpd: shcmd (105): /etc/rc.d/rc.libvirt start
Oct 10 17:45:15 JCbackup23 root: Starting virtlockd...
Oct 10 17:45:15 JCbackup23 root: Starting virtlogd...
Oct 10 17:45:15 JCbackup23 root: Starting libvirtd...
Oct 10 17:45:15 JCbackup23 kernel: tun: Universal TUN/TAP device driver, 1.6
Oct 10 17:45:15 JCbackup23 kernel: mdcmd (36): check correct
Oct 10 17:45:15 JCbackup23 kernel: md: recovery thread: check P ...
Oct 10 17:45:15 JCbackup23 dnsmasq[4654]: started, version 2.89 cachesize 150
Oct 10 17:45:15 JCbackup23 dnsmasq[4654]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile
Oct 10 17:45:15 JCbackup23 dnsmasq-dhcp[4654]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Oct 10 17:45:15 JCbackup23 dnsmasq-dhcp[4654]: DHCP, sockets bound exclusively to interface virbr0
Oct 10 17:45:15 JCbackup23 dnsmasq[4654]: reading /etc/resolv.conf
Oct 10 17:45:15 JCbackup23 dnsmasq[4654]: using nameserver 192.168.0.1#53
Oct 10 17:45:15 JCbackup23 dnsmasq[4654]: read /etc/hosts - 3 names
Oct 10 17:45:15 JCbackup23 dnsmasq[4654]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names
Oct 10 17:45:15 JCbackup23 dnsmasq-dhcp[4654]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Oct 10 17:45:15 JCbackup23 kernel: r8169 0000:03:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
Oct 10 17:56:01 JCbackup23 emhttpd: cmd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh /tmp/user.scripts/tmpScripts/Rclone Dropbox2 Mount/script
Oct 10 17:56:20 JCbackup23 kernel: docker0: port 1(veth23cef79) entered blocking state
Oct 10 17:56:20 JCbackup23 kernel: docker0: port 1(veth23cef79) entered disabled state
Oct 10 17:56:20 JCbackup23 kernel: device veth23cef79 entered promiscuous mode
Oct 10 17:56:21 JCbackup23 kernel: eth0: renamed from veth9f7a3f9
Oct 10 17:56:21 JCbackup23 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth23cef79: link becomes ready
Oct 10 17:56:21 JCbackup23 kernel: docker0: port 1(veth23cef79) entered blocking state
Oct 10 17:56:21 JCbackup23 kernel: docker0: port 1(veth23cef79) entered forwarding state

7 hours ago, able-resume8851 said:

Any thoughts? Could this be a result of using Krusader?

Unlikely that Krusader would make the server crash, but you can try using for example rsync to rule that out, if there's nothing logged it would suggest more a hardware issue.

Any thoughts on what the hardware problem could be?  Or how to diagnose it? I've pulled pretty much everything that I would think could cause an issue.  I could try a different cache drive (or removing the cache).