bonienl

Community Developer
  • Posts

    10233
  • Joined

  • Last visited

  • Days Won

    65

Report Comments posted by bonienl

  1. 9 hours ago, Hoopster said:

    Both of my onboard NICs are Intel.  One uses the igb driver and the other uses the e1000e driver.

     

    7 hours ago, limetech said:

    Yes that is true though at present only one server has both connected to a switch,  using bonding driver - simple failover mode.

    Did you re-assign the interfaces under network rules?

     

    Only when a re-assignment is done, the file network-rules.cfg is created and a driver reload happens upon reboot (this is the point where the call trace occurs).

     

    As a test you could change the assignment of the interfaces (e.g. swap eth0 and eth1) and reboot the system.

  2. 1 hour ago, limetech said:

    Interesting, some of my test servers also use igb but do not crash:

    Are these servers with a single ethernet interface?

     

    The driver(s) reload only takes place when more than one interface exists and the file "network-rules.cfg" exists on the flash device.

  3. 6 minutes ago, johnnie.black said:

    Another one using the igb NIC driver, that's my main suspect.

    I made the capture using the mirror function of syslog.

     

    The next thing after the syslog daemon is started is the initialization of the network, starting with a reload of the network drivers. Your suspect seems very plausible.

  4. I made a capture of my system. Issues start at the line "BUG: unable to handle kernel NULL pointer..."

    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fac
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fac
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fad
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fad
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6c
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6d
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf
    Jan 23 18:19:58 vesta kernel: EDAC MC0: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#0: DEV 0000:ff:12.0 (INTERRUPT)
    Jan 23 18:19:58 vesta kernel: EDAC sbridge:  Ver: 1.1.2 
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 4 transid 37220822 /dev/sdg1
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 3 transid 37220822 /dev/sdf1
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 1 transid 37220822 /dev/sdh1
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 2 transid 37220822 /dev/sdk1
    Jan 23 18:19:58 vesta kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
    Jan 23 18:19:58 vesta kernel: PGD 1ff7e0d067 P4D 1ff7e0d067 PUD 1fbaaff067 PMD 0 
    Jan 23 18:19:58 vesta kernel: Oops: 0000 [#1] SMP NOPTI
    Jan 23 18:19:58 vesta kernel: CPU: 2 PID: 2532 Comm: modprobe Not tainted 4.19.94-Unraid #1
    Jan 23 18:19:58 vesta kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.1a 10/24/2018
    Jan 23 18:19:58 vesta kernel: RIP: 0010:kernfs_name_hash+0x9/0x6d
    Jan 23 18:19:58 vesta kernel: Code: 48 33 04 25 28 00 00 00 74 05 e8 c3 3d ea ff 48 83 c4 60 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 83 c9 ff 31 c0 49 89 f8 <f2> ae 48 f7 d1 8d 79 ff 31 c9 48 39 cf 74 1f 49 0f be 04 08 48 ff
    Jan 23 18:19:58 vesta kernel: RSP: 0018:ffffc90006b27cb8 EFLAGS: 00010246
    Jan 23 18:19:58 vesta kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffffff
    Jan 23 18:19:58 vesta kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
    Jan 23 18:19:58 vesta kernel: RBP: ffff889fcab65198 R08: 0000000000000000 R09: ffffffff811ac100
    Jan 23 18:19:58 vesta kernel: R10: ffffea007f2ad900 R11: ffff889fff075001 R12: ffff889fcab65348
    Jan 23 18:19:58 vesta kernel: R13: 0000000000000000 R14: ffff889ff3a6c190 R15: 0000000000000000
    Jan 23 18:19:58 vesta kernel: FS:  0000150ae6c37b80(0000) GS:ffff889fff680000(0000) knlGS:0000000000000000
    Jan 23 18:19:58 vesta kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Jan 23 18:19:58 vesta kernel: CR2: 0000000000000000 CR3: 0000001ff3b5a002 CR4: 00000000003606e0
    Jan 23 18:19:58 vesta kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Jan 23 18:19:58 vesta kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    Jan 23 18:19:58 vesta kernel: Call Trace:
    Jan 23 18:19:58 vesta kernel: kernfs_find_ns+0x5e/0xa8
    Jan 23 18:19:58 vesta kernel: kernfs_remove_by_name_ns+0x49/0x75
    Jan 23 18:19:58 vesta kernel: remove_files+0x38/0x5a
    Jan 23 18:19:58 vesta kernel: sysfs_remove_group+0x55/0x6f
    Jan 23 18:19:58 vesta kernel: sysfs_remove_groups+0x28/0x2f
    Jan 23 18:19:58 vesta kernel: device_remove_attrs+0x33/0x63
    Jan 23 18:19:58 vesta kernel: device_del+0x18d/0x2ed
    Jan 23 18:19:58 vesta kernel: cdev_device_del+0x10/0x26
    Jan 23 18:19:58 vesta kernel: posix_clock_unregister+0x1c/0x41
    Jan 23 18:19:58 vesta kernel: ptp_clock_unregister+0x69/0x6d
    Jan 23 18:19:58 vesta kernel: igb_ptp_stop+0x1a/0x44 [igb]
    Jan 23 18:19:58 vesta kernel: igb_remove+0x39/0xfd [igb]
    Jan 23 18:19:58 vesta kernel: pci_device_remove+0x36/0x8e
    Jan 23 18:19:58 vesta kernel: device_release_driver_internal+0x144/0x225
    Jan 23 18:19:58 vesta kernel: driver_detach+0x6d/0x77
    Jan 23 18:19:58 vesta kernel: bus_remove_driver+0x60/0x7c
    Jan 23 18:19:58 vesta kernel: pci_unregister_driver+0x1c/0x7f
    Jan 23 18:19:58 vesta kernel: __se_sys_delete_module+0x10f/0x1ac
    Jan 23 18:19:58 vesta kernel: ? exit_to_usermode_loop+0x55/0xa2
    Jan 23 18:19:58 vesta kernel: do_syscall_64+0x57/0xf2
    Jan 23 18:19:58 vesta kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
    Jan 23 18:19:58 vesta kernel: RIP: 0033:0x150ae6d72047
    Jan 23 18:19:58 vesta kernel: Code: 73 01 c3 48 8b 0d 49 7e 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 19 7e 0c 00 f7 d8 64 89 01 48
    Jan 23 18:19:58 vesta kernel: RSP: 002b:00007ffc617bb338 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
    Jan 23 18:19:58 vesta kernel: RAX: ffffffffffffffda RBX: 0000000000427cf0 RCX: 0000150ae6d72047
    Jan 23 18:19:58 vesta kernel: RDX: 0000000000000000 RSI: 0000000000000800 RDI: 0000000000427d58
    Jan 23 18:19:58 vesta kernel: RBP: 0000000000427d58 R08: 1999999999999999 R09: 0000000000000000
    Jan 23 18:19:58 vesta kernel: R10: 0000150ae6de9ac0 R11: 0000000000000206 R12: 0000000000000000
    Jan 23 18:19:58 vesta kernel: R13: 0000000000000000 R14: 0000000000427d58 R15: 0000000000425480
    Jan 23 18:19:58 vesta kernel: Modules linked in: sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore intel_rapl_perf i2c_i801 ipmi_ssif igb(-) i2c_algo_bit i2c_core smartpqi ahci libahci scsi_transport_sas wmi pcc_cpufreq ipmi_si button [last unloaded: atlantic]
    Jan 23 18:19:58 vesta kernel: CR2: 0000000000000000
    Jan 23 18:19:58 vesta kernel: ---[ end trace bfd77ca2011f6527 ]---
    Jan 23 18:19:58 vesta kernel: RIP: 0010:kernfs_name_hash+0x9/0x6d
    Jan 23 18:19:58 vesta kernel: Code: 48 33 04 25 28 00 00 00 74 05 e8 c3 3d ea ff 48 83 c4 60 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 83 c9 ff 31 c0 49 89 f8 <f2> ae 48 f7 d1 8d 79 ff 31 c9 48 39 cf 74 1f 49 0f be 04 08 48 ff
    Jan 23 18:19:58 vesta kernel: RSP: 0018:ffffc90006b27cb8 EFLAGS: 00010246
    Jan 23 18:19:58 vesta kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffffff
    Jan 23 18:19:58 vesta kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
    Jan 23 18:19:58 vesta kernel: RBP: ffff889fcab65198 R08: 0000000000000000 R09: ffffffff811ac100
    Jan 23 18:19:58 vesta kernel: R10: ffffea007f2ad900 R11: ffff889fff075001 R12: ffff889fcab65348
    Jan 23 18:19:58 vesta kernel: R13: 0000000000000000 R14: ffff889ff3a6c190 R15: 0000000000000000
    Jan 23 18:19:58 vesta kernel: FS:  0000150ae6c37b80(0000) GS:ffff889fff680000(0000) knlGS:0000000000000000
    Jan 23 18:19:58 vesta kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Jan 23 18:19:58 vesta kernel: CR2: 0000000000000000 CR3: 0000001ff3b5a002 CR4: 00000000003606e0
    Jan 23 18:19:58 vesta kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Jan 23 18:19:58 vesta kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    Jan 23 18:19:59 vesta rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="2507" x-info="https://www.rsyslog.com"] start 

    After the call trace the system hangs, and I need to do a power off / power on to get a "proper" boot up

    • Like 1
  5. I experience the exact same call trace and subsequent hanging of the system.

    To me, it looks like an issue with reading the USB device.

     

    To get around the issue, I need to completely power-off and power-on the system.

     

    The line

    /etc/rc.d/rc.M: line 164:   modprobe -r $DRIVERS

    is the point at which all network drivers are removed and then re-assigned according to the "interface rules", this could play a role too

     

  6. 22 minutes ago, SliMat said:

    Urgent seemed appropriate as Urgent shows "Server crash, data loss, or other showstopper"

    You need to see this in the grand scheme of things.

    I understand it is urgent to you, but the problem is limited to your specific hardware.

    Anything marked "urgent" requires immediate attention because it potentially impacts a large portion of users, which isn't the case here.

     

    A good start for situations like this, is "minor" and since there is a workaround now "annoyance" is appropriate.

  7. 9 hours ago, Vynce said:

    I've copied the benchmark script to a gist at @ljm42's suggestion: https://gist.github.com/Vynce/44f224c2846de5fa4cf1d5b1dcad2dc4. Anyone is welcome to hack on it as they like 😊.

    Thanks for your script. Out of curiosity I made a test too on my main server.

     

    I do not experience any noticable slowdowns when operating the array under 6.8.1, the biggest folder I have has 2000+ items.

    # bash benchmark_shfs.sh
    Unraid 6.8.1
    Hard Link support: no
    
    100000 files
    Writing files
    Benchmarking disk: 0.29
    Benchmarking SHFS: 3.46
    
    200000 files
    Writing files
    Benchmarking disk: 0.59
    Benchmarking SHFS: 9.48
    
    300000 files
    Writing files
    Benchmarking disk: 0.88
    Benchmarking SHFS: 20.75
    # bash benchmark_shfs.sh
    Unraid 6.8.1
    Hard Link support: yes
    
    100000 files
    Writing files
    Benchmarking disk: 0.25
    Benchmarking SHFS: 10.14
    
    200000 files
    Writing files
    Benchmarking disk: 0.50
    Benchmarking SHFS: 22.67
    
    300000 files
    Writing files
    Benchmarking disk: 0.77
    Benchmarking SHFS: 39.73