• [6.8.1] crash on boot (sometimes)


    JorgeB
    • Solved Minor

    There are been several reports of serves crashing on boot after updating to v6.8.1, and they all crash in the same place:

     

    /etc/rc.d/rc.M: line 164:   modprobe -r $DRIVERS

     

    Some examples

     

    https://forums.unraid.net/topic/87378-server-fails-to-start-after-update/

    https://forums.unraid.net/topic/87435-system-hang-681/

    https://forums.unraid.net/topic/87664-sometimes-my-unraid-isn’t-starting-since-last-update/

     

    I just had the same happening to me after upgrading one of my servers to v6.8.1:

     

    iKVM_capture.jpg.63546531bbd2d37540b34bd53115223a.jpg

     

    Rebooted and it crashed again:

     

    26682209_iKVM_capture(1).jpg.89bec61a909f4369c31bfd11b1b61d92.jpg

     

    Ran chkdsk but no errors found:

     

    1670611132_flasherror1.PNG.296176ca39ed9c3f37678083bd8f4210.PNG

    729240766_flasherror2.PNG.c8518c4aea818ef4c30e4c64289b5e2d.PNG

     

    Rebooted again and this time server started, rebooted two times successfully and then it crashed again on the 2 next ones, one more attempt and again it booted again correctly, attaching diags more to see the hardware used, but other reports are with very different hardware, AMD and Intel based servers.

     

    It's strange because it only happens sometimes, my last boots:

     

    X X (ran chkdsk) V V V X X V

     

    X - crashed

    V - booted

     

    tower5-diagnostics-20200123-0752.zip

     

    Video showing where it crashes:

     

    This server uses

    Edited by johnnie.black




    User Feedback

    Recommended Comments



    Most of the similar reports I remember reading were using LSI HBA(s), myself included, so to rule that out anyone having this problem not using one?

    Link to comment
    20 minutes ago, Stripe said:

    All disks attached directly to mobo.

    Just to make sure, using just the onboard Intel/AMD SATA ports correct? Since a few board have an onboard LSI controller.

    Link to comment

    Just another .02 cents here - are all the people reporting the issue using plug-ins? I use the unassigned drive plug-in among others.... It may be a coincidence, but after my 1st crash I restarted many times (w/out plug-in support from the boot gui) and never saw the error or crash at boot. I've only seen the crash on boot when regular full startup (w/plugins) happens.

    Link to comment
    4 minutes ago, Ruby-Rube said:

    I use the unassigned drive plug-in

    Can't be that one as it's not installed on the server I had the problem.

    Link to comment
    17 minutes ago, Ruby-Rube said:

    Maybe Plug-ins being turned on in general?

    I didn't test myself, though I have very few plugins installed on that server, but according to at least one user in one of the threads I linked above it also happens in safe mode without any plugins.

     

     

     

    Link to comment
    54 minutes ago, johnnie.black said:

    Just to make sure, using just the onboard Intel/AMD SATA ports correct? Since a few board have an onboard LSI controller.

    Yes. I have H370M-ITX/ac motherboard. I have UD plugin installed among others but I've never experienced such problems with earlier versions of unRaid.

    Link to comment
    2 minutes ago, Stripe said:

    Yes. I have H370M-ITX/ac motherboard

    Thanks, so we can rule out LSI, I also don't think it's plugin related, strange thing is happening with very different hardware.

    Link to comment

    Looking at the various diags/board models used I can only see one thing in common, Intel gigabit NIC, anyone having this issue not using an Intel NIC that loads the igb driver?

     

    Alternatively, and I can't do it now, but if anyone else affected try disabling the Intel NIC on your BIOS and see if the crash still happens.

     

     

     

     

     

    Link to comment

    I experience the exact same call trace and subsequent hanging of the system.

    To me, it looks like an issue with reading the USB device.

     

    To get around the issue, I need to completely power-off and power-on the system.

     

    The line

    /etc/rc.d/rc.M: line 164:   modprobe -r $DRIVERS

    is the point at which all network drivers are removed and then re-assigned according to the "interface rules", this could play a role too

     

    Edited by bonienl
    Link to comment

    Really need to see the beginning of that call trace to see what's happening (which is difficult to capture), but probably is an issue with a network driver:  I don't see any network driver in the "Modules linked in" list.

    Link to comment

    I made a capture of my system. Issues start at the line "BUG: unable to handle kernel NULL pointer..."

    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fac
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fac
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fad
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fad
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6c
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6d
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf
    Jan 23 18:19:58 vesta kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf
    Jan 23 18:19:58 vesta kernel: EDAC MC0: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#0: DEV 0000:ff:12.0 (INTERRUPT)
    Jan 23 18:19:58 vesta kernel: EDAC sbridge:  Ver: 1.1.2 
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 4 transid 37220822 /dev/sdg1
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 3 transid 37220822 /dev/sdf1
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 1 transid 37220822 /dev/sdh1
    Jan 23 18:19:58 vesta kernel: BTRFS: device fsid 15cea296-4aa8-4a45-b03f-1d4bd2587221 devid 2 transid 37220822 /dev/sdk1
    Jan 23 18:19:58 vesta kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
    Jan 23 18:19:58 vesta kernel: PGD 1ff7e0d067 P4D 1ff7e0d067 PUD 1fbaaff067 PMD 0 
    Jan 23 18:19:58 vesta kernel: Oops: 0000 [#1] SMP NOPTI
    Jan 23 18:19:58 vesta kernel: CPU: 2 PID: 2532 Comm: modprobe Not tainted 4.19.94-Unraid #1
    Jan 23 18:19:58 vesta kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.1a 10/24/2018
    Jan 23 18:19:58 vesta kernel: RIP: 0010:kernfs_name_hash+0x9/0x6d
    Jan 23 18:19:58 vesta kernel: Code: 48 33 04 25 28 00 00 00 74 05 e8 c3 3d ea ff 48 83 c4 60 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 83 c9 ff 31 c0 49 89 f8 <f2> ae 48 f7 d1 8d 79 ff 31 c9 48 39 cf 74 1f 49 0f be 04 08 48 ff
    Jan 23 18:19:58 vesta kernel: RSP: 0018:ffffc90006b27cb8 EFLAGS: 00010246
    Jan 23 18:19:58 vesta kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffffff
    Jan 23 18:19:58 vesta kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
    Jan 23 18:19:58 vesta kernel: RBP: ffff889fcab65198 R08: 0000000000000000 R09: ffffffff811ac100
    Jan 23 18:19:58 vesta kernel: R10: ffffea007f2ad900 R11: ffff889fff075001 R12: ffff889fcab65348
    Jan 23 18:19:58 vesta kernel: R13: 0000000000000000 R14: ffff889ff3a6c190 R15: 0000000000000000
    Jan 23 18:19:58 vesta kernel: FS:  0000150ae6c37b80(0000) GS:ffff889fff680000(0000) knlGS:0000000000000000
    Jan 23 18:19:58 vesta kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Jan 23 18:19:58 vesta kernel: CR2: 0000000000000000 CR3: 0000001ff3b5a002 CR4: 00000000003606e0
    Jan 23 18:19:58 vesta kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Jan 23 18:19:58 vesta kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    Jan 23 18:19:58 vesta kernel: Call Trace:
    Jan 23 18:19:58 vesta kernel: kernfs_find_ns+0x5e/0xa8
    Jan 23 18:19:58 vesta kernel: kernfs_remove_by_name_ns+0x49/0x75
    Jan 23 18:19:58 vesta kernel: remove_files+0x38/0x5a
    Jan 23 18:19:58 vesta kernel: sysfs_remove_group+0x55/0x6f
    Jan 23 18:19:58 vesta kernel: sysfs_remove_groups+0x28/0x2f
    Jan 23 18:19:58 vesta kernel: device_remove_attrs+0x33/0x63
    Jan 23 18:19:58 vesta kernel: device_del+0x18d/0x2ed
    Jan 23 18:19:58 vesta kernel: cdev_device_del+0x10/0x26
    Jan 23 18:19:58 vesta kernel: posix_clock_unregister+0x1c/0x41
    Jan 23 18:19:58 vesta kernel: ptp_clock_unregister+0x69/0x6d
    Jan 23 18:19:58 vesta kernel: igb_ptp_stop+0x1a/0x44 [igb]
    Jan 23 18:19:58 vesta kernel: igb_remove+0x39/0xfd [igb]
    Jan 23 18:19:58 vesta kernel: pci_device_remove+0x36/0x8e
    Jan 23 18:19:58 vesta kernel: device_release_driver_internal+0x144/0x225
    Jan 23 18:19:58 vesta kernel: driver_detach+0x6d/0x77
    Jan 23 18:19:58 vesta kernel: bus_remove_driver+0x60/0x7c
    Jan 23 18:19:58 vesta kernel: pci_unregister_driver+0x1c/0x7f
    Jan 23 18:19:58 vesta kernel: __se_sys_delete_module+0x10f/0x1ac
    Jan 23 18:19:58 vesta kernel: ? exit_to_usermode_loop+0x55/0xa2
    Jan 23 18:19:58 vesta kernel: do_syscall_64+0x57/0xf2
    Jan 23 18:19:58 vesta kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
    Jan 23 18:19:58 vesta kernel: RIP: 0033:0x150ae6d72047
    Jan 23 18:19:58 vesta kernel: Code: 73 01 c3 48 8b 0d 49 7e 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 19 7e 0c 00 f7 d8 64 89 01 48
    Jan 23 18:19:58 vesta kernel: RSP: 002b:00007ffc617bb338 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
    Jan 23 18:19:58 vesta kernel: RAX: ffffffffffffffda RBX: 0000000000427cf0 RCX: 0000150ae6d72047
    Jan 23 18:19:58 vesta kernel: RDX: 0000000000000000 RSI: 0000000000000800 RDI: 0000000000427d58
    Jan 23 18:19:58 vesta kernel: RBP: 0000000000427d58 R08: 1999999999999999 R09: 0000000000000000
    Jan 23 18:19:58 vesta kernel: R10: 0000150ae6de9ac0 R11: 0000000000000206 R12: 0000000000000000
    Jan 23 18:19:58 vesta kernel: R13: 0000000000000000 R14: 0000000000427d58 R15: 0000000000425480
    Jan 23 18:19:58 vesta kernel: Modules linked in: sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore intel_rapl_perf i2c_i801 ipmi_ssif igb(-) i2c_algo_bit i2c_core smartpqi ahci libahci scsi_transport_sas wmi pcc_cpufreq ipmi_si button [last unloaded: atlantic]
    Jan 23 18:19:58 vesta kernel: CR2: 0000000000000000
    Jan 23 18:19:58 vesta kernel: ---[ end trace bfd77ca2011f6527 ]---
    Jan 23 18:19:58 vesta kernel: RIP: 0010:kernfs_name_hash+0x9/0x6d
    Jan 23 18:19:58 vesta kernel: Code: 48 33 04 25 28 00 00 00 74 05 e8 c3 3d ea ff 48 83 c4 60 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 83 c9 ff 31 c0 49 89 f8 <f2> ae 48 f7 d1 8d 79 ff 31 c9 48 39 cf 74 1f 49 0f be 04 08 48 ff
    Jan 23 18:19:58 vesta kernel: RSP: 0018:ffffc90006b27cb8 EFLAGS: 00010246
    Jan 23 18:19:58 vesta kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffffff
    Jan 23 18:19:58 vesta kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
    Jan 23 18:19:58 vesta kernel: RBP: ffff889fcab65198 R08: 0000000000000000 R09: ffffffff811ac100
    Jan 23 18:19:58 vesta kernel: R10: ffffea007f2ad900 R11: ffff889fff075001 R12: ffff889fcab65348
    Jan 23 18:19:58 vesta kernel: R13: 0000000000000000 R14: ffff889ff3a6c190 R15: 0000000000000000
    Jan 23 18:19:58 vesta kernel: FS:  0000150ae6c37b80(0000) GS:ffff889fff680000(0000) knlGS:0000000000000000
    Jan 23 18:19:58 vesta kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Jan 23 18:19:58 vesta kernel: CR2: 0000000000000000 CR3: 0000001ff3b5a002 CR4: 00000000003606e0
    Jan 23 18:19:58 vesta kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Jan 23 18:19:58 vesta kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    Jan 23 18:19:59 vesta rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="2507" x-info="https://www.rsyslog.com"] start 

    After the call trace the system hangs, and I need to do a power off / power on to get a "proper" boot up

    • Like 1
    Link to comment
    6 minutes ago, johnnie.black said:

    Another one using the igb NIC driver, that's my main suspect.

    I made the capture using the mirror function of syslog.

     

    The next thing after the syslog daemon is started is the initialization of the network, starting with a reload of the network drivers. Your suspect seems very plausible.

    Link to comment

    I am having the same or similar issue after upgrading to 6.8.1 from 6.7.2.  I scanned through the syslog after generating the zip and I don't see this error in the log.  I do see the same error as OP on my monitor when I try to boot into GUI mode.  I can get to the console login with a monitor plugged in and with a normal boot but the web UI is not available no matter which option I select.

     

    Same issue when trying safe mode.

     

    I downloaded a fresh copy of Unraid and replaced all of the bz files as mentioned by the troubleshooting section of the wiki but still had the same results.

     

    Edit: I just swapped the ethernet cable from the onboard Intel NIC to the second onboard NIC which is a 2.5G Realtek and now things are fully working again.

     

    X570 ROG Crosshair VIII mobo with most recent BIOS

    tower-diagnostics-20200123-1057.zip

    Edited by blink515
    Edited for clarity and ability to add diagnostics
    Link to comment
    2 hours ago, bonienl said:

    I made a capture of my system. Issues start at the line "BUG: unable to handle kernel NULL pointer..."

     

    The call trace does implicate igb:

    Jan 23 18:19:58 vesta kernel: igb_ptp_stop+0x1a/0x44 [igb]
    Jan 23 18:19:58 vesta kernel: igb_remove+0x39/0xfd [igb]

    Interesting, some of my test servers also use igb but do not crash:

    1. Micro-Star International Co., Ltd X399 GAMING PRO CARBON AC (MS-7B09)
    2. Supermicro X10SRM-F 

     

    I'm about to release 6.8.2 with latest 4.19 kernel, let's see if that solves it; otherwise, I'll look into replacing the in-tree igb driver with Intel's out-of-tree igb driver.

    Link to comment
    3 hours ago, johnnie.black said:

    Another one using the igb NIC driver, that's my main suspect.

    Both of my onboard NICs are Intel.  One uses the igb driver and the other uses the e1000e driver.

     

    eth0 (igb) is the NIC used by unRAID.

     

    I have never seen a server crash with 6.8.1 when booting the server and I have rebooted several times.  The crashes certainly do appear to be related to igb but perhaps only on certain motherboards or other hardware combinations? 

     

    As @limetech pointed out he has crash-free servers using igb.

    Jan 13 13:15:58 MediaNAS kernel: e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
    Jan 13 13:15:58 MediaNAS kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
    Jan 13 13:15:58 MediaNAS kernel: igb: Copyright (c) 2007-2014 Intel Corporation.
    Jan 13 13:15:58 MediaNAS kernel: mpt3sas version 26.100.00.00 loaded
    Jan 13 13:15:58 MediaNAS kernel: mpt3sas 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
    Jan 13 13:15:58 MediaNAS kernel: mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32689028 kB)
    Jan 13 13:15:58 MediaNAS kernel: cryptd: max_cpu_qlen set to 1000
    Jan 13 13:15:58 MediaNAS kernel: pps pps0: new PPS source ptp1
    Jan 13 13:15:58 MediaNAS kernel: igb 0000:03:00.0: added PHC on eth0
    Jan 13 13:15:58 MediaNAS kernel: igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
    Jan 13 13:15:58 MediaNAS kernel: igb 0000:03:00.0: eth0: (PCIe:2.5Gb/s:Width x1) d0:50:99:XX:XX:XX
    Jan 13 13:15:58 MediaNAS kernel: igb 0000:03:00.0: eth0: PBA No: 001300-000
    Jan 13 13:15:58 MediaNAS kernel: igb 0000:03:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
    Jan 13 13:15:58 MediaNAS kernel: e1000e 0000:00:1f.6 eth1: (PCI Express:2.5GT/s:Width x1) d0:50:99:XX:XX:XX
    Jan 13 13:15:58 MediaNAS kernel: e1000e 0000:00:1f.6 eth1: Intel(R) PRO/1000 Network Connection
    Jan 13 13:15:58 MediaNAS kernel: e1000e 0000:00:1f.6 eth1: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF

     

    Link to comment
    1 hour ago, limetech said:

    Interesting, some of my test servers also use igb but do not crash:

    Are these servers with a single ethernet interface?

     

    The driver(s) reload only takes place when more than one interface exists and the file "network-rules.cfg" exists on the flash device.

    Link to comment
    18 hours ago, bonienl said:

    Are these servers with a single ethernet interface?

     

    The driver(s) reload only takes place when more than one interface exists and the file "network-rules.cfg" exists on the flash device.

    My server has two Intel NICs.  The network-rules.cfg file exists on my flash drive.  I just looked in the syslog and did not find "modprobe -r $DRIVERS"

     

    I guess in my case no drivers reload means no crash?   No cable is currently plugged in to eth1 (e1000e driver) although the interface is active.  Is that causing the reload to be skipped?

     

    I am certainly not complaining that I do not have server crashes; just trying to add some data points for those looking into it.

    Edited by Hoopster
    Link to comment
    47 minutes ago, bonienl said:

    Are these servers with a single ethernet interface?

     

    The driver(s) reload only takes place when more than one interface exists and the file "network-rules.cfg" exists on the flash device.

    When you say 'interfaces' you mean simply if besides eth0 you have eth1, 2, .. etc.?  Yes that is true though at present only one server has both connected to a switch,  using bonding driver - simple failover mode.  Also there is no network-rules.cfg file 🤨

    Link to comment

    Running into the same issue here.  Brand new install (6.8.1) on a SuperMicro X10DRL-i  which has 2x Intel i210 Gigabit Adapters and a Realtek RTL8211E (Dedicated for IPMI). Tried also booting Safe Mode without any success.  Worked fine on first boot-up and configuration.  First time I rebooted the system, I got stuck at modprobe -r $DRIVERS. 

     

    Let me know if theres anything I can do to help.

    Link to comment
    11 hours ago, johnnie.black said:

    I didn't test myself, though I have very few plugins installed on that server, but according to at least one user in one of the threads I linked above it also happens in safe mode without any plugins.

     

     

     

    I get the error regardless of whether I have plugins operating at boot. Initially I thought it was plugins as my system would boot fine but if I added plugin it wouldnt. Later I worked out that if I did a fresh install it would boot once, if I rebooted it wouldnt work. It seemed to me that when the boot completed it changed soemthing in the O/S. This change caused the error to manifest on a reboot. I did a fresh install of 6.8.1, booted and then rebooted immediately having not changed anything and the problem arose. At the moment I get the error 100% of the time.

     

    I also have an LSI SATA Controller.

     

    Today I am going to reimage back to 6.8.0 and see if I get the error. If I dont Ill remain on that version.

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.