I won't bite. It took me a while to upgrade from 6.9.2 to 6.11.1 and that didn't go very well, as I had to redo and fight a lot of problems. I guess see you at 6.12.X branch  

Let me know when the 11 branch reaches stable  

9 minutes ago, Michael_P said:

Let me know when the 11 branch reaches stable  

So far, no issues reported with 6.11.5 but there has been a run of releases with issues leading up to it.  Fortunately, they were identified and resolved quickly.

 

11.2 - can't format drives larger than 2TB

11.3 - can't autostart the array with a trial license

11.4 - ipvlan/macvlan display issues when attempting to switch between network types.

 

11.5 - so far all the above appear to be fixed and 11.5 is settling in for a longer shelf life than its predecessors.  When the 11.x branch becomes "stable" for you is, of course, up to you to decide.

 

"Stable" really means "I want everything I commonly do to work and not be annoyed by things I don't do often that don't work."   For me, all of the above releases were "stable" since I was not adding drives to the array, was not running a trial license and had switched to ipvlan many months ago.  I never would have noticed any of those problems but I dutifully upgraded to each new release then watched the bug reports.  😁

updated 2 machines from 6.11.x to this. no issues.

 

Almost at 48 hours on 6.11.5, no issues.

Attempted upgrade from 6.11.4 to 6.11.5 and getting a PANIC and hang... no other changes to hardware or any recent hardware issues that I know about and have been upgrading at every point release... Guess I can try to downrev my USB Boot stick but not sure how to proceed and or troubleshoot since it never finishes a boot.

 

This is occurring directly after "Starting Samba: on 4th line "/usr/sbin/winbindd -D"

VERIFY3(size <= rt-›rt_space) failed (281442933186560 <=

PANIC at range_tree.c:436 :range_tree_remove _impiO)

 

Edited November 23, 20222 yr by SuperW2

2 minutes ago, SuperW2 said:

Guess I can try to downrev my USB Boot stick but not sure how to proceed

You can manually downgrade by downloading the 6.11.3 (6.11.4 is not available) release from the download page, extracting the zip file and copying all the bz* files to the root of your unRAID flash drive.

2 hours ago, Hoopster said:

You can manually downgrade by downloading the 6.11.3 (6.11.4 is not available) release from the download page, extracting the zip file and copying all the bz* files to the root of your unRAID flash drive.

Thanks...didn't help...now get the PANIC with either 6.11.3 or 6.11.5.  I CAN boot into Safe Mode in both so maybe I had some type of hardware failure that just happened at the same time... unfortunantly almost zero info anywhere with the error i'm seeing.

3 hours ago, SuperW2 said:

I CAN boot into Safe Mode

That suggests a plugin issue, uninstall all plugins (or rename the *.plg files) to see if you can find the culprit.

On 11/21/2022 at 11:10 PM, itimpi said:

This is already available under the Update OS tool and has been for a very long time.

 

It might make sense to have for their to be a RC out for a period (not sure if it needs to be 14 days) and that a stable release is then made that has NO changes other than the change to the release number.   The problem would be if enough people would run such a RC to show up the problems that were encountered.

 

Were any of these releases (6.11.1, .2, .3, .4 or .5) put out as RC's before they became stable? - I feel like 14 days is reasonable considering the normal Unraid release schedule prior to the rapid fire we've had recently, would probably have resulted in only a .2 release being a thing with .3 .4 and .5 not having been necessary.

13 hours ago, SuperW2 said:

Attempted upgrade from 6.11.4 to 6.11.5 and getting a PANIC and hang... no other changes to hardware or any recent hardware issues that I know about and have been upgrading at every point release... Guess I can try to downrev my USB Boot stick but not sure how to proceed and or troubleshoot since it never finishes a boot.

 

This is occurring directly after "Starting Samba: on 4th line "/usr/sbin/winbindd -D"

VERIFY3(size <= rt-›rt_space) failed (281442933186560 <=

PANIC at range_tree.c:436 :range_tree_remove _impiO)

 

I think those are related to ZFS.

https://github.com/openzfs/zfs/issues/11691

Updated 6.11.4 to 6.11.5 no longer can boot, any suggestions please?

 

37 minutes ago, bobC said:

any suggestions please?

Where is the boot stopping?

40 minutes ago, bobC said:

Updated 6.11.4 to 6.11.5 no longer can boot, any suggestions please?

 

I am getting a device descriptor read/64 error 110

and then

cannot mount /dev/sda1 - press enter to reboot, 

I can not boot into safe mode either,  same errors

any one have any suggestions i can try ?

image of  booting log attached

 

23 minutes ago, bobC said:

any one have any suggestions i can try ?

Backup current flash drive, recreate using the USB tool or manually, restore the config folder from the backup.

48 minutes ago, bobC said:

I am getting a device descriptor read/64 error 110

and then

cannot mount /dev/sda1 - press enter to reboot, 

I can not boot into safe mode either,  same errors

any one have any suggestions i can try ?

image of  booting log attached

 

 

How did you reboot the server following download/install of new Unraid OS version?  Console messages imply an unclean reboot...

Think you are correct,   the update appeared to be complete, but still showed prior version, so I manually did a reboot from the Gui,  and it never booted after that,

I took out the USB Flash and tried to read it in and tried to read in a Linux Machine and had read errors there also,

I had a previous flash backup and restored on a new flash and pulled down a new key,  everything is now update and appears to be working fine!

Appreciate the response.

 

Bob

I upgraded without thinking much, and this update appears to have killed my setup, something crashes in the kernel when starting the array:
 

[  143.967784] rcu: INFO: rcu_preempt self-detected stall on CPU
[  143.967787] rcu:     2-....: (59996 ticks this GP) idle=caf/1/0x4000000000000000 softirq=7170/7170 fqs=14996 
[  143.967802]  (t=60000 jiffies g=16485 q=749575 ncpus=8)
[  143.967804] NMI backtrace for cpu 2
[  143.967806] CPU: 2 PID: 58 Comm: kworker/u16:2 Not tainted 5.19.17-Unraid #2
[  143.967808] Hardware name: ASUS All Series/Z97-A, BIOS 2801 11/11/2015
[  143.967810] Workqueue: btrfs-cache btrfs_work_helper
[  143.967816] Call Trace:
[  143.967819]  <IRQ>
[  143.967821]  dump_stack_lvl+0x44/0x5c
[  143.967825]  nmi_cpu_backtrace+0xda/0x104
[  143.967838]  ? lapic_can_unplug_cpu+0x95/0x95
[  143.967842]  nmi_trigger_cpumask_backtrace+0x72/0x107
[  143.967854]  trigger_single_cpu_backtrace+0x2a/0x31
[  143.967858]  rcu_dump_cpu_stacks+0xa0/0xd6
[  143.967861]  rcu_sched_clock_irq+0x44b/0xaf7
[  143.967864]  ? perf_event_task_tick+0x5e/0x1e3
[  143.967869]  ? tick_nohz_restart_sched_tick+0x83/0x83
[  143.967871]  update_process_times+0x62/0x81
[  143.967874]  tick_sched_timer+0x43/0x71
[  143.967876]  __hrtimer_run_queues+0x110/0x1b7
[  143.967879]  hrtimer_interrupt+0x9c/0x16e
[  143.967882]  __sysvec_apic_timer_interrupt+0xc5/0x12f
[  143.967885]  sysvec_apic_timer_interrupt+0x80/0xa6
[  143.967888]  </IRQ>
[  143.967889]  <TASK>
[  143.967890]  asm_sysvec_apic_timer_interrupt+0x16/0x20
[  143.967893] RIP: 0010:native_queued_spin_lock_slowpath+0x7f/0x1d0
[  143.967897] Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 0f
[  143.967899] RSP: 0018:ffffc90000257c60 EFLAGS: 00000202
[  143.967901] RAX: 0000000000000101 RBX: ffffc90000257cc8 RCX: 0000000000b83f02
[  143.967902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffc90000257cc8
[  143.967904] RBP: ffff888102fadd90 R08: 0000000000000000 R09: 0000000000000200
[  143.967905] R10: ffff888102fadd90 R11: 0000000000030d80 R12: 0000000000000001
[  143.967906] R13: ffff88816b309018 R14: ffff888166ad4000 R15: 000000003772a000
[  143.967910]  do_raw_spin_lock+0x14/0x1a
[  143.967912]  __btrfs_remove_free_space_cache+0xe/0x2d
[  143.967916]  load_free_space_cache+0x230/0x2dd
[  143.967919]  ? kmem_cache_free+0xdb/0x176
[  143.967925]  caching_thread+0x7e/0x43d
[  143.967929]  ? raw_spin_rq_unlock_irq+0x5/0x10
[  143.967932]  ? finish_task_switch.isra.0+0x140/0x21c
[  143.967936]  btrfs_work_helper+0x114/0x2a5
[  143.967940]  process_one_work+0x1ab/0x295
[  143.967943]  worker_thread+0x18b/0x244
[  143.967945]  ? rescuer_thread+0x281/0x281
[  143.967947]  kthread+0xe7/0xef
[  143.967950]  ? kthread_complete_and_exit+0x1b/0x1b
[  143.967952]  ret_from_fork+0x22/0x30
[  143.967957]  </TASK>

 

Unfortunately this seems severe enough that I cannot extract diagnostics from the system (hangs during collection), pressing Reboot in the UI does not reboot, issuing "shutdown -r now" in the console does not reboot, etc. My only option is to "yank the plug" so to speak and try again. I will need to try safe mode.

 

I managed to downgrade to 6.11.2 (which was the previous version on the USB stick) through the UI but the problem remains the same, it hangs forever while starting the array. I am not ruling out a hardware issue, because the downgrade should have fixed it?

 

I will try to downgrade further, maybe go back to 6.10.x.

For what it's worth, there doesn't seem to be an easy way to downgrade to 6.10.3 from 6.11.2. First the download link always points to the latest version, but even editing the .plg to point to the old 6.10.3 version & Linux kernel, failed a manual plugin install ("not installing old version"). In the end I had to reinstall 6.11.5 again.

 

In any case, booting the system in safe mode allows the array to start just fine. I am now going through a forced Parity check (which should take a couple of days), likely caused by the multiple hard shutdowns I was forced to do to regain control of the system

On 11/22/2022 at 6:21 AM, augot said:

Upgraded from 6.11.4 to 6.11.5 without any issues except that it still says "Starting services..." on the bottom bar on the GUI dashboard. Can't see anything in the log that hasn't started yet (server's been up for a few hours now) so not sure if it's a graphical bug or not:

 

 

 

I'm having the same "problem". Everything appears to be working as expected, however.

6 hours ago, Fulgurator said:

For what it's worth, there doesn't seem to be an easy way to downgrade to 6.10.3 from 6.11.2.

Pull the flashdrive, backup, download the release you want from : https://unraid.net/download

Then replace the bz*files and changes.txt on the drive by those from the zip.

You are good to go.

10 hours ago, Fulgurator said:

For what it's worth, there doesn't seem to be an easy way to downgrade to 6.10.3 from 6.11.2.

You can always use the process documented here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page to go to any Unraid release for which you can obtain the zip file for the release.

1 hour ago, itimpi said:

You can always use the process documented here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page to go to any Unraid release for which you can obtain the zip file for the release.

 

Looks like I just forgot to scroll and only saw the big orange button on the top right. Anyway downgrading isn't really the issue here, but I'll remember next time. Thanks!

 

Parity check is already halfway done, so tomorrow I should be able to reboot again normally and see if I "hang" starting the array again. The system "works" (UI is not frozen, shell works, etc) but of course anything requiring the array is down, and trying to reboot the system normally (or generating Diagnostics logs) stalls forever.

Edited November 24, 20222 yr by Fulgurator

Interestingly, searching for "btrfs-cache btrfs_work_helper" yields the same callstack in the Linux Kernel mailing list:

 

https://lore.kernel.org/linux-btrfs/e1cde76ba6cf7b14d6f38310113588d6486d5a00.1659989333.git.josef@toxicpanda.com/

 

Not sure if related but it sure seems suspicious.

27 minutes ago, Fulgurator said:

Not sure if related but it sure seems suspicious.

Assuming you are using btrfs only for the pool(s) you can try clearing the space cache, it should be pretty safe to run but always good to make sure backups are up to date:

 

With the array stopped:

btrfs check --clear-space-cache v2 /dev/sdX1

If this is an old pool that might have v1 space cache also run:

btrfs check --clear-space-cache v1 /dev/sdX1

It won't hurt if there's no v1 cache