How to unzip and zip bzroot

[pants]

2 hours ago, ich777 said:

Why would you do that?

In my case, I did it to fix a bug introduced by limetech into the "upgradepkg" script that exists in Unraid versions 6.10-6.11 (fixed in 6.12). Specifically the version_compare() function (included below) does not properly populate the ARCH, and VER variables and ends up only comparing packages by build number which prevents newer packages from being installed if they have a build number that is lower than an older package with the same name:

# limetech -
# Here's a function that figures out if one package is a newer version than
# another package.  We use this to prevent "upgrading" an existing package
# to an older one.
# We're lazy so use PHP's version_compare() function to do the heavy lifting.

version_compare() {
  STRING="$1"
  BUILD=${STRING##*-}
  STRING=${STRING%*-}  // <-- Should be STRING=${STRING%-*}
  ARCH=${STRING##*-}
  STRING=${STRING%*-}  // <-- Should be STRING=${STRING%-*}
  VER=${STRING%*-}
  OLDVERSION="$VER.$BUILD"

  STRING="$2"
  BUILD=${STRING##*-}
  STRING=${STRING%*-}  // <-- Should be STRING=${STRING%-*}
  ARCH=${STRING##*-}
  STRING=${STRING%*-}  // <-- Should be STRING=${STRING%-*}
  VER=${STRING%*-}
  NEWVERSION="$VER.$BUILD"

  php -r "echo version_compare('$OLDVERSION', '$NEWVERSION');"
}

Normally, to fix a bug like this I would build a custom package with the fixed file and put it in /boot/extra to be applied during startup but that approach itself relies on "upgradepkg" to install the fix which presented a bit of a chicken-and-egg paradox. To solve the problem, I ended up grabbing the "upgradepkg" script from Unraid 6.12.5 and copying it into the initramfs in my bzroot.

 

In general, I agree that modifying the initramfs in bzroot should be avoided if possible and only done as a last resort. However, I do think a working procedure for modifying bzroot should be easily available in the rare case it is necessary as it would save hours of time.

[ich777]

2 minutes ago, pants said:

Normally, to fix a bug like this I would build a custom package with the fixed file and...

To fix the bug it would be best to make limetech aware of that but may I ask why you are on such an old version anyways?

 

4 minutes ago, pants said:

In general, I agree that modifying the initramfs in bzroot should be avoided if possible and only done as a last resort. However, I do think a working procedure for modifying bzroot should be easily available in the rare case it is necessary as it would save hours of time.

I disagree heavily on that, back in the days where my Unraid-Kernel-Helper was around which modified the bz* files to bundle specific drivers it was really hard some times to troubleshoot because users had customized bz* files, which at the time was not obvious and not part of the Diagnostics.

Keep in mind if someone modifies their bz* files and maybe change something by accident this could cause other issues too...

 

9 minutes ago, pants said:

upgradepkg

Wouldn't that be solved by --reinstall or even using installpkg instead of upgradepkg, there would be other ways around this to BTW...

Another overlay fs or you even could use the driver plugin installation logic for that...

 

However I would strongly discourage users to modify their bz* files.

[pants]

1 hour ago, ich777 said:

To fix the bug it would be best to make limetech aware of that but may I ask why you are on such an old version anyways?

I didn't file a report since the bug is fixed in 6.12. Unfortunately, I have had limited success getting limetech to fix bugs I've filed in the past (see this report and this comment) and have often had to resort to my own patches / customizations.

 

As for why I do not run the current version of Unraid, in the interest of stability I prefer to stay at least one major release behind current and only upgrade to the final minor release after the subsequent major release comes out. Upgrading can be time consuming depending on how smoothly it goes so I prefer to limit upgrades to at most once a year if possible.

 

1 hour ago, ich777 said:

I disagree heavily on that, back in the days where my Unraid-Kernel-Helper was around which modified the bz* files to bundle specific drivers it was really hard some times to troubleshoot because users had customized bz* files, which at the time was not obvious and not part of the Diagnostics.

I certainly understand your point of view from a supportability perspective and I'm not suggesting that modifying the initramfs in bzroot be encouraged in general. Nowadays it probably only makes any sense to do in very unusual circumstances. I wasn't around in the old days but I'm guessing Unraid-Kernel-Helper made modifying the bz* files a little too easy for the average user to the point where people were doing it in situations where it wasn't strictly necessary; if that's the case discontinuing it was likely a smart move. 

 

Having said that, I still think a working set of commands to make changes to bzroot should be documented (and ideally kept up to date). Limiting the procedure to a list of commands rather than building a plugin or app around it should discourage less tech-savvy users from making changes while still helping more experienced folks avoid unnecessary pitfalls. While it's certainly possible that an inexperienced user might introduce problems by improperly modifying bzroot, the nearly working set of commands already present in this thread is actively causing frustration and wasting people's time.

 

1 hour ago, ich777 said:

Wouldn't that be solved by --reinstall or even using installpkg instead of upgradepkg, there would be other ways around this to BTW...

Another overlay fs or you even could use the driver plugin installation logic for that...

 

The issue is that I assumed upgradepkg itself is being called during startup to install the packages in /boot/extra. If that's the case, to fix the bug the script really needs to be replaced before that point in startup (otherwise some packages in /boot/extra may not be properly installed). I don't know much about the init sequence and I was worried that trying to replace the script after boot would leave a window for the buggy script to be invoked so I thought the safest thing to do was to replace it entirely. Given that the bug is fixed in 6.12, it's also a fairly clean solution for me since my changes will be automatically wiped away during upgrade without requiring me to do any manual cleanup.

Edited December 22, 2023 by pants

[ich777]

7 hours ago, pants said:

As for why I do not run the current version of Unraid, in the interest of stability I prefer to stay at least one major release behind current and only upgrade to the final minor release after the subsequent major release comes out. Upgrading can be time consuming depending on how smoothly it goes so I prefer to limit upgrades to at most once a year if possible.

I have to say, I'm always on the latest stable/beta/RC and everything is working as it should. In my opinion this update strategy is really bad but that's just my opinion and the worst, you won't get support by most community developers or even the support desk itself if something is not working.

 

7 hours ago, pants said:

Having said that, I still think a working set of commands to make changes to bzroot should be documented (and ideally kept up to date).

It is more complicated than you think, a lot changed with 6.12.0+ and not everything which was in bzroot is there where it was below version 6.12.0

It is okay to post the command here on the forums I think but you also have to know the risk that somebody that is not that experienced will discover these commands and starts editing files and repacking the bzroot since strictly speaking there are only seven commands needed, just saying...

cd /your/work/dir/bzroot
dd if=/boot/bzroot bs=512 count=$(cpio -ivt -H newc < /boot/bzroot 2>&1 > /dev/null | awk '{print $1}') of=/your/work/dir/microcode
dd if=/boot/bzroot bs=512 skip=$(cpio -ivt -H newc < /boot/bzroot 2>&1 > /dev/null | awk '{print $1}') | xzcat | cpio -i -d -H newc --no-absolute-filenames

cd /your/work/dir/bzroot
find . | cpio -o -H newc | xz --check=crc32 --ia64 --lzma2=preset=9e --threads=$(nproc --all) >> /your/work/dir/bzroot.part
cat /your/work/dir/microcode /your/work/dir/bzroot.part > /your/work/dir/bzroot
sha256sum /your/work/dir/bzroot | awk '{print $1}' > /your/work/dir/bzroot.sha256

 

7 hours ago, pants said:

I have had limited success getting limetech to fix bugs I've filed in the past

Maybe you should issue a PR here or even create an Issue.

 

7 hours ago, pants said:

Given that the bug is fixed in 6.12, it's also a fairly clean solution for me since my changes will be automatically wiped away during upgrade without requiring me to do any manual cleanup.

But think again about that, if you would have gone straight to 6.12.6 then you wouldn't have these issues at all.

 

BTW there would be also a way easier fix, just rename the package in /boot/extra if it won't install properly.

[pants]

On 12/23/2023 at 1:36 AM, ich777 said:

I have to say, I'm always on the latest stable/beta/RC and everything is working as it should. In my opinion this update strategy is really bad but that's just my opinion and the worst, you won't get support by most community developers or even the support desk itself if something is not working.

I think both upgrade strategies are valid and the right one to employ depends on how a particular system is used. For people who regularly work on their server and want to take advantage of the newest features, moving to the latest release makes sense. For others who invested a lot of time initially to get things working and then moved on to other projects but still rely on their server to always be up and running, upgrading infrequently may meet their needs better. Limetech seems to tacitly support both upgrade strategies given that the USB Creator app allows users to select 6.11.5, 6.12.4, 6.12.5, 6.12.6 when deploying Unraid on a new server implying that there is a "limetech endorsed" reason for at least some users to start on 6.11.5 rather than 6.12.X.

 

On 12/23/2023 at 1:36 AM, ich777 said:

It is more complicated than you think, a lot changed with 6.12.0+ and not everything which was in bzroot is there where it was below version 6.12.0

 

I'm aware of the changes to the bz* files in Unriad 6.12. Again for future readers, the only change to bzroot in 6.12 is that the /usr directory tree was moved from bzroot to bzfirmware. The full set of changes to all the bz* files is well-documented by limetech here.

 

On 12/23/2023 at 1:36 AM, ich777 said:

strictly speaking there are only seven commands needed, just saying...

cd /your/work/dir/bzroot
dd if=/boot/bzroot bs=512 count=$(cpio -ivt -H newc < /boot/bzroot 2>&1 > /dev/null | awk '{print $1}') of=/your/work/dir/microcode
dd if=/boot/bzroot bs=512 skip=$(cpio -ivt -H newc < /boot/bzroot 2>&1 > /dev/null | awk '{print $1}') | xzcat | cpio -i -d -H newc --no-absolute-filenames

cd /your/work/dir/bzroot
find . | cpio -o -H newc | xz --check=crc32 --ia64 --lzma2=preset=9e --threads=$(nproc --all) >> /your/work/dir/bzroot.part
cat /your/work/dir/microcode /your/work/dir/bzroot.part > /your/work/dir/bzroot
sha256sum /your/work/dir/bzroot | awk '{print $1}' > /your/work/dir/bzroot.sha256

 

Nice! This is much more concise and functional than the set of commands I posted earlier. Just to reiterate for posterity though, using the --no-absolute-filenames option when expanding the cpio archive might improperly alter symlinks with relative paths on 6.10+ (although if I remember correctly those symlinks are in /usr/local/ so it may be safe to use the option again on 6.12+). Also, recompressing with --ia64 is probably not ideal for users who aren't running on itanium cpus. For those who want to incorporate these changes, line 3 and 5 of the commands above would become:

dd if=/boot/bzroot bs=512 skip=$(cpio -ivt -H newc < /boot/bzroot 2>&1 > /dev/null | awk '{print $1}') | xzcat | cpio -i -d -H newc

find . | cpio -o -H newc | xz --check=crc32 --x86 --lzma2=preset=9e --threads=$(nproc --all) >> /your/work/dir/bzroot.part

 

On 12/23/2023 at 1:36 AM, ich777 said:

Maybe you should issue a PR here or even create an Issue.

 

When I found this bug, I actually did search for the upgradepkg script in that repository but could not find it within the sbin directory. Perhaps slackware files modified by limetech (as opposed to source code originally authored by limetech) are not committed to that repo since they may need to be manually recreated each release if the upstream file changes in slackware?

 

On 12/23/2023 at 1:36 AM, ich777 said:

But think again about that, if you would have gone straight to 6.12.6 then you wouldn't have these issues at all.

 

While it's true that I would not have experienced this particular issue, there are many reasons why upgrading immediately is not always possible. As one example, if people rely on plugins that are not compatible with the latest release they must either wait for the plugins to be updated or take the time to find and deploy an alternative if the plugins have been abandoned by their author.

 

On 12/23/2023 at 1:36 AM, ich777 said:

BTW there would be also a way easier fix, just rename the package in /boot/extra if it won't install properly.

I did consider manually changing the build numbers of the package names in /boot/extra but ultimately decided against it because that approach would require that I remember to continually modify the build numbers of any packages I download in the future. It also has the undesirable side-effect of making the names of local packages differ from their official release names which may cause confusion down the line. Patching the upgradepkg script itself in bzroot is undeniably the "correct" solution since that is how limetech themselves eventually fixed it -- any other approach would just be a workaround.

Edited December 24, 2023 by pants

[JonathanM]

23 hours ago, pants said:

As one example, if people rely on plugins that are not compatible with the latest release they must either wait for the plugins to be updated or take the time to find and deploy an alternative if the plugins have been abandoned by their author.

Definitely valid, but the reverse can also be true. One of the largest by usage plugins in the Unraid ecosystem is updated only on 6.12+.