Messed up bigtime

[YoloSwagLord]

Dear community,

 

all of a sudden some dockers on my unRaid wont start, I did some research and there was a error message saying that disk 1 was read only or full. So I checked and disk 1 still had some space on it. So I started googling and found this topic, suggesting that the share is broken. So I checked that and all my shares seem to be gone. I tried to add the shares but they didn't show up nor gave it an error message.

 

So I thought it was a smart idea to update the OS, I did this using the tool tab in Unraid. after the update I rebooted it and got the screen below:

 

I tried to make a password a few times but it just shows a blank page after I clicked set password. a simple refresh got me back to the page as shown in the screenshot above.

 

 

I decided to hard reboot the system and watch the monitor hooked up to the unraid server. Then I got very very worried when I saw this screen:

 

 

 

I added the last diagnostics file to the attachments.

 

 

any help would be so helpful I have no idea how to solve this.

 

projectplex-diagnostics-20231204-1146.zip

Edited December 4, 2023 by YoloSwagLord

[JorgeB]

Try booting in safe mode.

[YoloSwagLord]

43 minutes ago, JorgeB said:

Try booting in safe mode.

Hi in safe mode I can boot and login, I cant seem to add shares. or start dockers.
any suggestions how to move onward from here?

***EDIT*** if I boot back in to "normal" mode the same error occurs.

Edited December 4, 2023 by YoloSwagLord

[itimpi]

In safe mode no plugins are loaded which suggest one of your plugins is causing an issue.    It is probably worth booting in safe mode and creating new diagnostics at that point to see if we can spot something.

[YoloSwagLord]

44 minutes ago, itimpi said:

In safe mode no plugins are loaded which suggest one of your plugins is causing an issue.    It is probably worth booting in safe mode and creating new diagnostics at that point to see if we can spot something.

I added the new diagnostics to the attachments, thanks for your help!

 

projectplex-diagnostics-20231204-1637.zip

[JorgeB]

I don't see any attempt to start the array, do you get an error?

[YoloSwagLord]

10 minutes ago, JorgeB said:

I don't see any attempt to start the array, do you get an error?

my bad this in the attachments the diagnostics with the array started.

projectplex-diagnostics-20231204-1659.zip

[JorgeB]

ATA errors and filesystem corruption on disk1, check/replace cables then check filesystem, run it without -n

[YoloSwagLord]

15 hours ago, JorgeB said:

ATA errors and filesystem corruption on disk1, check/replace cables then check filesystem, run it without -n

I checked the cables, but now disk 1 shows up disabled, and I have no idea how to enable the drive.

 

[JorgeB]

Is the emulated disk fixed? Post new diags.

[YoloSwagLord]

Oke it seems like it did correctly in the check filesystem.

 

I added the 4 documents in the attachments:

 

1. filesys_check-n.txt
2. filesys_check.txt
3. file sys_check-L.txt
4. projectplex-diagnostics-20231205-1152

 

filesys_check-L.txt filesys_check.txt filesys_check-n.txt projectplex-diagnostics-20231205-1152.zip

[JorgeB]

You're using a Marvell controller with SATA port multipliers, either one alone is not recommended, but both together can be especially bad, not surprisingly the disable disk is connected there, but there are al lot of ATA errors for the other devices connected to that controller, I would strongly recommend that you replace it with one of these.

 

Post new diags after array start to see if the emulated disk1 is mounting.

[YoloSwagLord]

4 minutes ago, JorgeB said:

You're using a Marvell controller with SATA port multipliers, either one alone is not recommended, but both together can be especially bad, not surprisingly the disable disk is connected there, but there are al lot of ATA errors for the other devices connected to that controller, I would strongly recommend that you replace it with one of these.

 

Post new diags after array start to see if the emulated disk1 is mounting.

Thanks for your quick reply, I will buy a new one for sure! 

and I uploaded the new diagnostics in the attachments. The disk still seems to be disabled when the array is started. 

projectplex-diagnostics-20231205-1212.zip

[itimpi]

13 minutes ago, YoloSwagLord said:

The disk still seems to be disabled when the array is started.

 

The disk will stay disabled until you successfully rebuild it.   The fact that the emulated disk1 appears to now be mounting fine is a good sign, but you need to sort out the disk connection issues before attempting a rebuild or you are likely to get a write error during the rebuild and the disk promptly disabled again as a result.

 

[YoloSwagLord]

1 hour ago, itimpi said:

 

The disk will stay disabled until you successfully rebuild it.   The fact that the emulated disk1 appears to now be mounting fine is a good sign, but you need to sort out the disk connection issues before attempting a rebuild or you are likely to get a write error during the rebuild and the disk promptly disabled again as a result.

 

Thanks for your reply, I rebooted in to "normal" mode and the issues as described in the topic still occur (GLIBC_2.34 not found & new password).
when the disc is emulated this shouldn't be the case right? I should be able to boot in to "normal" unraid and my dockers should be running right?

 

btw. im gonna order a new raid controller for sure, but need to be able to get the server back up online while the new controller is in transit.

[YoloSwagLord]

I just ordered this controller: https://www.amazon.nl/dp/B08L7W8QFT?psc=1&smid=A21CTMDLFXECIV&ref_=chk_typ_imgToDp

and I put the SATA cable directly in to the motherboard for the rebuild. lets hope this will work.

[YoloSwagLord]

I just rebuild disk1 with the check file system with the arg -v. But the disk stays offline/disabled even when I stopped the array en started it again.

 

[itimpi]

2 hours ago, YoloSwagLord said:

Thanks for your reply, I rebooted in to "normal" mode and the issues as described in the topic still occur (GLIBC_2.34 not found & new password).
when the disc is emulated this shouldn't be the case right? I

That message is coming from a plugin since you say this only occurs if you do not boot in Safe Mode.  You need to find the culprit and remove it to get rid of the message.

 

2 minutes ago, YoloSwagLord said:

I just rebuild disk1 with the check file system with the arg -v. But the disk stays offline/disabled even when I stopped the array en started it again.

 

Did you check file system or rebuild?   They are completely different operations.    Only a rebuild cleared the disabled state.

 

[YoloSwagLord]

Quote

That message is coming from a plugin since you say this only occurs if you do not boot in Safe Mode.  You need to find the culprit and remove it to get rid of the message.

is there a landingpage/tutorial on how to find the plugin that is causing issues? or a easy way to find out?

 

Quote

Did you check file system or rebuild?   They are completely different operations.    Only a rebuild cleared the disabled state.

I used the xfs_repair status: of the check file system tab. I posted the logg in the attachment. Is there a guide on how to rebuild the disk?

filesys_check-v.txt

[JorgeB]

I would recommend rebuilding only when the new controller arrives, and only rebuild on top of the old disk if the emulated disk is mounting and contents look correct.

 

https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself

[itimpi]

1 hour ago, YoloSwagLord said:

is there a landingpage/tutorial on how to find the plugin that is causing issues? or a easy way to find out?

 

It is basically a trial-and error approach.

 

You start by renaming .plg files in the config folder on the flash drive to have a different extension and then rebooting as that stops a particular plugin loading.   Renaming back to the .plg and rebooting reactivates that plugin.   Whether you try in batches or individually is up to you and how patient your are.

[YoloSwagLord]

1 hour ago, JorgeB said:

I would recommend rebuilding only when the new controller arrives, and only rebuild on top of the old disk if the emulated disk is mounting and contents look correct.

 

https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself

Thanks for your reply, I chose to do the rebuild since the SATA is now connected to the motherboard instead of the controller. all the checks looks to be correct as in the attachments of previous reply's, but another issue occured.

 

the speed of the data rebuild is very slow (500kb), with that speed its gonna take 200 days to complete the rebuild. I did the rebuild in maintenance mode and clicked the button sync as described in the topic. 

 

[JorgeB]

Likely still having issues with the other disks connected to that controller, check the syslog for ATA errors

[YoloSwagLord]

On 12/5/2023 at 6:56 PM, JorgeB said:

Likely still having issues with the other disks connected to that controller, check the syslog for ATA errors

Hi I received my new controller (https://www.amazon.nl/dp/B08L7W8QFT?psc=1&smid=A21CTMDLFXECIV&ref_=chk_typ_imgToDp) installed it rebooted in to safe mode. started the array, started the sync and the speeds are still unbearable slow, like 150kb/s. I posted my diagnostics in the attachments.

 

projectplex-diagnostics-20231207-2018.zip

[JorgeB]

That controller also has a SATA port multiplier, see here for a recommend controller: