Upgrade

• As usual followed the flash drive backup via connect. 
• Ran the update assistant which came back clear
• Performed the update then proceeded to attempt the Plex DB corruption issue
• Plex DB corruption repair attempt
• Post Plex docker restart to make sure it was fine
• Server restart and the array will no longer start.

 

System log shows
Nov 23 18:52:38 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Nov 23 18:55:22 Tower nginx: 2023/11/23 18:55:22 [error] 8765#8765: *6037 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.0.108, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "192.168.0.225", referrer: "http://192.168.0.225/Main"

I'm unable to get a diagnostics log as I just hangs and the dashboard shows the CPU's slowing one core at a time maxing out.

If I restart the server ( Auto Start Disabled ) the server boots, until an attempt to start the array, 

I've attempted to remove a disabled drive which appeared to be fine but still the array refuses to start.

 

Any ideas would be much appreciated.

 

tower-syslog-20231123-1904.zip DIagnostics.zip

Boot in safe mode and get the array issues sorted out.

Can you also attach the complete diagnostics.zip file (not the commands that were run) here.

Looks like a corrupt zfs pool, you can try to import it read-only

 

zpool import -o readonly=on cache

 

If it works backup and recreate

@ljm42

I'll take a look later on and get back to you. 

I've a few months ago added the pair of ssd as cache zfs pool ( I think) oddly only one appears to have had any data stored on it and up to now performance on plex has considerably quicker. 

 

Could I for instance turn docker service off, switch location of the content back to mechanical disk with mover and recreate the cache pool? Make sure it boots and runs correctly then switch back again? 

 

I'll try and get diagnostic the last time I tried with the array attempting to start just hung. 

Is it worth diagnostics without the array operational? 

Did you try to mount the pool read-only? If it mounts that would make it easier to restore the data.

On 11/24/2023 at 1:40 AM, dlandon said:

Boot in safe mode and get the array issues sorted out.

Just started in safemode and cannot see a way to gather diagnostics in "Tools" am I missing something?

I followed this thread re SafeMode and selected GUI option.

 

Edited November 27, 20232 yr by frustratedtech71

On 11/24/2023 at 9:00 AM, JorgeB said:

Looks like a corrupt zfs pool, you can try to import it read-only

 

zpool import -o readonly=on cache

 

If it works backup and recreate

I assume the array needs to be running for me to do this, which I'm unable to do right now.

I've switched to SafeMode with GUI but don't have the option to download diagnostics but have managed to get syslogs

tower-syslog-20231127-1922.zip

I've also attempted to remove the "Disk 6" which was showing as unsupported FS.
Which I did with the array stopped changing the disk allocation to "no disk" and attempted to start the array again but no change there either.

On 11/24/2023 at 9:00 AM, JorgeB said:

Looks like a corrupt zfs pool, you can try to import it read-only

 

zpool import -o readonly=on cache

 

If it works backup and recreate

I've run the command but no idea how to check if thats mounted correctly, so clearly no idea no the backup/recreate neither.

With the array stopped I can't see anything as mounted as you'd expect I suppose. 

Sorry for all the dumb question, I haven't needed to really touch unraid apart from a brief issue when swapping underlying hardware into new chassis, new cpu, GPU and new SAS card at which point the two SSD's were added purely for cache and moved appdata and related elements to the SSD for a performance increase.

Diagnostics from the CLI produced are attached now.

I'm learning but embarrassingly incapable it seems

tower-diagnostics-20231127-1948.zip

So a little further but not sure if its a good thing.
In "Safe Mode" I've been able to start the array, however the "Cache Pool" which is a pair of SSD are showing as "Unmountable disks present" which were a zfs pool hosting my appdata, domains, isos and system.
So now worried I've lost all my dockers etc that had been working flawlessly.

Any ideas would be very gratefully received. Thanks

 

There is this section of the manual:

 

     https://docs.unraid.net/unraid-os/manual/troubleshooting/

 

It would probably help if you were to read through it.  However, if you don't understand it, STOP and wait until @JorgeB has a chance to get back to you.  (He is based in Europe and the time difference means he is not available in the late afternoon and evening here in the USA.  Panic driven actions by users has caused more data loss than bad hardware!) 

 

By the way, you can access the manual by clicking on the     DOCS      link in the header of any Forum page or from the   Manual     link on the Taskbar of the Unraid user GUI

Thanks @Frank1940 I'll take a look but also hang fire for @JorgeB I'm GMT based so JorgeB and I will not be too far apart time wise.

The array is currently doing a parity check as it stands (in GUI safemode) having removed the reported failed drive ( which still remains in the chassis just incase ) which as I currently understand it shouldn't be an issue as the data was being emulated from that drive anyway.

 

The last diags do confirm the pool mounted read-only, so the order of things would be this, reboot the server to start over, don't start the array, mount the pool manually and read-only with the command above, now start the array, the pool will still show unmountable in the GUI, but the data will be accessible under /mnt/cache, you can then copy any important data to the array or elsewhere.

Thanks @JorgeB that's certainly reassuring. 
Can you see any configuration errors in how I'd configured the cache pool in a ZFS mirror/

Just wondering how best to recreate it once the data is relocated? Could I reverse the decision to switch to cache over array, have mover relocate and then delete, recreate the content etc, or would you suggest a totally separate location maybe a unassigned USB drive for instance?

12 hours ago, frustratedtech71 said:

Can you see any configuration errors in how I'd configured the cache pool in a ZFS mirror/

Not really, but the pool is corrupt, possibly due to some hardware issue.

 

12 hours ago, frustratedtech71 said:

Just wondering how best to recreate it once the data is relocated? Could I reverse the decision to switch to cache over array, have mover relocate and then delete, recreate the content etc, or would you suggest a totally separate location maybe a unassigned USB drive for instance?

If you copy the data to the array you can then move back to a pool using the mover.

Thanks again @JorgeB

I'll check the smart reports of the SSD, which to be fair were old surplus but matching so I guess some new WD Red small capacity ones would be a good replacement choice?

They should be but the ones you have now look healthy, unlikely the problem was caused by them.

Good to hear. I'll feedback once I've found the time to suss it all out. Thanks again @JorgeB

Finally grabbed some time to look at this.
Oddly the plan was to copy the data from cache to disk3 which despite the "main" view indicating plenty of space free I almost immediately received 
the below error...repeatedly
 

cp: error writing '/cache/cache/appdata/plex/Cache/PhotoTranscoder/55/55524ed89b9c4fbb35e82c1482d70681a003cfbc.jpg': No space left on device

So created a new directory on Disk4 and attempted again.
This time around a slightly different issue, which does look slightly odd, but might also explain the Plex database corruption and backup failure messages.
 

cp: cannot stat '/mnt/cache/appdata/plex/Cache/PhotoTranscoder/ce/cea4dd8d5e6bf803a56eecfc9b23cf23'$'\230\021\372''Á'$'\210\377\377''.jpg': No such file or directory

During which this was occurring I also lost access to the WebGUI, which does appear to be an intermittent fault

Logs show
 

Dec  2 11:35:42 Tower rc.diskinfo[20655]: PHP Warning: syntax error, unexpected end of file, expecting TC_DOLLAR_CURLY or TC_QUOTED_STRING or '"' in /var/local/emhttp/disks.ini on line 250
Dec  2 11:35:42 Tower rc.diskinfo[20655]:  in /usr/local/emhttp/plugins/unassigned.devices.preclear/scripts/rc.diskinfo on line 294
Dec  2 11:35:42 Tower rc.diskinfo[20655]: PHP Exception: array_map(): Argument #2 ($array) must be of type array, bool given in /usr/local/emhttp/plugins/unassigned.devices.preclear/scripts/rc.diskinfo on line 295
Dec  2 11:38:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Dec  2 11:39:01 Tower crond[1518]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Dec  2 11:39:30 Tower root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token
Dec  2 11:39:47 Tower monitor: Stop running nchan processes

Which is making life awkward in regards to removing the existing pool and recreating it and of course moving the data back with mover.

Any ideas @JorgeB
 

Edited December 2, 20232 yr by frustratedtech71

Try using midnight commander (mc on the console), easier to se any errors and skip some files if needed.

Thanks yet again @JorgeB every day is becoming a school day for me, with a great tutor. Really appreciate your help so far.

Once the original cache data has been copied via mc to disk4\cache\... would I be correct in thinking that the steps are

1. Perform compare within mc to ensure as much is indentical

2. Deselect each of the current SSD from the cache pool

3. Restart the server

4. Configure a new cache pool having cleared the original contents of the existing SSD

5. Start array

6. Configure cache for /appdata /system /iso /domains to prefer cache and manually trigger mover

7. Once confident all data is on cache. (need to find these settings again) setup exclusive access to cache and unconfigure the mover setup?

or am I totally over thinking/complicating this?

 

1 hour ago, frustratedtech71 said:

2. Deselect each of the current SSD from the cache pool

To clear the pool you can just stop the array, click on the first device and then click "erase", after array start you have the option to re-forma the pool.

 

Remaining steps look OK to me.

@JorgeB I do think the array is now sorted and working.
Only thing I've just noticed is that the cache pool has now formatted as btrfs which is slightly annoying.

Now just need to resolve the plex database corruption issue. 

Would appreciate it if you could have a look at the diagnostics and let me know if you can see anything I need to be aware of.. still learning.

Thanks again

tower-diagnostics-20231203-1449.zip