WebUI faulty and array undefined while parity check

smeeee.again · March 27, 2019

Hey.

i have a problem with Unraid 6.6.7 (rolled back from 6.7.0-rc4)

Whenever the SWystem performes a parity check at 60-80% the array chrashes.

WebUI is available but faulty and the shares are shown via smb but can't be accesed.

After reboot everything is fine.

the problembs began after changing partiy disk to brand new 8TB device. the parity was build withoute error and the old drive became a datadisk within the array.

The Diagnostics.zip after crash is nearly empty :-/

Hardware:

CPU: Intel(R) Core(TM) i5-7600

MB: B250M-D2V Gigabyte

RAM: Ballistix Sport LT 12GB

Syba SI-PEX40064 4 Port SATA III PCI-EXPRESS 2.0 x 1 Controller

1x parity, 1 chache ssd, 7 Drives

kartoffelhq-diagnostics-after crash.zip

kartoffelhq-diagnostics-running normal.zip

trurl · March 27, 2019

23 minutes ago, smeeee.again said:

...

After reboot everything is fine.

the problembs began after changing partiy disk to brand new 8TB device. the parity was build withoute error and the old drive became a datadisk within the array.

...

Just to be safe, can you confirm that you rebooted after the new parity build and everything was fine? The reason I ask is because we need to be sure your new disk assignments were saved so it won't think your old parity (now data) disk is still parity.

Can't tell much from those diagnostics, but the symptoms, including the empty diagnostics, suggest flash drive has been dropped.

Put flash in your PC and let it checkdisk. While there make a backup.

Make sure you use a USB2 port on your server for the flash.

For future reference, you can always download a zipped backup of flash at Main - Boot Device - Flash - Flash Backup.

smeeee.again · March 28, 2019

Thanks for your help!

19 hours ago, trurl said:

Just to be safe, can you confirm that you rebooted after the new parity build and everything was fine? The reason I ask is because we need to be sure your new disk assignments were saved so it won't think your old parity (now data) disk is still parity.

I'm not shure, can i check this at some point?

Workflow was: Addet new drive as second parity Drive -> let it run throug -> Parity check (everything working, 0 errors) -> made the new Drive the only parity drive (at this point i dont remember if i made a parity check)

Or is it poissible to "delete" the parity Drive to build it new?

19 hours ago, trurl said:

Put flash in your PC and let it checkdisk. While there make a backup.

Make sure you use a USB2 port on your server for the flash.

I checked the flashdrive with checkdisk without errors, the flashdrive is connected on the intern USB port via Adapter (https://smile.amazon.de/gp/product/B00DKWY0S8/ref=oh_aui_search_asin_title?ie=UTF8&psc=1) and works since i build the computer

trurl · March 28, 2019

18 minutes ago, smeeee.again said:

Workflow was: Addet new drive as second parity Drive -> let it run throug -> Parity check (everything working, 0 errors) -> made the new Drive the only parity drive (at this point i dont remember if i made a parity check)

So are you saying you now have a disk assigned to the parity2 slot but nothing assigned to the parity slot?

smeeee.again · March 28, 2019

6 minutes ago, trurl said:

So are you saying you now have a disk assigned to the parity2 slot but nothing assigned to the parity slot?

No, the drive was assigned to the first slot, sry for not mention this.

trurl · March 28, 2019

Still unclear. You said you added new drive as second parity drive. Did you not mean you put that new drive in the parity2 slot?

smeeee.again · March 28, 2019

1 minute ago, trurl said:

Still unclear. You said you added new drive as second parity drive. Did you not mean you put that new drive in the parity2 slot?

Yes, with old parity drive still in the first slot.

trurl · March 28, 2019

You seem to have left out a lot of important details and it makes me think you have done all of this wrong.

You had a disk in the parity slot. You put a new disk in the parity2 slot and built parity2.

.....

Somehow you got from there to having the new parity2 disk in the parity slot and the old parity disk as a new data disk.

Is that correct?

Parity and parity2 are not the same and can't be exchanged.

smeeee.again · March 28, 2019

1 minute ago, trurl said:

Is that correct?

Yeah, this is correct... seems so I fucked this up pretty hard.

is it possible to build the parity drive from scratch without data loss?

trurl · March 28, 2019

You shouldn't lose any data since parity doesn't have any data and all of the data disks are independent.

I guess it really doesn't matter at this point exactly how you got to where you are. Rebuilding parity is what you need to do and maybe even that is what it was doing when you said it was doing a parity check since if you put a new disk in the parity slot it is going to want to rebuild parity.

Did you have any data on that new data (old parity) disk yet?

smeeee.again · March 28, 2019

5 minutes ago, trurl said:

Did you have any data on that new data (old parity) disk yet?

yes, it is unfortunately already in use.

trurl · March 28, 2019

That's OK.

The reason I was concerned about whether your new disk assignments had been saved successfully was because if it thought that disk was still parity it would overwrite its data with parity.

If you still have that flash drive out, put it in your PC and edit config/disk.cfg to set autostart to no so you can boot up and check your disk assignments.

smeeee.again · March 28, 2019

5 minutes ago, trurl said:

If you still have that flash drive out, put it in your PC and edit config/disk.cfg to set autostart to no so you can boot up and check your disk assignments.

I can't find the point autostart in the file. Do you mean startArray?

I changed it from "yes" to "no" and started the server.

The hard disks are all where they belong.

Is a rebuild still triggered that way?

trurl · March 28, 2019

Post new diagnostics

smeeee.again · March 28, 2019

8 minutes ago, trurl said:

Post new diagnostics

new diagnostics.zip

trurl · March 28, 2019

Looks OK.

Since you haven't started the array yet, you can just skip the first step

~~Stop array~~

Unassign disk

Start array with disk unassigned

Stop array

Reassign disk

Start array to begin rebuild

(In this case, "disk" refers to your parity disk, but those instructions are the same for rebuilding a data disk to the same disk since it is all the same parity calculation.)

smeeee.again · March 28, 2019

Thanks for the help! I do a parity check after the rebuild and then report again.

trurl · March 28, 2019

If it seems to be going wrong try to get another diagnostic.

If you look at Main - Array Devices, it should have a lot of writes to parity, a lot of reads of all other disks, and zero errors on all disks.

smeeee.again · March 29, 2019

The rebuild just finished without errors. Now it looks like this in webui.

Docker Services and the VM are running.

trurl · March 29, 2019

Post diagnostics. If the webUI won't let you get them you can try to get the from the command line as explained here:

https://forums.unraid.net/topic/37579-need-help-read-me-first/

smeeee.again · March 29, 2019

I can't get a diagnostics file via the webUI.
Via the console I get the error message from the screenshot.

Now I tried to capture the syslog, but to get it I have to shut down the server and read the USB stick on my computer.

At the bottom you can see how the messages "Array undefined" and "Array Started" alternate in different intervals.

Squid · March 29, 2019

What is the output of

df -h

smeeee.again · March 29, 2019

3 minutes ago, Squid said:

What is the output of

df -h

root@KartoffelHQ:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          5.8G  5.8G     0 100% /
tmpfs            32M  236K   32M   1% /run
devtmpfs        5.8G     0  5.8G   0% /dev
tmpfs           5.9G     0  5.9G   0% /dev/shm
cgroup_root     8.0M     0  8.0M   0% /sys/fs/cgroup
tmpfs           128M  884K  128M   1% /var/log
/dev/sda1        30G  402M   29G   2% /boot
/dev/loop0      8.2M  8.2M     0 100% /lib/modules
/dev/loop1      4.9M  4.9M     0 100% /lib/firmware
/dev/md1        3.7T  3.1T  563G  85% /mnt/disk1
/dev/md2        3.7T  3.1T  604G  84% /mnt/disk2
/dev/md3        4.6T  4.0T  620G  87% /mnt/disk3
/dev/md4        4.6T  4.1T  552G  89% /mnt/disk4
/dev/md5        4.6T  2.3T  2.3T  50% /mnt/disk5
/dev/md6        3.7T  711G  3.0T  20% /mnt/disk6
/dev/md7        5.5T   11G  5.5T   1% /mnt/disk7
/dev/sdf1       239G  187G   52G  79% /mnt/cache
shfs             31T   18T   13T  57% /mnt/user0
shfs             31T   18T   14T  57% /mnt/user
/dev/loop2       30G   19G   11G  64% /var/lib/docker
/dev/loop3      1.0G   17M  905M   2% /etc/libvirt
shm              64M     0   64M   0% /var/lib/docker/containers/89466d7ab026469d123e4ba4a13c0ee3163565b62ee4192cc0a300e297511fd2/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/9e9c6a6a0422683abc65ab76b338e2bf5cb40e157081cdfd717b6c5d7a8cc62c/mounts/shm
shm              64M  8.0K   64M   1% /var/lib/docker/containers/3c081b4b83ec61bb8d5bb09b6f841c7148f633c8c4e2cea0011ed04424dffae7/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/05f143591df703c25bbfb1cc4161157e32415ee6e8e658b8b17000613c308ba6/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/aa2c6ac117296e52339feed1afaf51b01598c216489bfb13e7d452432c69cb32/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/7f409b380de76f8a41ca237cc8d3ebd13df2955cfd17b341485a437dd6ec5aa5/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/e83688f60259f94fb25cd06a4f97841f71fcb6f0d81bce36ccc7f94bce107969/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/01b2dd903585a67a0e33ca24857538b3651e107fde03e6727961c13e9760e7b4/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/267de35ba0fc79b147a96475186cd10e3136036fb16b991cb4253988e549a723/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/e6ad761dbd49b663c38a3c1e822e8da16f16e399c9c133375029bbb65e671844/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/8bbf7613bd879d9c29b0d53f2245e29e2ed858aaa9c72d993f3286b85c177aae/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/b74a0e0c2b90fef37bf683085cb1e117c8173a3daf49db809e99a8572673fab9/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/03ed9eea62d0dc11dc1a74ffc10ecfab7ae285b8d14cec8000e49d664797ffe4/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/e67eab68be52d12d55f9ba2c95483589a90e7c447f4736ec0c077ac726e8533c/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/3413a746470808c88aa678f661bec03c67d5bef4018dd476beb3030cd5b32994/mounts/shm

Squid · March 29, 2019

1 minute ago, smeeee.again said:

rootfs 5.8G 5.8G 0 100% /

This is going to play havoc with your system. Most likely a misconfigured docker container is writing to RAM instead of a share (ie: writing to /mnt/appdata instead of /mnt/user/appdata)

Your only recourse though right now is to reboot

powerdown -r

and then investigate

smeeee.again · March 29, 2019

The Docker Containers are configured correctly. Most of them have been running without errors for some time and the newest ones are deactivated to exclude them as a source of errors.

The problem only seems to occur during a parity check or, as it turns out, during the rebuild.

To see if this is really the case, I will run the server for a few days without a parity check, unless there are other ideas.

WebUI faulty and array undefined while parity check

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation