Hey all,

 

I recently bought the Unraid OS Plus license after trying it out and overall I've really enjoyed the OS, but I'm having some serious problems with it recently.

 

The issues started when I had a disk removed from my pool because of some write errors. I swapped out the disk and started up the array, and since then I've been having serious issues with almost everything. Last night almost nothing was accessible, the docker tab didn't load, and I couldn't even access the Apps tab, and the parity sync is expected to take over 100 days to complete (speed varies, but is currently 732.7 KB/sec, estimated 232 days, 6 hours, 33 minutes).

 

I checked the docker daemon and found that my apps were still running, and I was wondering if apps writing was the cause of my parity sync slowness, so I tried force stopping docker, but the command just hung.

/etc/rc.d/rc.docker stop

 

So next I kill -9 the docker daemon, which has resulted in finally killing the docker daemon, but now I can't start it again. When I check status, it says it's running, and then hangs.

root@Tower:/mnt/user/appdata/binhex-plex/Plex Media Server/Logs#  /etc/rc.d/rc.docker status
status of dockerd: running

 

At this point I'm hoping to figure out how to (1) fix the slowness of the parity-sync, and (2) fix Docker and be able to control my containers.

 

I've attached diagnostics.

 

Looking forward to hearing back,

 

-Jon

 

 

tower-diagnostics-20211121-0938.zip

8 minutes ago, BostonJon said:

swapped out the disk

How did you decide it was bad?

 

Bad connections are much more common than bad disks, and are a likely cause of your continuing problems. 

 

Do you still have the original disk?

 

On mobile now so can't look at Diagnostics yet. 

Emulated disk2 unmountable, connection problems on multiple disks. Stop, shutdown, check all connnections, SATA and power, both ends, including splitters. If you still have the original disk you might plug it in also so I can check it out.

 

After that reboot and post new diagnostics.

Ah thanks for the response. I double checked the connections already, so I might just need to replace the sata cables. I'll shut it all down, replace the cables, put the old drive back, and see how that works out.

 

(Re initial response, the only indication of any errors was the unraid write error that removed it from the array, but SMART didn't find anything wrong, and the drive is very new an lightly used.)

02:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)

Marvell controllers are NOT recommended, and since you don't have many disks you should be able to eliminate it.

 

Nov 20 19:39:29 Tower kernel: scsi host9: pata_atiixp
Nov 20 19:39:29 Tower kernel: scsi host10: pata_atiixp

Also go into BIOS and make sure all your disks are using AHCI

Thanks for the additional info. I'll take those steps as well!

1 minute ago, BostonJon said:

Ah thanks for the response. I double checked the connections already, so I might just need to replace the sata cables. I'll shut it all down, replace the cables, put the old drive back, and see how that works out.

 

(Re initial response, the only indication of any errors was the unraid write error that removed it from the array, but SMART didn't find anything wrong, and the drive is very new an lightly used.)

Don't put the old drive back in the array, just plug it in so we can see its SMART in the next diagnostics you post. Currently the emulated disk isn't mounting, so rebuild would result in an unmountable disk. Maybe after you get your connections fixed it will be emulated better and be mountable. If not we can try to repair the emulated filesystem before rebuilding, or maybe see if the original disk is mountable.

1 minute ago, trurl said:

Don't put the old drive back in the array, just plug it in so we can see its SMART in the next diagnostics you post. Currently the emulated disk isn't mounting, so rebuild would result in an unmountable disk. Maybe after you get your connections fixed it will be emulated better and be mountable. If not we can try to repair the emulated filesystem before rebuilding, or maybe see if the original disk is mountable.

 

Okay, got it.

So my mobo only has 6 sata ports. To add the old one back, and to stop using the pcie card, I'd need to remove either one of the cache or parity drives. Is removing a cache drive okay at this point?

If it matters, the ssd cache drives are on the pcie controller, and all hdds are all connected to the mobo.

Also, if the objective of adding the old drive back in is to get the SMART report, here's on I pulled down last night before doing the swap.

tower-smart-20211120-1708.zip

8 minutes ago, BostonJon said:

get the SMART report

That looks OK, no need to plug it in then, just keep it handy in case we need it. We will work with the replacement disk for now.

I did everything you suggested, and everything is looking much better! Parity sync is processing at a much more reasonable speed of 132.4 MB/sec, and will be done in less than a day. Docker is responding again, and I can start/stop containers.

 

I've attached another diagnostic in case you see anything else I need to do.

 

Thanks for all the help, this was really starting to frustrate me!

tower-diagnostics-20211121-1335.zip

3 hours ago, trurl said:

Maybe after you get your connections fixed it will be emulated better and be mountable. If not we can try to repair the emulated filesystem before rebuilding

Emulated (and rebuilding) disk2 still unmountable. Did it have any files on it?

 

 

Oh interesting. The new drive does not have any file on it, and I had formatted it last night, but I see that I needed to format it again. It's now mounted, and parity is running.

 

The old drive *did* have files on it, and I'm noticing that they are all now missing from /mnt/user. Is it possible to reconnect the old drive to restore them? I also have those files backed up on another drive, so I could restore them once parity is complete if necessary,

 

tower-diagnostics-20211121-1549.zip

This is where users often make mistakes that lead to data loss. Shouldn't be a problem since you still have the original disk, but I see by your response that you have some misconceptions that might lead you to make those mistakes.

 

24 minutes ago, BostonJon said:

I had formatted it last night

Do you mean you formatted it before putting it in the array? Or did you format it after putting it in the array?

 

If you did format it before putting it in the array, what filesystem did you format it as? Just curious, because it doesn't matter. No point in formatting a drive that is going to be used for rebuild, since every bit of the disk is going to be completely overwritten by the rebuild operation.

 

If instead you formatted it after putting it in the array, then you have already made the mistake I am talking about.

 

Many people have a vague idea of what "format" means. They seem to think it means something like "get a disk ready to use", whatever that might mean. But computers don't work on vague ideas.

 

"Format" means "write an empty filesystem to this disk". That is what it has always meant in every operating system you have ever used. When you format a disk in the parity array, Unraid treats that write operation exactly as it does any other, by updating parity. So, after formatting a disk in the parity array, the only thing rebuild can result in is an empty filesystem.

 

Format is NEVER part of rebuild. If you have an unmountable disk that should have files on it, you must try to repair the filesystem. When you have a disabled and emulated disk, and the emulated disk is unmountable, we usually recommend repairing the emulated filesystem before attempting the rebuild, since the rebuild is going to result in exactly what the emulation has. This is especially important when rebuilding to the same disk, because you want to make sure you don't overwrite the original disk with an unmountable filesystem.

 

4 hours ago, trurl said:

we can try to repair the emulated filesystem before rebuilding

 

Since you are already rebuilding, too late to do the repair before rebuild. But since it is to a new disk, and you still have the original disk with all its contents intact, you should be OK.

 

Let the rebuild complete then post new diagnostics and we will see about repairing the filesystem on the rebuilt disk.

 

Thanks for the response.

 

It appears that the timing of the format is what caused my issues. I understand that formatting the disk would clear the filesystem. I expected UNRAid to format the disk, then add it to the array, and then start the repair. What it sounds like happens is that the format happens while the array is online, so that gets written to the parity disk as well.

 

For clarity, can you confirm that if I had formatted the disk with the array offline that I would've been okay at that point? Ultimately, what I want to know are the steps I should take when a drive actually fails and I need to replace it.

1 minute ago, BostonJon said:

if I had formatted the disk with the array offline

How exactly did you do that?

2 minutes ago, BostonJon said:

the timing of the format is what caused my issues

It remains to be seen if format has caused any issues, but I don't see how it could have caused the issues of having a disabled disk or an unmountable disk.

 

3 minutes ago, BostonJon said:

I expected UNRAid to format the disk, then add it to the array, and then start the repair

Just as pointless for Unraid to format a disk that is going to be rebuilt as it would be for you to do so. Rebuild overwrites every bit of the disk. The filesystem is part of the bits on the disk, so no point in creating an empty filesystem only to overwrite it with a filesystem that has contents.

40 minutes ago, trurl said:

Rebuild overwrites every bit of the disk.

 

I'm definitely missing something. So you're saying that the rebuild will build everything, including the filesystem, so the format is not necessary. Then why was UNRaid saying that the disk is unmountable, and offers formatting as a solution?

 

I'm still curious about what the right option was here, since when a disk does fail and I need to replace it I don't want to make the same mistake again. What should I do when replacing a disk, and it says it's unmountable and suggest formatting, but I want to ultimately restore the files from parity? Is there a guide I should be referencing?

 

2 minutes ago, BostonJon said:

UNRaid saying that the disk is unmountable

Filesystem corruption can result in the disk being unmountable.

 

1 hour ago, trurl said:

If you have an unmountable disk that should have files on it, you must try to repair the filesystem

The disk was already disabled, and the emulated disk was unmountable, when you posted your diagnostics at the beginning of this thread. Remains to be seen whether the original disk is also unmountable.

 

 

6 minutes ago, BostonJon said:

I don't want to make the same mistake again

Nothing you did caused this. Unless you want to call your hardware problems a mistake. Maybe you could have asked before doing anything and saved some time in getting those hardware problems corrected.

I think this is what I was really looking for here.

 

https://wiki.unraid.net/Replacing_a_Data_Drive

 

I definitely made more than just a hardware mistake, as I expected to be able to just allow the data to come back from parity. One of the main selling points for me was the ability to restore data if a disk failed, and this was kind of a run through of that for me. In the end, I didn't follow the steps right to trigger a rebuild.

What exactly did you do differently? It is rebuilding.

I didn't unassign the old drive before powering down.

 

Maybe I'm still confused here, but after I initially uploaded my diagnostics, you mentioned that the disk2 was still not mounted.

 

Quote

Emulated (and rebuilding) disk2 still unmountable. Did it have any files on it?

 

At this point, I did what I thought was the right move by formatting the disk to mount it. The disk was mounted, but then the files were gone.

If I had said "no" at this point, how would you have suggested that I solve the unmountable disk issue?