25 days for parity build?

[salvdordalisdad]

Hi Guys,

 

I just recovered my 3rd unraid server ("Plus" license) & replaced a couple of disks intending it to become a backup target.

So I zeroed all the disks & created a new array of 7 x 4Tbyte disks, with one of them as parity.

All the disks are completely blank, freshly formatted in xfs.

These were happy in the old servers, so I assumed they'd be OK for this job.

 

However, it seems not.

The parity build (1st time as it's a new array)  says it will take 25 days to complete.

When it first started, it said 10 hours, and has crept up since then.

 

So clearly something's wrong, probably a dying disk, but I don't know how to diagnose it, so please can someone read the diags & point me in the direction of the faulty disk?n3-diagnostics-20220519-1514.zip

 

Thanks in advance. Much appreciated.

Lee

Edited May 19, 2022 by salvdordalisdad

[JorgeB]

There's nothing logged, I would suggest canceling the sync and running a test with the diskspeed docker to confirm all disks are performing normally, also make sure this high CPU utilization is expected:

 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
27587 root      20   0   16452  15688   2888 R  96.2   0.1 240:12.17 port_ping+
30062 root      20   0    6004   4060   2712 R  92.3   0.0   0:01.02 lsof

 

[salvdordalisdad]

So I did some digging into those diags & saw a LOT of dead tcp connections, and spotted that port_ping+

 

I cancelled all of the (legacy from previous config) user scripts & rebooted.

Now those connections are gone, which is as expected.

The parity build time has gone down to 9 days - which is an improvement - woohoo!

I think I've spotted another slight issue...

 

Can't run docker diskspeed

Docker ... "path does not exist"

There was no /mnt/system/docker  folder, so I created one - now it doesn't complain about that anymore.

There's no /mnt/user/appdata   folder - but I can't create one of those...  "no medium found"

 

 

Contents of /mnt/user:

root@n3:/mnt# cd /mnt/user
root@n3:/mnt/user# ls -l
total 0
drwxrwxrwx 1 nobody users 6 May 19 11:07 filestore/

 

I'd expected a bunch of folders, system, appdata, domains...

Do I just create those, or is something lower level going to be required to fix that? - open to suggestions.

 

Tried creating "system" as a share - didn't do anything, no folder in /mnt/user at all.

So I created the folders    /mnt/system/docker   and   /mnt/user/filestore/appdata

But even with those fudged the docker refused to start.

 

Definitely something lower level needed.
I will tiidy up that mess I just made...and await the right instructions.

(Even if those instructions involve starting from scratch)_.

 

Thanks in advance

Lee

Edited May 19, 2022 by salvdordalisdad

[JorgeB]

23 minutes ago, salvdordalisdad said:

There's no /mnt/user/appdata   folder - but I can't create one of those...  "no medium found"

Doe it work if you type for example

mkdir /mnt/disk1/appdata

 

[salvdordalisdad]

Hmm, yes it dort of works, but it still says "path does not exist".

 

I think I'm going to scratch this server & start again.

Now I need to find out how to do that on the existing USB drive...

 

some searching to do I think.

But not right now, out of time, have to be after the weekend now.

;-/

 

Thanks for the nudges, much appreciated...

[trurl]

45 minutes ago, salvdordalisdad said:

/mnt/system/docker

This folder would be in RAM since it is not a disk path or a user share path.

 

docker.cfg in those earlier diagnostics has a path in /mnt/cache for your docker.img, since you don't have cache that path is also in RAM.

 

[trurl]

And appdata, domains, system shares are configured cache-only

[JorgeB]

8 minutes ago, trurl said:

And appdata, domains, system shares are configured cache-only

Yeah, that makes sense, since there's no cache it generates an error.

[salvdordalisdad]

That's really interesting, but if I have no cache?

Anyway, I started with a fresh 6.10 trial on a new USB, as I had clearly got something wrong & I am comfortable with starting from scratch.

So I ran the diskspeed benchmarks & was a little surprised at the max & min values.

disk    MB/sec    MB/sec
sdf    200    97
sdg    157    70
sdh    200    87
sdb    150    73
sdc    200    94
sdd    183    84
sde    142    67

 

So I chose the fastest (sdc) as the parity.

Are these figures good enough? Do they indicate any underlying issues?

Anyway, just rebuilt the array again & started it & it's now saying 16 days .

The CPU is trundling along at 3% util, so it's not that.

All the SATA ports are 6Gbps, all the disks are 6Gbps.

 

Any suggestions, or just "yeah it might do that"

 

Thanks in advance for any pointers.

 

Also - when I want to move back to the old USB, what do I copy from this USB drive to that one & what do I delete on that one? 

(I know that the license in in the plus.key  file, but virtually naff all else of what's on there).

[trurl]

1 hour ago, salvdordalisdad said:

That's really interesting, but if I have no cache?

That was entirely our point. You have no cache, but many things were configured to write to that cache you don't have.

 

1 hour ago, salvdordalisdad said:

now saying 16 days

Attach new diagnostics to your NEXT post in this thread.

[JorgeB]

16 hours ago, JorgeB said:

and running a test with the diskspeed docker to confirm all disks are performing normally

Besides new diags see if you can run this now.

[salvdordalisdad]

Hi JorgeB

 

Thanks, I did a quick & dirty single disk array & got docker up & running on fresh demo license & new USB stick (seemed quickest!)

These are the stats from the disks:

 

 

disk    MB/sec    MB/sec
sdf    200    97
sdg    157    70
sdh    200    87
sdb    150    73
sdc    200    94
sdd    183    84
sde    142    67

 

They all seem pretty much within a similar range, no single one stands out as clearly having difficulties.

Any thoughts?

Thanks in advance.

(Nice picture - Slava Ukraine! )

[JorgeB]

Those numbers look fine, please post new diags during parity sync.

[salvdordalisdad]

Hi All,

 

Rather than fret about it, I figured I'd just ask it every few days & see.

I asked it today & found this message:

 

Last check completed on Tue 24 May 2022 08:04:42 AM BST (two days ago)
 Duration: 4 days, 7 hours, 14 minutes, 8 seconds. Average speed: 10.8 MB/s
 Finding 0 errors

 

And it's all running fine, no errors. Pesky calculator got it massively wrong!

Oh well, it was a very ueful learning experience - stop fretting & let it do its thing!

Thanks to all who chipped in.

 

[salvdordalisdad]

...and a post scriptum for the record...

After the parity build had completed, the system was left idle for a few more days, adjusting this & adding that etc.

The parity drive started giving READ errors, only a few hundred, but even a handful is too many for a fresh build.

So I swapped it out - reluctantly expecting another 5 - 25 day parity build - and was shocked to see the estimate for the new parity build should take around 8 hours...

It actually took 10 hours, but heavens to betsy what a difference!

 

Just a shame that the original cause (the Toshiba 7200 rpm) parity drive problem could not be spotted earlier...I had my suspicions but no evidence.

So what have we learned here today?

If the parity build says it's going to take 25 days - one of the disks is probably faulty. Now go & figure out which one!

[trurl]

1 hour ago, salvdordalisdad said:

So what have we learned here today?

If the parity build says it's going to take 25 days - one of the disks is probably faulty. Now go & figure out which one!

After you have already ruled out other problems.

 

Bad connections are much more common than bad disks.

 

Diagnostics can often tell what the problem is, but don't just assume that slow parity checks are bad disks.