Jump to content
Can0nfan

is there a way to restart the webgui?

51 posts in this topic Last Reply

Recommended Posts

Posted (edited)

so....booted into safe mode and noticed docker and vm services start automatically

shouldn't safemode mean that none of these services would automatically start so as to enable troubleshooting of issues?

 

i've disabled those services and started the array, so docker and vms aren't running

also, i am mirroring syslog to flash so see if something gets caught that perhaps the syslog server itself is missing

Edited by 905jay

Share this post


Link to post
Posted (edited)

server booted in safe mode, with docker and VM services disabled, it's still standing strong.

 

Some observations:

The parity rebuild has been increasing in time, from 20 days, to 190 days, to now over 900 days. Would it make sense for me to format and redo parity fro scratch? The parity disk was brand new for this server build 2 weeks ago.

 

It appears that without docker containers and VM services running (and in safe mode) the server is stable for now. 

Would you agree that this isn't a hardware issue per-se, but rather misconfigured software (somehow /somewhere)?

 

Diagnostics file is attached

 

 


 

Screenshot_20190826-080714.png

i5unraid-diagnostics-20190826-1229.zip

Edited by 905jay

Share this post


Link to post
8 minutes ago, 905jay said:

Can someone offer some advice on the parity rebuild encountering so many errors and taking over 400 days to rebuild?

My guess would be connection or controller issues, but don't have any evidence presented. Post diagnostics.

Share this post


Link to post

Probably the sync errors are caused by the lockups, whatever problem you have that is causing all the dumps in your syslog. And also that is probably the reason your parity check is taking so long that it is pointless. My guess is a hardware issue.

 

Is your BIOS up-to-date? Are you using BIOS defaults (no overclocking, for example)? Is your cooling OK?

 

Possibly your Marvell SATA controller is involved. I know a lot of people have issues with those. Do you have any disks using those ports? 

Share this post


Link to post

Bios is fully up to date, yes
Defaults in BIOS are being used, no overclocking

Cooling is a Noctua CPU Cooler, brand new

I have 2 disks hooked up to the Marvell. That can't be changed until I get my LSI Logic card delivered so I have to wait on that.

 

I have decided that it may be best to remove the parity disk from the array, preclear it, and add it back in. See if that at least helps that situation

Share this post


Link to post

Those call traces during a parity check can usually be resolved by lowering md_sync_thresh (settings -> disk settings), try a value of 100, better to reboot before starting the next check.

Share this post


Link to post

@trurl when I get my LSI card (hopefully next week) and I plug it in and boot the server, do I have to completely reconfigure all my disks?

is this a major undertaking on my part?

 

Or will unRAID just not give a shit, as long as it can see all the disks?

Share this post


Link to post
4 minutes ago, johnnie.black said:

Those call traces during a parity check can usually be resolved by lowering md_sync_thresh (settings -> disk settings), try a value of 100, better to reboot before starting the next check.

thanks for that info, i have made the change

Share this post


Link to post
48 minutes ago, 905jay said:

when I get my LSI card (hopefully next week) and I plug it in and boot the server, do I have to completely reconfigure all my disks?

Unraid keeps track of disk assignments using each disk's serial number so it should be fine.

Share this post


Link to post

@trurl would you be able to help me figure out if it is a container (which one, or which combination of containers) or if it is a VM that is causing all these issues that I'm seeing?

 

I've replaced the memory on the server, (2x8GB + 2x4GB), and formatted the parity disk and am rebuilding the parity now as we speak.

I feel that perhaps I've misconfigured something but i'm unsure of what it may be, and don't want to have to constantly have to go through this parity rebuild due to sudden freezing and power off situations.

 

Is it possible that somewhere I have over-provisioned memory or CPU resources to a docker (or multiple dockers) ?

Is there anything else that I can provide the community that may be able to point to where the shortcomings are with this system?

 

Image 330.png

Image 331.png

Image 332.png

Share this post


Link to post

Nothing really stands out in those screenshots. Just try to complete a parity check without any dockers, VMs, or other addons, and making the setting change johnnie.black recommended. No point in digging into all that other stuff if it won't even do that much.

Share this post


Link to post

Or just let it finish the parity sync it is already doing. Maybe that setting will have fixed it.

Share this post


Link to post

Hey @trurl and @johnnie.black the parity was running for about 24 hours. I started it Monday morning and it went into Tuesday Morning.

I decided to stop it yesterday morning and restart it because it was stuck at about 90% and showing 900+ days to complete.

 

I figured something was wrong, and restarted it yesterday morning.

At 11:30pm last night it shows 93%, 1 hour to complete (approx.)

This morning it shows 93% and a day left to complete (the transfer rate went down from 100MB/s to about 5MB/s)

 

Can anyone help me isolate what the issue is here?

It's been stable otherwise, and hasn't crashed on me at all but that problem has lead to this problem.

 

Diagnostics and syslog attached

i5unraid-diagnostics-20190828-1309.zip unRAID(1)

Share this post


Link to post

Do you have something like       mover       scheduled to run overnight?

Share this post


Link to post

mover runs hourly, however there is nothing running at this time in terms of services that I am aware of
docker service is disabled, as is VM service

 

Share this post


Link to post

There are a couple of ATA errors on the parity disk, not enough to justify that speed, so there might be a disk issue, swap both cables with a different disk and try again, if same it's likely the disk.

Share this post


Link to post
49 minutes ago, 905jay said:

mover runs hourly

Why?

Share this post


Link to post
Posted (edited)

I have no explanation for that other than it is how it was shown to me, and recommended that I setup this way.

But it seems like there are 10 people, who all know what they are talking about, giving me 10 variations on doing everything under the sun.

Typical internet stuff...everyone is an expert on everything behind the cloak of an avatar on a forum.

 

So far a 6TB parity drive sync has taken approx 2 days to complete and still isn't close to being finished.

At 11:30pm last night it shows 93%, 1 hour to complete (approx.)

This morning it shows 93% and a day left to complete (the transfer rate went down from 100MB/s to about 5MB/s)

from 8am this morning to now, it has moved 3% (give or take)

 

Image 344.png

Edited by 905jay

Share this post


Link to post
Posted (edited)

I am looking at the screen capture that you provided in the post above.  What is surprising to me if that all of the data disks are still spun up when the parity check is 96.8% complete.  Looking at the writes, the only disk that has any substantial number of writes is disk5.  disk4 appears to have substantially more reads than the equivalent size disk3.  I would conclude that something is carrying on some type of disk operations on both disk4 and disk5 and probably while the parity check is running.  Plus, by this point the only thing the parity check is doing is verifying that the parity disk can be read...

 

Disk activity during the parity check will cause the check to run more slowly but this seems to be a lot more than would be expected.  There are a few other threads floating around (including one bug report) about file speed issues.  @905jay, do check and see if you can figure out what is causing those extra reads and writes.  I would suspect it is something that is happening in the background as you are only complaining about the speed of the parity check at this point.  That is the 'why' for the questions about the mover.  The mover was added to Unraid so that it could move the files off of the cache drive in the middle of the night so that it would not affected normal disk access speeds.  Most folks have it setup to run once a day or once a week-- not once an hour.  

Edited by Frank1940

Share this post


Link to post

@Frank1940 thanks for the follow up on this.

Yes initially the server was rebooting every night for an unknown reason. That seems to have been fixed by not having docker and vm services running.

The parity is taking forever to run, that is the current concern.

due to the random freezing from before (unresolved) the parity was totally messed up.

 

I see what you mean in terms of the read and writes to the disk in the screenshot, but i'm unable to isolate what is causing that, to those disks only.

I thought the point of unRAID was to spread it out across all disks (read /write /data), not necessarily evenly, but better than this.

 

Some people point to the fact that 6 disks are on the Intel controller, and 2 are on the Marvell controller on the motherboard.

I am holding in my hand an LSI card that arrived today. Once the parity rebuild is complete, I intend to install this card and use it exclusively.

 

Do you think this parity will ever rebuild? Or am I best to stop this now, install the card, and rebuild the parity again?

Share this post


Link to post
36 minutes ago, 905jay said:

Do you think this parity will ever rebuild? Or am I best to stop this now, install the card, and rebuild the parity again?

It looks like it is still moving.  Of course, the other thing is that you now have valid parity for the data disks.  (Everything beyond the 4Tb mark is just for the 'record'.)   (If something should happen to one of the other disks, you can rebuild that disk from the parity information that is on the parity disk at this point.)  If you are the patient or paranoid type (Take your choice  😈 ), you will let it finish, or you could just shut it down and get on to installing that new LSI card.  

Share this post


Link to post
1 hour ago, 905jay said:

I thought the point of unRAID was to spread it out across all disks (read /write /data), not necessarily evenly, but better than this.

It tries to but you have the final say in how things are actually done.  If you go to Shares  and click on one of the User Shares, you can select the "share SEttings" tab.  Now turn on the Help function (Icon on the toolbar) and look for the 'Allocation method:'.  The Help will give you a description of the four choices.  I believe that 'High Water' is the default.   (It tends to spread the files across all on the data disks which means file access speeds are faster --because the outer tracks of the disks are all used before you start using the slower inner tracks--  without spinning up any more disks than necessary.)  But if you feel that another method would be better for your situation, you can chose it from the other three choices.  The only one that is missing is a random distribution of files among the data disks but who would really want that???

Share this post


Link to post

Left the parity to runovernight, and it is at the exact same place it was at yesterday late afternoon

98.7% 3 Hours 50 minutes remaining for at least the last 12 hours.

 

I've given up on the process now, i've aborted and will install the LSI card and see if that helps any any

i5unraid-diagnostics-20190829-1236.zip unRAID

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.