(solved) unRAID 6.9.2 - /var/log is getting full (currently 100 % used)

[openam]

I tried to pull diagnostics, but it just seems to hang. I was able to copy syslog to a separate share, and zip it up. I do have nerd pack installed, but it is not installing 'atop'.

 

I'm gonna give it a reboot like suggested in fix common problems. 

202311-05-syslog.zip

Edited November 14, 2023 by openam

[openam]

Well it said it detected a dirty shutdown. I just clicked the reboot button in the UI. Now it says it's doing a parity check and it's going to take 30+ days. Guess I'll see if it's running any faster in the morning. Also attaching some diagnostics from after the reboot.

 

palazzo-diagnostics-20231105-2132.zip

[JorgeB]

Try booting in safe mode and/or closing any browser windows open to the GUI, only open when you need to use it then close again.

[openam]

I cancelled the original sync. It had shown 0 correction written. I have started a new one in safe mode. Still showing a long time. I have also closed the browsers windows that I had open.

 

Edited November 6, 2023 by openam

[openam]

I just sat down by my unraid server, and I can hear a drive doing the occasional click. I got a warning the other day about disk 3 seek errors, and picked up a renewed drive. Is there a way to tell if that's the drive that's making the clicking noise?

[JorgeB]

Post new diags.

[openam]

New diagnostics attached.

palazzo-diagnostics-20231106-0825.zip

[JorgeB]

Logs is clean, there's something writing to disk6, though not much, but stop that and if the speed doesn't improve run the diskspeed docker to check all disks are performing normally.

[openam]

I have docker disabled, I don't have any vms. I was trying to think what would be writing to disk 6, turns out disk 6 has time machine. Cancelled the backup from my laptop. Is there an easy way to figure out what else may be writing to the disk?

[JorgeB]

You can try the open files plugin.

[openam]

I'm still in safemode, so I have 0 plugins.

[openam]

Since the speed hasn't increased with stopping Time Machine. Should I go ahead an reboot into normal mode, and run the "diskspeed docker" (I didn't know this was a thing before)? Also disk 3 is showing error now on dashboard. Attaching new diagnostics.

palazzo-diagnostics-20231106-1100.zip

[JorgeB]

21 minutes ago, openam said:

hould I go ahead an reboot into normal mode, and run the "diskspeed docker" (I didn't know this was a thing before)?

You can, but docker works in safe mode.

[openam]

I don't have `diskspeed docker` installed, and the app store isn't available in safemode. Going to reboot.

[trurl]

28 minutes ago, openam said:

disk 3 is showing error now

  7 Seek_Error_Rate         PO-R--   017   017   067    NOW  199100657

Not an attribute that is normally monitored, but FAILING NOW should be replaced

[openam]

I assumed this is what that clicking noise was coming from. Do I need to finish the parity check before replacing the drive? This is why I bought that renewed drive.

[trurl]

Cancel parity check 

[openam]

I ran the diskspeed docker, and this is what it shows. I guess I'm a bit worried about the wavy ones now 😬.

[JorgeB]

That looks normal.

[openam]

So should I just replace disk 3, since it shows it's failing with the seek error rate? Do I need to be worried about the previous "dirty shutdown" (2nd post)

palazzo-diagnostics-20231107-0820.zip

[JorgeB]

Not sure disk3 SMART error is causing that slow sync, but you can try replacing it.

 

20 minutes ago, openam said:

Do I need to be worried about the previous "dirty shutdown" (2nd post)

That's difficult to answer for certain, there could be sync errors or not, ideally you'd run a parity check before.

[openam]

Well I stopped the system to add the new devices and run pre-clear on it while I ran check on the parity array, and now it says disk 3 is bad. Should I just power back down, and re-check the cable and power connections again, and retry.

 

palazzo-diagnostics-20231107-1125.zip

[JorgeB]

Looks more like a power/connection issue, and the disk dropped offline, try replacing cables, or if you were going to try and replace the disk do it now.

[JorgeB]

ov  7 11:24:13 palazzo kernel: md: disk5 read error, sector=1964161432
Nov  7 11:24:13 palazzo kernel: md: disk5 read error, sector=1964161440
Nov  7 11:24:13 palazzo kernel: md: disk5 read error, sector=1964161448
Nov  7 11:24:13 palazzo kernel: md: disk5 read error, sector=1964161456

 

Hmm, also issues with disk5, so most likely power/cable issues.

[openam]

I actually bought a whole new case and modular power supply. I wanna get rid of the power splitters. It seems like they may be causing me issues. I might just swap everything over tonight.