DontWorryScro

Following. Just curious. Say you boot into Safe mode. then what? what is the troubleshooting process?

I too have been getting this. This happens while my Plex is idle, even. Nothing is being watched. Ive deleted EasyAudioDecoder before due to some suggestion i stumbled on before but clearly that hasnt worked. I suspect it just gets redownloaded on the next restart or update. Whats the answer here? For the sake of thoroughness I'll also include the transcode settings i have in Unraid and in Plex to confirm that's not the issue: Unraid lsio Plex containter Plex

also noticed this in my logs today. I'm on 6.11.5 Still nothing ey? php-fpm[14714]: [WARNING] [pool www] server reached max_children setting (50), consider raising it

this? root@Snuts:~# cat /etc/cron.d/root # Generated docker monitoring schedule: 10 0 * * * /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/dockerupdate check &> /dev/null # Generated system monitoring schedule: */1 * * * * /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null # Generated mover schedule: 40 1 * * * /usr/local/sbin/mover |& logger # Generated parity check schedule: 0 3 * 2,5,8,11 0 [[ $(date +%e) -le 7 ]] && /usr/local/sbin/mdcmd check &> /dev/null || : # Generated plugins version check schedule: 10 0 * * * /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugincheck &> /dev/null # Generated btrfs scrub cache nvme schedule: 25 2 7 * * /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub start /mnt/cache_nvme -r &> /dev/null # Generated Unraid OS update check schedule: 11 0 * * * /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/unraidcheck &> /dev/null # Generated Schedule for CA Auto Turbo Mode 0 0 * * * /usr/local/emhttp/plugins/ca.turbo/scripts/turboSchedule.php disable 420 > /dev/null 2>&1 # Generated cron settings for docker autoupdates 10 2 * * * /usr/local/emhttp/plugins/ca.update.applications/scripts/updateDocker.php >/dev/null 2>&1 # Generated cron settings for plugin autoupdates 5 0 * * * /usr/local/emhttp/plugins/ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1 # CRON for CA background scanning of applications 35 * * * * php /usr/local/emhttp/plugins/community.applications/scripts/notices.php > /dev/null 2>&1 # Generated system data collection schedule: */1 * * * * /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null # Generated schedules for parity.check.tuning */6 * * * * /usr/local/emhttp/plugins/parity.check.tuning/parity.check.tuning.php "monitor" &>/dev/null # Purge recycle bin at 5:00 AM on the first day of the week: 0 5 * * 0 /usr/local/emhttp/plugins/recycle.bin/scripts/rc.recycle.bin cron &> /dev/null # Refresh Recycle Bin trash sizes every five minutes: */5 * * * * /usr/local/emhttp/plugins/recycle.bin/scripts/get_trashsizes &> /dev/null # Generated cron schedule for user.scripts */10 * * * * /usr/local/emhttp/plugins/user.scripts/startCustom.php /boot/config/plugins/user.scripts/scripts/chown_soulseek_complete/script > /dev/null 2>&1 0 */1 * * * /usr/local/emhttp/plugins/user.scripts/startCustom.php /boot/config/plugins/user.scripts/scripts/delete media that is stuck/script > /dev/null 2>&1

ok with Testing mode on the failure to resume the data rebuild occurred again over night. The rebuild is still paused as I type this, even over 7 hours after the mover operation finished. Diagnostics attached. Edit: I also just now noticed after posting this that there was a Parity Check Tuning plugin update available to me now. I've gone ahead and updated. I also manually restarted the Parity-Sync. 2nd Edit: Shortly there after I notification popped up from Parity Check Tuning at 12:42 that "[SNUTS] mover no longer running" I'll just paste in the additional information that can be found from the logs after I manually resumed the parity-sync: Jul 4 12:40:03 Snuts kernel: mdcmd (41): check resume Jul 4 12:40:03 Snuts kernel: md: recovery thread: recon P ... Jul 4 12:40:15 Snuts Parity Check Tuning: TESTING: ... not a parity check so always treat it as an automatic operation Jul 4 12:40:15 Snuts Parity Check Tuning: TESTING: actionDescription(recon P, 1, AUTOMATIC, 1) = Parity Sync/Data Rebuild Jul 4 12:40:15 Snuts Parity Check Tuning: DEBUG: Parity Sync/Data Rebuild running Jul 4 12:40:22 Snuts root: plugin: running: anonymous Jul 4 12:40:22 Snuts root: plugin: creating: /boot/config/plugins/parity.check.tuning/parity.check.tuning-2023.07.04.txz - downloading from URL https://raw.githubusercontent.com/itimpi/parity.check.tuning/master/archives/parity.check.tuning-2023.07.04.txz Jul 4 12:40:22 Snuts root: plugin: running: /boot/config/plugins/parity.check.tuning/parity.check.tuning-2023.07.04.txz Jul 4 12:40:22 Snuts root: plugin: running: anonymous Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING: ... not a parity check so always treat it as an automatic operation Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING: actionDescription(recon P, 1, AUTOMATIC, 1) = Parity Sync/Data Rebuild Jul 4 12:40:22 Snuts Parity Check Tuning: DEBUG: Parity Sync/Data Rebuild running Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG ----------- CONFIG begin ------ Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG /boot/config/forcesync marker file present Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG tidy marker file present Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG progress marker file present Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG automatic marker file present Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG paused marker file present Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG disks marker file present Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG mover marker file present Jul 4 12:40:22 Snuts Parity Check Tuning: Versions: Unraid 6.11.5, Plugin Version: 2023.06.25 Jul 4 12:40:22 Snuts Parity Check Tuning: Configuration: Array#012(#012 [parityTuningScheduled] => 0#012 [parityTuningManual] => 0#012 [parityTuningAutomatic] => 0#012 [parityTuningFrequency] => 0#012 [parityTuningResumeCustom] => 5,15,25,35,45,55 * * * *#012 [parityTuningResumeDay] => 0#012 [parityTuningResumeHour] => 1#012 [parityTuningResumeMinute] => 15#012 [parityTuningPauseCustom] => 0,10,20,30,40,50 * * * *#012 [parityTuningPauseDay] => 0#012 [parityTuningPauseHour] => 7#012 [parityTuningPauseMinute] => 30#012 [parityTuningNotify] => 1#012 [parityTuningRecon] => 0#012 [parityTuningClear] => 0#012 [parityTuningRestart] => 1#012 [parityTuningMover] => 1#012 [parityTuningBackup] => 1#012 [parityTuningHeat] => 0#012 [parityTuningHeatHigh] => 3#012 [parityTuningHeatLow] => 8#012 [parityTuningHeatNotify] => 1#012 [parityTuningHeatShutdown] => 0#012 [parityTuningHeatCritical] => 2#012 [parityTuningHeatTooLong] => 30#012 [parityTuningLogging] => 2#012 [parityTuningLogTarget] => 0#012 [parityTuningMonitorDefault] => 17#012 [parityTuningMonitorHeat] => 7#012 Jul 4 12:40:22 Snuts Parity Check Tuning: TESTING:CONFIG Creating required cron entries Jul 4 12:40:22 Snuts Parity Check Tuning: DEBUG: Created cron entry for 6 minute interval monitoring Jul 4 12:40:23 Snuts Parity Check Tuning: DEBUG: Updated cron settings are in /boot/config/plugins/parity.check.tuning/parity.check.tuning.cron Jul 4 12:40:23 Snuts Parity Check Tuning: TESTING:CONFIG ----------- CONFIG end ------ Jul 4 12:40:23 Snuts root: plugin: parity.check.tuning.plg updated Jul 4 12:40:30 Snuts Parity Check Tuning: TESTING: ... not a parity check so always treat it as an automatic operation Jul 4 12:40:30 Snuts Parity Check Tuning: TESTING: actionDescription(recon P, 1, AUTOMATIC, 1) = Parity Sync/Data Rebuild Jul 4 12:40:30 Snuts Parity Check Tuning: DEBUG: Parity Sync/Data Rebuild running Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING: ... not a parity check so always treat it as an automatic operation Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING: actionDescription(recon P, 1, AUTOMATIC, 1) = Parity Sync/Data Rebuild Jul 4 12:42:24 Snuts Parity Check Tuning: DEBUG: Parity Sync/Data Rebuild running Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR ----------- MONITOR begin ------ Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR /boot/config/forcesync marker file present Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR tidy marker file present Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR progress marker file present Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR automatic marker file present Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR paused marker file present Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR disks marker file present Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR mover marker file present Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR isArrayOperationActive - parityTuningActive:1, parityTuningPos:2622003844 Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR ... not a parity check so always treat it as an automatic operation Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR backGroundTaskHandling: configName=parityTuningMover, value=1, isMoverRunning=0, Array: Active=1, Paused=0 Jul 4 12:42:24 Snuts Parity Check Tuning: Send notification: mover no longer running: Jul 4 12:42:24 Snuts Parity Check Tuning: TESTING:MONITOR ... using /usr/local/emhttp/webGui/scripts/notify -e Parity Check Tuning -i normal -l /Settings/Scheduler -s [SNUTS] mover no longer running Jul 4 12:42:25 Snuts Parity Check Tuning: TESTING:MONITOR Deleted mover marker file Jul 4 12:42:25 Snuts Parity Check Tuning: TESTING:MONITOR Deleted paused marker file Jul 4 12:42:25 Snuts Parity Check Tuning: TESTING:MONITOR ... no action required as array operation not paused Jul 4 12:42:25 Snuts Parity Check Tuning: TESTING:MONITOR backGroundTaskHandling: configName=parityTuningBackup, value=1, isBackupRunning=0, Array: Active=1, Paused=0 Jul 4 12:42:25 Snuts Parity Check Tuning: TESTING:MONITOR Temperature monitoring switched off Jul 4 12:42:25 Snuts Parity Check Tuning: TESTING:MONITOR ----------- MONITOR end ------ Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING: ... not a parity check so always treat it as an automatic operation Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING: actionDescription(recon P, 1, AUTOMATIC, 1) = Parity Sync/Data Rebuild Jul 4 12:48:26 Snuts Parity Check Tuning: DEBUG: Parity Sync/Data Rebuild running Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR ----------- MONITOR begin ------ Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR /boot/config/forcesync marker file present Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR tidy marker file present Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR progress marker file present Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR automatic marker file present Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR disks marker file present Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR isArrayOperationActive - parityTuningActive:1, parityTuningPos:2642689172 Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR ... not a parity check so always treat it as an automatic operation Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR backGroundTaskHandling: configName=parityTuningMover, value=1, isMoverRunning=0, Array: Active=1, Paused=0 Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR backGroundTaskHandling: configName=parityTuningBackup, value=1, isBackupRunning=0, Array: Active=1, Paused=0 Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR Temperature monitoring switched off Jul 4 12:48:26 Snuts Parity Check Tuning: TESTING:MONITOR ----------- MONITOR end ------ snuts-diagnostics-20230704-1236.zip

Turns out one of my parity drives fell off again last night so I replaced all the data and power cables to that cluster of drives and am rebuilding now. This means I will be able to test this again when Mover runs tonight. Ill activate the testing mode you mentioned to see if I can capture what is going on if it does in fact repeat itself.

OK I managed to re-enable the parity drive referenced in my original post that had dropped off (sdi) and after 3 days of rebuilding and finishing without issue it was fine for a couple days. Today I checked on the server to find that same drive had dropped off with i/o errors. Seems like this has been happening during scheduled mover operations overnight. Can you take a look at the latest diagnostics I've attached and tell me if it is exhibiting the same behavior as before? Do I need to just completely re-cable the sata power and mini SAS cables to my Unraid NAS to rule that out? The PSU is an EVGA SuperNOVA 1000 G+ 80 Plus Gold so I merely assume the power itself is probably adequate. snuts-diagnostics-20230703-1034.zip

My rebuild has completed however I do have another drive that I have been meaning to swap in which will require another 3 day rebuild. If need be I am willing to take steps to help you troubleshoot what is going on. I manually made the fixes to the php file that were represented in the pull request by user robobub two posts before this. Not sure if that will have any effect but thought I'd let you know.

oddly i have mine set this way but for whatever reason it's not working properly. I feel like there's some cron job running somewhere that is interrupting how this is supposed to work for me. I have both those options you mentioned set to Yes but what happens for me is the Parity-Sync will indeed pause when mover starts running nightly at 1:40am. Then in the morning i return to see that the Parity-Sync never resumed. I'll check the logs to see that the mover did indeed finish a while ago. So why did the Parity-Sync not resume? At this point I can either wait for the random notification to pop up mid day that mover has finished and parity sync is resuming (even though I know mover to have finished hours previously) or I manually resume. Maybe if I paste my settings someone can tell me where I brain farted? snuts-diagnostics-20230629-1425.zip

so am i just looking at buying a higher wattage PSU? heres the other thing. the three drives that have popped offline this week are all connected differently. I was getting btrfs errors on the cache drive which was a m.2 nvme on a pcie adapter plugged directly into the motherboard on the main NAS. i swapped that out for an enterprise SSD today. the initial drive that was giving similar errors to this latest parity drive dropoff was in a different case with its own PSU in an HP expander connected DAS. This parity drive that fell off tonight during a data rebuild of the initial HDD is connected via an LSI HBA on the main NAS. seems like an odd confluence of events in the span of 3 days. all separate. all related to power and cables? i mean of course its possible. just seems like something else is going on and its killin me.

why do i keep getting random errors on my drives? theres no rhyme or reason. I just replaced a HDD that was giving i/o errors. then replaced a cache drive also giving errors that looked like it was a failing drive. and now one of my parity drives have just dropped offline with io errors while rebuilding data on that first data drive. am i chasing ghosts here? this all happening within 3 days, am i having some sort of hardware issue that i cant pinpoint? with one of my 2 parity drives down if i lose one more drive during this data rebuild i am screwed. At this rate Ill have another failure before this rebuild finishes. Is this maybe RAM going bad? I have ECC Example of error message on the latest issue on one of my parity drives: Jun 23 23:16:52 Snuts kernel: I/O error, dev sdi, sector 2269569328 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0 Jun 23 23:16:52 Snuts kernel: sd 1:0:7:0: [sdi] tag#2497 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Jun 23 23:16:52 Snuts kernel: sd 1:0:7:0: [sdi] tag#2497 Sense Key : 0x2 [current] Jun 23 23:16:52 Snuts kernel: sd 1:0:7:0: [sdi] tag#2497 ASC=0x4 ASCQ=0x0 Jun 23 23:16:52 Snuts kernel: sd 1:0:7:0: [sdi] tag#2497 CDB: opcode=0x8a 8a 00 00 00 00 00 87 46 e5 30 00 00 04 00 00 00 Jun 23 23:16:52 Snuts kernel: I/O error, dev sdi, sector 2269570352 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0 ran a smart test on the drive and it came back no errors. Diagnostics attached. snuts-diagnostics-20230623-2348.zip

I am honored to have been mentioned in the Release notes for the lstat error fix lol. Thanks for all you guys do!

ok so to fix this behavior I can merely set "Move All from Cache-Yes shares when disk is above a certain percentage:" to NO?

at the moment.

I did notice in all my system log error spam of move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/* that it seems this only happens when Mover runs on its nightly schedule. When I run it manually I do not get these missing file/directory messages.

I am completely up to date on plugins, have the very latest and greatest and yet im still getting tons of: move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/* any idea why this is still happening?

getting a ton of these lately move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/[etc] why? When I track down the problematic directories on the cache drive I find that indeed the files do not exist. Just the empty folders they were living in.

I stumbled across another similar post where Squid suggested uninstalling Dynamix Share Floor. I uninstalled and sure enough I have control of my Scheduler settings again.

This ever get sorted? I am having a similar problem and no one seems to be able to figure it out. I'm necroing this bc it was never solved and I have this problem now. All settings in Scheduler won't apply. They either launch a tab that has nothing in it and shows "error: /update.php: missing csrf_token" in logs or in the case of Mover the Apply button never turns orange to apply the updated setting. I've made sure every plugin is up to date. I've confirmed that no other browsers on any PCs or smartphones are even open let alone pointed to my unnraid dashboard. Ive rebooted a few times. Ive run Docker Safe New Perms just for the heck of it I really dont know what else to do. My last resort seems to be updating to the latest RC of Unraid and crossing my fingers? Update: Sorry, I stumbled across another similar post where Squid suggested uninstalling Dynamix Share Floor. I uninstalled and sure enough I have control of my Scheduler settings again. snuts-diagnostics-20230428-0904.zip

Upon further investigation it seems every single thing in all of Scheduler is borked. I thought it was only Parity Tuning because that was the only thing I had been trying to update. Would my best bet at this point be trying to update to the latest RC of Unraid to see if it fixes my Scheduler?

Yikes and I just now noticed that my issues arent just with Parity Tuning. I just now tried to change frequency of Mover from Daily to Monthly just to confirm that other settings weren't also acting up. And unfortunately they are. Changing the frequency of Mover does not get recognized as changed. The Apply button stays greyed out. I'm also attaching another Diagnostics file since I have uninstalled and reinstalled Parity Tuning since the last time I had attached diagnostics and behavior is slightly different now. Any help is appreciated. This is sort of a problem for me if I can't change settings in Scheduler. Tell me you see something in my diagnostics that is a red flag now. Crossing fingers. snuts-diagnostics-20230427-1817.zip

Any other ideas? There's no open browsers. Do my diagnostics show anything revealing?

I just went around and shut down any and all browsers open on any PC/phone to be sure and I still am getting the error. I uninstalled and reinstalled the plugin and now it is set to the default parameters but I still cant update the settings at all without getting the missing csrf_token and a new tab that never loads anything opened when I click on Apply. Maybe Ill try a reboot of Unraid. It did finish a 14TV data rebuild yesterday and hasn't been restarted since then. Maybe a nice reboot will shake out the cobwebs. Edit: No dice. A full reboot did not change this behavior.

error: /update.php: missing csrf_token this is in my error logs. When I try to apply Parity Tuning settings it just launches a new blank tab and the above message shows up in the logs. Ive made sure I am up to date on all Plugins. What else can I do? snuts-diagnostics-20230423-1703.zip

Kinda the same deal here. Noticed the other day I was getting Internal Server Error when trying to log in. Luckily I had one tab open in Nextcloud on my laptop. It seems if I was already logged in I have access but the act of logging in is failing. Anyone with any ideas?