AgentXXL

If your power is failing regularly (and even if it's stable), consider adding a UPS to protect your systems from 'instant shutdowns'. UPS units are relatively inexpensive and very beneficial when you have irregular power. If the outages are brown-outs (short duration) then a UPS will prevent the problem you're encountering. And longer duration outages can use UPS signalling to the OS to do controlled shutdowns if the power level on the UPS drops too low. Instead of trying to make it work under UD, the real answer is correcting/alleviating your power issues.

Likely the cause - the single USB connection is identifying the 3 drives incorrectly via it's internal controller. Happens with port multiplier setups too. There's no easy way to correct this situation other than putting each of the 3 drives into separate USB enclosures. Or just attach them via SATA to your unRAID if possible, and then transfer the data from them.

Possibly.... note that the 4th drive that is mounting properly also has the same ending sequence as the 3 drives that appear identical in their identification info. Could be a compatibility issue with the 1TB hard drives having a model/serial number that's longer and doesn't differentiate until later in the sequence. Only way to know is to try. If the user chooses to remove the forgotten devices at the bottom of the UD section, that might help eliminate the potential for them to get identified as the same drive.

Simple work-around is to disconnect 2 of the USB drives and only do the rename of the mountpoint with one drive attached. Then remove it and attach the next and lastly do the 3rd.

If you have another PC to use you could boot from a USB Linux distro, mount the image and then share it over your network for copying to the unRAID array. Or copy the data onto another disk that's formatted in a format that UD can use. Mounting it in the QNAP and transferring from there is probably just as simple though.

I waited until the preclears finished before going any further. Sure enough the CPU usage dropped to more normal levels once the preclears were complete. I suspect my system just doesn't have enough CPU cores/threads and PCIe lanes to adequately handle the large number of physical drives attached - 22 SATA connections in total: 2 parity, 18 data, 1 cache SSD, 1 UD-mounted app/VM SSD. I'm watching for a used Threadripper 2950 setup on eBay/local classifieds to upgrade the system, which is currently a 4 core i7-6700K @ 4GHz with 32GB of RAM on a microATX motherboard. I did upgrade to 6.8 now that it's been released and the NVidia build is available. So far things are settling back down in somewhat normal operation but the system upgrade is still planned. I only use preclear mainly to weed out any disks that are bad right from the factory. If you just add a disk and let unRAID do the clear, it takes just as much time but if it fails, I'm still not sure if you're left with a degraded array or if it aborts the addition of the new drive? Regardless, I'm fine with spending the time doing the preclear in advance of adding the disk(s) to the array. As for the remote shares and them possibly slowing down the system, I have indeed seen that when one of them went offline. I've cleaned up those remote mounts and still have two of them active while I shuffle data between them and the unRAID build. So far I've not experienced the lock-ups that others have seen when remote mounts go offline, but as soon as I've moved my data to where I want it, those shares will be unmounted and removed. Dale

I'm preclearing all 4 drives at the same time, but from the looks of things, all 4 will complete at around the 46 - 48 hr mark after starting, so late tomorrow afternoon. Each major stage (pre-read, zero, post-read) takes about the same amount of time and the 1st stage completed in 16 hrs for all 4 drives. The zero stage is 85% complete at 13hrs so also looks like 16hrs to complete. Add another 16 hrs for the post-read and that's 48 hrs. 4 days is twice the time that mine usually take so it could be related to the SATA controller you're using. Regardless, I'm going to wait it out as I'd rather not waste the time already spent doing the preclears. In the past I've been able to reboot unRAID and the preclear was resumeable where it left off. I'm just not sure I want to trust that it'll work the same after all the recent updates. I'm actually curious if the excessive CPU usage reported by the web GUI (and the poor disk I/O I'm seeing) might be that I'm preclearing all 4 drives at the same time. I haven't seen preclear slow down the system in the past, but I've previously only done 2 drives simultaneously. Regardless, I'll wait now unless I can find a way to save the state of the preclear and ensure I can resume after a reboot. EDIT: I've attached the diagnostics that I was able to get by using the 'diagnostics' command from a terminal shell. Dale animnas-diagnostics-20191209-2301.zip

As this question relates to both UD and Preclear plugins, I’ll post it in both support threads. I’m one of the users who hasn’t had any UD or preclear issues until recently. I am currently preclearing 4 x 10TB drives (USB attached) via the UD integration with the preclear plugin. I’m 60% through the zero stage on all 4 drives but have seen some unusual issues today. The only change to unRAID prior to these issues was updating the UD plugin to 2019.12.08 earlier this morning. I have been deploying the various updates for UD and preclear as they’ve been released over the last 2 weeks and up until today things seemed normal. Earlier this afternoon I started seeing a LOT of sporadic buffering/disk I/O issues and the CPU performance on my unRAID system is showing 65 - 75% load pretty consistently. This is based on the unRAID Dashboard tab of the web GUI. TOP in a terminal session reports about 15% utilization. The buffering/disk I/O issues are real as transfers to/from unRAID speed up and slow down at random. I’m also seeing major interruptions to streams from Plex. As I can’t determine why the web GUI shows high CPU load but TOP doesn’t I’m wondering if the last update(s) to UD/Preclear might have broken something? I’m still running unRAID 6.7.2 and note that most of the updates I’m seeing are for 6.8 related issues. I could update to 6.8 RC9 (with NVidia extensions) but I’ve held off. As the preclear is 60% through stage 2 (zeroing), I’m concerned about the upgrade and reboot possibly wiping the preclear state and making me start it over. If I have to I’ll tough it out until the preclear finishes tomorrow on all 4 drives. Any thoughts on whether the preclear status will be saved if I update? Is there a way I can manually stop the preclears and save the state for restarting after the upgrade? I’ve tried shutting down all unnecessary plugins/Dockers/VMs and the unRAID web GUI is still showing 60%+ CPU load and I’m still seeing disk I/O issues. What could cause the discrepancy between TOP in a terminal session and the web GUI? Thoughts? Suggestions? Diagnostics would be attached but 3 attempts to download them have taken over an hour during the collection stage…. another bug/related issue? Tried on 2 separate computers on 3 different browsers (Safari/Firefox on MacOS Mojave and Firefox/Chrome on Ubuntu 18.04. EDIT: I'm attaching the diagnostics that I was able to get by running the 'diagnostics' command from a terminal shell. Still couldn't get the Tools -> Diagnostics option in the web GUI to work. Dale animnas-diagnostics-20191209-2301.zip

As this question relates to both UD and Preclear plugins, I’ll post it in both support threads. I’m one of the users who hasn’t had any UD or preclear issues until recently. I am currently preclearing 4 x 10TB drives (USB attached) via the UD integration with the preclear plugin. I’m 60% through the zero stage on all 4 drives but have seen some unusual issues today. The only change to unRAID prior to these issues was updating the UD plugin to 2019.12.08 earlier this morning. I have been deploying the various updates for UD and preclear as they’ve been released over the last 2 weeks and up until today things seemed normal. Earlier this afternoon I started seeing a LOT of sporadic buffering/disk I/O issues and the CPU performance on my unRAID system is showing 65 - 75% load pretty consistently. This is based on the unRAID Dashboard tab of the web GUI. TOP in a terminal session reports about 15% utilization. The buffering/disk I/O issues are real as transfers to/from unRAID speed up and slow down at random. I’m also seeing major interruptions to streams from Plex. As I can’t determine why the web GUI shows high CPU load but TOP doesn’t I’m wondering if the last update(s) to UD/Preclear might have broken something? I’m still running unRAID 6.7.2 and note that most of the updates I’m seeing are for 6.8 related issues. I could update to 6.8 RC9 (with NVidia extensions) but I’ve held off. As the preclear is 60% through stage 2 (zeroing), I’m concerned about the upgrade and reboot possibly wiping the preclear state and making me start it over. If I have to I’ll tough it out until the preclear finishes tomorrow on all 4 drives. Any thoughts on whether the preclear status will be saved if I update? Is there a way I can manually stop the preclears and save the state for restarting after the upgrade? I’ve tried shutting down all unnecessary plugins/Dockers/VMs and the unRAID web GUI is still showing 60%+ CPU load and I’m still seeing disk I/O issues. What could cause the discrepancy between TOP in a terminal session and the web GUI? Thoughts? Suggestions? Diagnostics would be attached but 3 attempts to download them have taken over an hour during the collection stage…. another bug/related issue? Tried on 2 separate computers on 3 different browsers (Safari/Firefox on MacOS Mojave and Firefox/Chrome on Ubuntu 18.04. Dale

I've been reading through this thread about the random unpacking failure with nzbget. I too started experiencing it about 2 weeks ago, at the time using the linuxserver.io container. After reading through their support thread, it appears that this issue affects both it and the binhex container. I've been using the binhex container for a few days to try and troubleshoot but now the issue is occurring very frequently. Scheduling restarts of the container every 30 minutes or 60 minutes hasn't worked as one of the downloads has been stuck repairing and the time estimate is more than 1hr. Every time the container restarts, the repair process kicks off again. I'm about to manually use the par commands on my Windows or Mac box to try the repair instead, but in the meantime other downloads have failed unpacking and are also looking like they need repair. No reports of health issues during the downloads so I wonder why the par checks are failing when they shouldn't and if that's part of the cause for stuck unpacks. I'm thinking of trying a sabnbzd container next but thought I'd at least post my comments. I can post some logs if required.

Although I'm not one of the experts here, 46 hours for one full pass of the preclear (pre-read, zero and post-read) isn't unreasonable. I get similar numbers for my 10TB drives, but my 8TB preclears usually take about 40 hrs. I've seen similar errors in my syslog as well, but as the drive passed the preclear and as no SMART errors were of concern, I've used those drives with no issues. I suspect you are fine, but I too would be interested to know why random sectors report errors that don't affect the preclear. Perhaps those are dealt with through the drive's sector reallocation list but SMART has shown no re-allocated sectors for any of the drives that saw those errors. In any case, I think you're safe to not be concerned with using the drive. Hope it all goes well... now that I've had my connection issues resolved for a month, I'm finding the system to be extremely stable. I haven't even seen one UDMA CRC error, which are fully recoverable but do tend to indicate connection issues.

Thanks @DanielCoffey and @Pauven for the quick responses. I guess I hadn't read up enough on what the tool does, but it was recommended to use since the tunables are what's been recommended to try and improve write performance. As it's only doing reads, I do feel a lot safer using it. My 'server room' is air conditioned so I might just set the air conditioner to run it cooler than normal to try and combat the heat generation of the test. Looking forward to the results!

I have yet to try this tool but as more drives have been added to my system, write speeds have definitely taken a hit. I did change a couple of the tunables from the defaults but that was when the system was running on fewer than 8 drives. I'm now at 16 drives so I realize parity calc will slightly slow down writes. That said, I'd like to optimize and try this tool, but one thing I haven't seen is mention of the safety of using the tool with array drives that are all populated with data. As the tool must do writes, how much danger is there that it could cause disk corruption like @dalben had happen? I haven't been able to read the whole thread and don't want to since the latest version is a far cry from what it was when 1st released. I'm sure it's very safe to use, but before I try I just wanted to query how often users have reported drive failure/corruption while running the tool? Also, should I reset the tunables to default before trying the tool? Thanks!

One other bug/minor annoyance. I use the 'Synchronze Folders...' function under the Tools menu to migrate data from outside the array. It works great and is easy to resume if you have to reboot or there's some sort of restart of Krusader/Docker. However, the window that opens always has the 'Name' field for the left pane expanded far out to the right, off the screen/window view. I grab the horizontal scroll bar at the bottom and slide it all the way over to the right and then resize the left pane 'Name' field so that both it and the right pane headings are visible. I've tried resetting this and closing/exiting Krusader properly to save the settings (like it does for other Krusader windows) but it never works. This does appear to be an issue with the Docker container as Krusader on my Ubuntu system seems to work fine. Similarly, any copy/move windows also open in a default location of the center of the screen, whereas I would like them to open off to the far right edge so my main Krusader window isn't obscured. Very minor issues, but would be nice if fixable.

I'm having issues with this method for extracting files from a rar set. For some reason the extract starts fine, but randomly it will stop popping up a Information --- Krusader error dialog saying 'access denied' to the file within the rar set. Sometimes it seems to work with no issues, but most of the time it fails on a rar set that's larger than 4GB in size, for example one containing an iso file. My Krusader log entries from an attempt made just a few minutes ago.... 2019-08-22 13:19:53,732 DEBG 'start' stderr output: kdeinit5: Got EXEC_NEW '/usr/lib/qt/plugins/kf5/kio/file.so' from launcher. kdeinit5: preparing to launch '/usr/lib/qt/plugins/kf5/kio/file.so' 2019-08-22 13:19:53,740 DEBG 'start' stderr output: kdeinit5: Got EXEC_NEW '/usr/lib/qt/plugins/kio_krarc.so' from launcher. kdeinit5: preparing to launch '/usr/lib/qt/plugins/kio_krarc.so' 2019-08-22 13:20:28,795 DEBG 'start' stderr output: kdeinit5: PID 225 terminated. This has happened with many different rar sets in the past but I got around it by copying the rar set to another system and extracting there, then copy the extracted files back to my unRAID. Note that this occurs whether I try to extract to the array, to the cache or to an unassigned device (SSD or hard drive). Any suggestions?

It's all good... your knowledge on other unRAID issues has helped me and many others countless times. You can't be expected to know about things you don't use or deal with yourself, at least until you get informed. I don't have a wife, but my pets often remind me that I'm ignoring them... not quite the same as ignorant, but similar! Ahh, so a Neil Young or Highlander fan are you? Of course that phrase has appeared in a few other pop culture references, including Kurt Cobain's suicide note and songs from many. Now that I've essentially been forced into retirement from my main career, I've joined the 'burn-out' culture in the form of my medical cannabis usage. Though not entirely true... I've found some of the sativa strains really do give me a burst of energy, focus and creativity! 🤣

A zero only pass of the plugin still takes the same amount of time as the pre-read and post-read stages would, so it's not just writing a signature... it's actually writing zeroes to every sector of the drive before it writes the signature. I agree that if all it did was write a signature, which would take less than a minute, you're never going to be sure.

If it does stop addition of the new drive, then that's great. For now I'll continue to use the plugin for the basic stress test on new drives, and for a simple zero pass for older but still functional drives. I'm potentially going to build a 2nd unRAID server to keep my media server isolated from my personal/work/backup data. I know I could just make a separate share with different credentials/access, but a 2nd unRAID box that I keep at a friends place would give me an offsite backup.

So in the long run, I've removed the SAS/SATA backplanes from the Norco 4220. I'm still using it as my enclosure although I'm watching eBay for a decent deal on a Supermicro chassis with at least 24 bays. As for my drives, I'm now direct cabled to them in the Norco enclosure. I purchased some new miniSAS (SFF-8087 host) to 4 SATA forward (HD targets) breakout cables. I've essentially lost the hot-swap capability but for a home server like this, that isn't an issue. The bigger concern was ensuring that the mess of cabling doesn't block airflow to and through the drive bays. The Norco isn't great for this anyhow as there's next to no airspace between drives. Because of this the drive temps run hotter than I'd like. I replaced the standard fans that Norco supplied with 2 x 140mm fans which both move a lot more air and are also much quieter than the stock fans. It required some metal work to open up the mounting plate to accommodate the larger fans. If you're fine with direct cabling and can live with a 15 bay enclosure, the Rosewill RSV-L4500 4U case is an inexpensive option from the Rosewill official store on eBay. Note that I also ordered a certified genuine LSI 9201-16i as the initial one was a Chinese knock-off. That said, the Chinese knock-off is now performing well with the drives direct-cabled, so in the end I didn't need to order the genuine one. I went the inexpensive direct cable route as I'm on disability income and just don't have enough disposable income to buy a Supermicro enclosure yet. I'm saving up for one as that seems to be one of the most recommended for those needing space for a lot of drives. The other solution is to replace the 4TB and 6TB drives with more 10TB+ drives so I can reduce the drive count. Right now I'm sitting at 114TB with 6 x 10TB, 4 x 8TB, 1 x 6Tb and 4 x 4TB. And I still have 3 bays left in the Norco for more array drives - it would be 5 but I've used the first 2 bays for my dual parity drives (which are Ironwolf NAS vs the rest of my drives which are shucked from USB enclosures).

For new drives I still prefer to do at least one pass (all 3 stages) of the preclear script/plugin. For drives that I've had in operation for a while, I'll typically only do a zero pass as long as the SMART report from the drive shows nothing concerning. Of course since unRAID will do that itself, it's not required to use the plugin. I just prefer being able to do a zero of the drive before adding it to the array. It takes the same amount of time whether you do the zero pass using the plugin or by adding the drive to the pool, but if by chance there's a failure during the zero, I'd rather know about it before adding it to the pool. One question about just adding a drive to the pool and letting unRAID do the zero: if by chance there's errors or a failure of the drive while unRAID is zeroing the drive, does it still add it to the pool? If so this means obtaining a replacement drive right away. If you do the zero pass with the plugin, you'll have the option of delaying the addition of the drive until it's replaced/repaired. Most of us tend to buy drives only when on sale, so using the plugin seems safer to me, although I do have dual parity in use.

This thread is a little long and the original post is dated but still hoping to find some answers/suggestions. I have a 2 month old unRAID build based on a Norcotek RPC-4220 storage enclosure. I'm experiencing lots of UDMA CRC errors and suspect some of it may be due to half the drive bays not having anything installed since I purchased the unit 6+ years ago. It ran fine as my FreeNAS, but was only equipped with 10 drives installed, in addition to the 2 internal 2.5" slots I used for SSDs. Now that I've migrated it to unRAID, only the 2 internal slots and the 1st shelf/backplane of 4 SATA drives are connected to the motherboard SATA ports. The remaining 4 shelves/backplanes (16 bays) are attached to a LSI 9201-16i with 30" long SFF-8087 to SFF-8087 miniSAS cables. The LSI card itself may not be genuine and I have a case submitted to LSI asking about it. I've also ordered new 6G certified miniSAS cables that are only 11.5" long. The existing cables (new, came with the card from the Chinese retailer via eBay) are so long they are looped back and forth a few times inside the case. I'm concerned this could lead to cross-talk and other errors so that's why I've ordered the shorter cables. Getting back to the enclosure, I'm concerned that the 10 bays which had nothing connected for 6+ years have oxidation or other connection issues that could also be the source of my errors. I'm prepared to disassemble and clean the connectors with contact cleaner and then apply contact stabilizer like Stabilant 22. This will be after I get unRAID back healthy - one 8TB drive is currently disabled and contents emulated. The rebuild from parity will commence soon, just waiting for a zero pass from the preclear plugin to re-confirm my spare 8TB. I'll also move the 8TB to a motherboard connected SATA port for the rebuild, as I'm still unsure if the LSI controller could also be partly or wholly at fault. As a last resort, I'm thinking of purchasing a new enclosure, which the above backstory explains. However there hasn't been any major enclosure recommendations for quite a while - the thread has been fairly dormant. Does anyone have any current model suggestions for a relatively inexpensive storage case with at least the same 20 drive bays as my Norcotek RPC-4220? Thanks in advance for any suggestions! Dale

The drive trays were left in all slots to prevent dust, etc from entering the unused bays, but yes, open air exposure for the SAS/SATA connectors on 2.5 shelves. I'm more than willing to do the cleanup and plan to apply some contact stabilizer like the Stabilant 22 I use for a lot of my audio gear. I just want to wait until the new MiniSAS cables arrive as I'm pretty convinced the current extra-long ones are part of the issue too.

Everyone has their own idea of what qualifies as a successful stress test of a new drive. In the past before using unRAID, I often would use the drive manufacturers test tools for at least one pass. Sometimes though it was as simple as doing a full format (not a quick format) of the drive (when using Windows). For me, I've not had many occurrences of drive 'infant mortality' in recent years so I'm now comfortable with a single 3-phase pass of the pre-clear plugin. With 8TB or 10TB drives, that usually takes up to 48hrs to do all 3 phases. Around 36 hours for the 8TB drives and around 45 hours for the 10TB ones. Right now I'm less concerned about the new drives but more with cabling, controller and potential oxidation on the hot-swap SATA backplanes in my Norcotek enclosure. Half of the 20 drive bays have never had anything installed in them until recently, but having moved from FreeNAS to unRAID, I now have a number of those previously unused bays in operation. They sat for over 6 years with nothing connected so oxidation of the connectors is a real potential issue that I now have to address. First I just want to get my array back to stable without the one drive - for some reason always Disk 8, that keeps disabling after reboots or adding another new drives to the array. I've got a new topic created about 10 minutes ago asking for suggestions on how to proceed. Regardless, I'm not experiencing the same bugs that some are with the pre-clear plugin on 6.7.2. I did have an unRAID VM setup on a Windows system that had an eSATA connection pass-through so that I could use it for pre-clearing, but the Windows box suffered a massive hardware failure and so that option isn't available until I get it back up and running. For now I'm back to doing pre-clears on my 'production' unRAID.

It's approximately the same amount of time for each of the 3 phases of a single-pass pre-clear. Since your pre-read took about 6 hours, the entire process should be complete in about 18 hours. There are 5 steps listed in the status, but only 3 actual phases. EDIT: Just noticed your write speed is quite a bit lower than your read speed. I haven't seen this with the drives I've pre-cleared... writing is usually slower, but not usually as much as what your screenshot shows. Do you have other tasks doing writing to other drives on that system? Or perhaps some DMA bound processes if your SATA controller is older and resides on the older PCI bus? Many older motherboards have this issue if it's a motherboard SATA port.

Are you trying to mount them after adding to the array or under Unassigned Devices? Either way they need to be formatted. But for disks that are pre-cleared and added to the unRAID array, the Format option is on the Main page just below the Start/Stop button. Don't format them under Unassigned Devices if you want to add them to the array, or you'll have to run at least a zero stage to pre-clear them again, or let unRAID do it once added to the array.