AgentXXL

So today I added another Ironwolf 10TB to my system and started pre-clearing/stress testing before I allocate it as a 2nd parity drive. I powered down to add the drive to a bay on my enclosure. As I wanted both parity drives on the motherboard SATA controllers, I had to move my recently rebuilt 8TB (Disk 8; rebuilt successfully earlier last week) to another bay. Upon rebooting, the recently rebuilt 8TB (Disk #8) threw a bunch of read errors. I suspected the bay might be at fault so I stopped the array and set it to not auto-start. I then powered down and moved the 8TB drive to another bay that had been previously occupied by a HGST 4TB which never threw any errors, read/write/UDMA CRC or otherwise. The HGST 4TB was relocated to another shelf and appears to be fine. As I had just finished rebuilding onto this 8TB drive earlier in the week, I decided it was safe to go ahead and use the 'Re-enable a drive' procedure. Note that on 6.7.2, the last step doesn't seem to apply as there was no confirmation box to select. Just the Start button and a description saying that a replacement disk was found and starting the array will do a parity sync/rebuild. This is what I wanted. After starting the array, the disc did start the rebuild. It did appear to be working fine, but I just checked and about 6 hrs in it started throwing some write errors. unRAID appears to have paused the parity rebuild but without thinking about it, I just clicked on Resume. The parity rebuild is continuing but I'm concerned about the write errors. EDIT: After I clicked resume, it's no longer doing a parity rebuild on Disk 8 - it's doing a parity read check. Not sure how this affects the partially completed rebuild on this drive before it started throwing write errors. What should I do? Should I pause the rebuild? Cancel the rebuild? Let the rebuild continue? I'm also seeing a lot of the following errors from during the parity rebuild and at least one since I resumed the rebuild: Jul 14 22:18:03 AnimNAS kernel: sd 2:0:1:0: Power-on or device reset occurred Jul 14 22:18:03 AnimNAS rc.diskinfo[5009]: SIGHUP received, forcing refresh of disks info. Diagnostics attached. I do still have one more 8TB drive that I could use as another replacement, but I need some suggestions on how to safely proceed. I'm fairly certain the write errors are not actual disk errors but controller/cable/SATA backplane related. I do have spare bays on other shelves that I can move the current rebuilding 8TB to, or to add the spare 8TB to. One other note: the new 10TB Ironwolf is also doing a pre-clear/stress test run before I add it as a 2nd parity drive. I'm willing to stop this procedure and leave it alone until I get the system back to having a valid Disk 8, on either the existing rebuilding disk, or the spare replacement. Note that the bay which has been producing previous errors is currently unused and will remain so until I can get to doing a full disassembly and clean of all of the SATA connections on all 5 backplane shelves. I've also got new 6G certified MiniSAS cables on the way as the extended length cables that came with my LSI 9201-16i are too long, looped back and forth a couple of times inside the case and potentially a cause of cross-talk/errors. Any help appreciated. Thanks! animnas-diagnostics-20190715-0426.zip

It's approximately the same amount of time for each of the 3 phases of a single-pass pre-clear. Since your pre-read took about 6 hours, the entire process should be complete in about 18 hours. There are 5 steps listed in the status, but only 3 actual phases. EDIT: Just noticed your write speed is quite a bit lower than your read speed. I haven't seen this with the drives I've pre-cleared... writing is usually slower, but not usually as much as what your screenshot shows. Do you have other tasks doing writing to other drives on that system? Or perhaps some DMA bound processes if your SATA controller is older and resides on the older PCI bus? Many older motherboards have this issue if it's a motherboard SATA port.

Thanks @johnnie.black - just encountered the same error after a rebuild on a replacement disk. While I'm sure the rebuild was successful, I want to mount the old suspect drive (which appeared fine on my Ubuntu system) so I can use the Krusader 'Synchronize' function to compare and verify the files from the suspect disk with the replacement disk after the rebuild. The command given worked like expected and I'm now comparing.

You can use Midnight Commander (mc) from terminal or you can try the Krusader docker container. I use Krusader but make sure you take the time to configure its template with your mount paths and of course, follow the Krusader setup guidelines.

Are you trying to mount them after adding to the array or under Unassigned Devices? Either way they need to be formatted. But for disks that are pre-cleared and added to the unRAID array, the Format option is on the Main page just below the Start/Stop button. Don't format them under Unassigned Devices if you want to add them to the array, or you'll have to run at least a zero stage to pre-clear them again, or let unRAID do it once added to the array.

Thanks @BiGs... I definitely have things configured as recommended on the LSI controller. And I've been lucky that this Norcotek unit has been quite reliable when it was in use as my FreeNAS server, but that system used 8 SATA ports on the motherboard and only 6 more from an older Intel HBA. So some of the bays have never had drives connected since I bought it over 6 years ago. I've ordered some short 6G certified MiniSAS cables - 11.5" compared to the 30" existing. I'm hoping they'll help reduce crosstalk as right now the excessive length of the existing cables has them looped back and forth a couple of times inside the case. Far too much length for what's needed. When the new cables arrive, I'm going to shut the whole system down so I can disassemble the unit to do a clean and apply some contact cleaner/stabilant on the MiniSAS connectors, the 4 x SAS/SATA connectors per shelf and the power connectors on each shelf. I'll especially pay attention to the shelves that hadn't seen any use until I built my unRAID setup in it 2 months ago. There could be oxidation on the connectors as they never saw any use until recently. Regardless, the parity rebuild of my failed disc is well underway - should be done sometime tomorrow. After that I'll use something like FreeFileSync (or maybe even Krusader natively in its unRAID docker) to compare the files on the rebuilt drive to the files on the original drive that is suspect. Since the original drive is still fully mountable and browseable on my Ubuntu system, I'm suspecting it's a controller/cable/oxidation issue that caused the failure after adding the new 10TB drive. If all files appear the same after the comparison, I'll re-do a pre-clear and stress test on the original 8TB and hopefully put in on the shelf as a spare ready to use should I experience another failure. And hopefully the cleanup of the connectors and application of some Stabilant 22/Deoxit will help where oxidation may have played a role. That and I'm pretty sure I'll also update the firmware on the LSI. Even though it's not likely at fault, it won't hurt to have the latest version with better support for larger drives. Hope your issues continue to remain resolved. And I agree with what was said in your thread about removing drives that aren't immediately needed for the array. It's nice to have some spare space, but it's also lifetime hours wasted on the drives that don't see any use. Dale

The re-enable process has now re-started the rebuild of Disk 8 from parity. The drive is still attached to the MB SATA port but once the rebuild is complete I'll likely want to move it back to the shelf with the other 8TB drives. I'm not happy with the new but long SFF-8087 cables I'm using for the connections to the 4 shelves controlled by the 9201-16i. I'll order some new shorter cables which should hopefully reduce potential for UDMA CRC errors. I'll also do some testing with the 9201-16i... I do note that it's running an older firmware (19.0) in IT mode so I may also upgrade it to the latest. Thanks for the help! Dale

So questions... if unRAID started the rebuild and for some reason had the write errors, moving the replacement disk to a new slot/SATA port would still leave the disk disabled? If so, since I've already pre-cleared and verified the replacement, could I not just remove it from the array and format it as XFS under Unassigned Devices? Then re-add it to the array while attached to the MB SATA port and hope it rebuilds? Dale

New diagnostics with drive attached to MB sata port.... animnas-diagnostics-20190711-1720.zip

No change. Tried the slot reserved for 2nd parity drive that's connected to MB SATA, and also a port on another shelf that's connected via a different SFF-8087 cable to the LSI 9201-16i. I'm thinking that I may need to use the unbalance plugin or Krusader to copy the emulated contents to elsewhere on the array. I have enough free space to allow this since adding the new 10TB which started this whole adventure. Or what about using dd on my Ubuntu to mirror the original suspect drive to the new replacement and then re-inserting it? I also still haven't tried a 'Check' even though that says it's a read-only check of all drives. One final option might be to first use dd to mirror the contents of the suspect 8TB to the replacement 8TB. Then do a new configuration and let parity rebuild. Thoughts? Dale

I'm suspecting I may have a controller issue... as mentioned above (in my 3rd post) I'm using a Norcotek enclosure with 20 hot-swap drive bays. The shelf that has 4 slots for 8TB drives shows the 1st 2 fine, but the 3rd slot is where the replacement (and the original suspect disk) are located. It's connected via a mini-SAS SFF-8087 cable to the LSI 9201-16i. I'll try stopping the array and moving the disk to a slot on another shelf or to the slot I've reserved for a 2nd parity drive (when I can afford it). The slot reserved for the 2nd parity drive is connected to a motherboard SATA port so that will tell us if it's a controller/cable issue or still something else.

Here's the diagnostics... I'm looking at syslog.txt and see that it's got a lot of info about the replacement disk but nothing that stands out so far. But I'm a newb at analyzing unRAID diagnostics. Any help appreciated! animnas-diagnostics-20190711-1616.zip

Well, the pre-clear on the new/replacement 8TB finished successfully so I added it to the array in the slot (Disk #8) with the red X, formerly occupied by the suspect 8TB disk. However it's just sitting there showing read errors. Was I supposed to format it with XFS after the pre-clear as that step isn't indicated in the 'Replace a Data Drive' procedure. Step 9 mentions needing to check the 'Yes I'm sure' checkbox when starting the array, but that wasn't offered. Right now the array is started with the replacement in the same slot as the suspect drive, and the suspect (but apparently fine when mounted on my Ubuntu system) drive is completely removed. The only option I see is to do a 'Check' which is apparently only a read-check of all disks. Is this necessary before it will start the rebuild? Right now it's showing that contents are still emulated. Suggestions on how to proceed? Dale

It's a Norcotek RPC-4220 enclosure... like this one: http://www.norcotek.com/product/rpc-4220/ . The new 10TB drive was just put into one of the empty drive trays and then inserted into a free bay on the front of the case. So nope, no new cables - the hot-swap drive trays are accessible from the front. The cabling is one SFF-8087 mini-SAS to 4 SATA breakout for the motherboard SATA ports, and the remaining 4 'shelves' of 4 disks are cabled with SFF-8087 to SFF-8087 cables fed from the LSI 9201-16i HBA. I did swap out one of the SFF-8087 to SFF-8087 cables for one shelf as drives on that shelf of 4 saw more UDMA CRC errors. The new 10TB was installed 2 shelves up and the 8TB that 'failed' is on a shelf with 2 other 8TB drives and has been error-free since the original build. I'm suspecting the 8TB drive is fine, but I'll let the pre-clear finish on the new 8TB before I decide how to proceed. That'll be sometime tomorrow. Dale

OK, I see another post made by @BiGs and responded to by @jonathanm and @itimpi, related to my questions above: Since I had powered down and then restarted after removing the suspect disk, my config now shows the suspect Disk 8 with a red X but the config field is now set to 'unassigned' instead of the name/model/sn of the missing disk. So inadvertently I've created a new config that is valid - I haven't tried starting the array yet though. I'm still letting my new 8TB finish its pre-clear but now I'm wondering if it is still possible to assign the new 8TB to the Disk 8 slot when cleared and have parity rebuild it? Because Disk 8 still has a red X but nothing assigned, I'm assuming that any disk large enough assigned to that slot will still have the data rebuilt from parity? Or am I now better off starting the array with the new config and then copying the apparently accessible data from the suspect disk over the network or via the Unassigned Devices plugin? Or will Disk 8 contents still be emulated and I'll need to use the unbalance plugin or Krusader/MC to copy the data from the emulated disk to free space in the array? Thanks again for any assistance or suggestions! Dale

I've had a glitch with adding a new 10TB drive to unRAID. My unRAID was running well and the last parity check succeeded, so I assumed all was good. The new 10TB drive was pre-cleared and stress tested successfully while still in the USB enclosure. To prep for adding the drive to my system, I disabled Docker and Array Autostart in Settings. I then did a clean power-down. While powered down, I shucked the new 10TB from its USB enclosure, and added it to a free bay in my case. I then powered up and it booted normally with no unusual messages that I saw during the boot. The array was stopped as expected, so I then assigned the new 10TB drive to a free slot in the unRAID webgui. I then started the array and the first thing I did was enable the option to format the newly added disk. While this format started, I then noticed that one of my new 8TB drives which had been in the system for at least 1 month had a red X beside it. Hovering over the drive with the mouse revealed that the drive was disabled and the contents being emulated. I then downloaded the SMART log (attached) and saw only a few reported UDMA CRC errors, which to my prior experience isn't a major concern as they just require a resend of the info to/from the drive and that's handled automatically. I then decided to stop the array after the format finished on the new 10TB. I tried to start the array again but the 8TB drive remained disabled. I then powered down, removed the suspect 8TB drive and attached it to my Ubuntu system where the drive was seen normally and a quick browse of the XFS filesystem shows that things appear to be intact. Regardless, I had a new 8TB that arrived in the mail this morning that I was planning to pre-clear and stress test and keep ready in case of failure. That new 8TB drive is now pre-clearing. My plan is to wait for the pre-clear to finish, then add the new drive to the array as a replacement for the potentially failed disk. In the meantime I've left the array in the STOPPED state (autostart still disabled). This will exclude use for my nightly backups of my other systems and of course my Plex docker. That's fine - I can tolerate missing a nightly backup from each of my other systems and of course, Plex use isn't required either. If I really want to watch something, I can use my Netflix/Amazon Prime/other streaming options. I assume once I add the new 8TB as a replacement, the next array start will trigger the rebuild. I have a couple of questions: 1. My suspected failed drive appears good, but is there anything I should do with it on my Ubuntu system to try and verify it further? 2. Since the new 10TB drive is already added, formatted and part of the array, would it be quicker to copy the data from the suspect drive on my Ubuntu system to the array? This would be over the network, but I suppose I could also add the disk as an unassigned volume and copy to the array. 3. If I don't do the copy as described in item 2, how long (estimated; system specs in my signature) would rebuild from parity take for the new 8TB? Once the 8TB drive is finished its pre-clear/stress test, I could add it as a new drive instead of a replacement, but this means I'd have to redo the configuration without the suspect disk and then let parity rebuild with the two new disks in place (1 x 10TB, 1 x 8TB). The parity rebuild would take about 24 hours based on the last runs, and then I'd have 18TB (raw) of empty storage. This is plenty for me to copy over the data (about 6.2TB) from the suspect 8TB drive. Once copied over, I can then redo the pre-clear/stress test on the suspect drive. If it passes, I can keep it as a spare for future failures. I'm fairly confident that my parity is fine, but rather than try to rebuild and/or re-use the suspect disk in a new config, I'm still leaning towards the copy method vs a parity rebuild using the new 8TB drive. Thoughts? I didn't examine the attached SMART log too closely. I removed the references to the serial number, not that it's really useful to anyone. Any suggestions/recommendations appreciated! Dale ST8000DM004-2CX188-20190710-1256.txt

But as I and others have mentioned, even using a SSD as an unassigned device experiences the same issue. I do believe it's the Plex client(s) at fault more than Mover.

I've experienced this as well. I've tried using a SSD as an unassigned device for Plex, and also on my SSD cache drive within unRAID. I do believe that part of the problem is the Plex clients themselves and will be reporting this to the Plex support forums shortly. I've found that even with Mover tuning priority set to Very Low and disk I/O priority set to Idle, higher bitrate media still has playback issues and will often get stuck 'buffering'. I believe part of the issue is that the Plex client on most smart TVs have limited RAM to work with. The Plex for LG WebOS client is a prime example of a Plex client that fails often. And surprisingly, even the Plex for Apple TV client often buffers for high-bitrate titles. It makes less sense for the Apple TV 4K as I have a 64GB model and it has plenty of unused storage according to the list of apps under Settings. However the 3rd party Infuse Pro client on my ATV4K rarely experiences a hiccup. My HTPC using either VLC or Zoomplayer also rarely experience the buffering that seems to plague the official Plex clients. I've even played around with CPU pinning with Plex getting 3 out 4 of my hyper-threaded cores (6 threads total) and all other unRAID processes with the 1st core (2 threads). I am planning a hardware upgrade eventually to a system with more CPU cores, but in reality it still seems that the Mover process and disk I/O priority take too many resources to allow for optimal media playback. At least I have work-arounds by using Infuse or my HTPC.

Why not use any number of email clients that support downloading of online email providers? On Macs, I use the default Mail app to download all the mail in my online email accounts. And from there, a Timemachine backup to unRAID. I haven't used the Windows built-in mail app, nor have I used Thunderbird (multi-platform) in quite a while, but I'm sure those would work as well. I haven't used it, but in the unRAID Apps section there's a Docker container called Kerio Connect that may also provide the functionality you're looking for. Not sure if you need to purchase a license to use the Docker container, but here's the Kerio Connect info from the developer: https://www.gfi.com/products-and-solutions/email-and-messaging-solutions/kerio-connect

Agreed. Especially for users like myself that are new to unRAID. I've learned lots since starting my trial 28 days ago, and migrated to my Pro license 2 days ago. I'm now aware that I need to check if an unRAID NVidia build exists BEFORE upgrading. A simple trap to see if the user is running the NVidia build should be possible, and then advising users to check 1st BEFORE upgrading.

For those of us running unRAID with the NVidia drivers, and I expect that's a fair number of users, perhaps update checks could verify that a build with the NVidia drivers is available before unRAID tells us that a new version is available? I just had to roll back as my Plex docker and my VM rely on the GPU passthrough and the NVidia drivers/tools. My Plex docker wouldn't even start with 6.7.2 because of the missing NVidia components. EDIT: it's actually worse than that... my install is hosed. The console at the server reports a 'segmentation fault'. I've done a shutdown after logging in at the console and have backed up my /config folder. About to use the unRAID creation tool to re-do the flash drive with a clean 6.7.1 and then restore some of my config including my Pro.key file. I noticed that network config was definitely corrupt so I'll replace that with one from last night's backup of 6.7.1. Hopefully nothing much else is wrong. EDIT 2: And now it's booting and the webgui is available, but it's started another parity check. 24 hrs to go.... EDIT 3: Had to stop the parity check as I had to reboot after re-installing the unRAID NVidia build for 6.7.1. It didn't restart the parity check after rebooting, but I suspect I'll go ahead and start a manual parity check to ensure all is OK. At least I'm back to what appears to be a functional 6.7.1 with NVidia extensions available and my Plex docker started fine. The only files I replaced from my previous flash config were network.cfg and ident.cfg.

Disk write caching is a feature that improves system performance by using RAM (volatile memory) to collect write commands sent to data storage devices and cache them until the slower storage device (ex: hard disk) can be written to later. As RAM is much faster than disk I/O, this allows data to move more quickly, at least until you exhaust your available RAM. It has some minor risk if your system isn't stable or protected from power failure with a UPS, but more when using move instead of copy. By default it appears that 6.7.x of unRAID set WCE to 1, so that all disks on the array are write cache enabled. Think of it as a RAM-disk buffer or storage device that's faster than even SSDs.

@TyantA As you want your /transcode path to be on a UD mounted disk, you just need to edit the default path for /transcode that's already in the Plex docker container. The attached image shows my /transcode mapped to my 1TB cache SSD, but in your case you want to change the Host Path to /mnt/disk/{yourUD-disk} and change the Access Mode to RW/Slave. It needs to be RW for /transcode, not RO.

As @Squid said, the UD mounted drive can be mapped directly, but you can also enable sharing for that drive by enabling the Share switch for that drive in the UD section of the Main tab. But the better method is to map the UD drive to a mountpoint in your Plex container. The image attached shows one of my 10TB drives mapped to the mountpoint /mnt/MoviesE. The Name field isn't crucial, but the Container Path Host Path and Access Mode should be set. Choose the 'Add another Path, Variable, etc' option to create this mountpoint path in your Plex Docker container. When you click in the Host Path field, you navigate to /mnt/disks and choose the UD mounted disk that you want available in the Plex container. Save your new path and then Apply it to the docker container and you'll have access to it in Plex. I chose RW/Slave for the Access Mode as I want to be able to delete media from my UD mounted drives from within the Plex interface.

Thanks! I'm OK with the slower speeds, as those older disks are mostly full now anyhow. And so far, their read speeds for use with Plex has been fine (all movies and TV, mostly remux vs re-encode). And eventually as I save up enough cash, I'll replace the older drives with newer drives of higher capacity.