RasterEyes

I shut down through the web interface, but that wasn't enough. I had to force power-down my array. When I started back up I took the drive I was trying to rebuild to out of the array first, figuring I'd start up without it, the former contents emulated. The array started up very fast this time. I think we have to consider that the real reason my rebuild was going so slowly before was some rogue process eating up a lot of CPU time and locking up resources, not the oh-so-easy-to-blame drive with a funny-sounding brand name. Sadly I didn't realize that the emulated contents of drive 5 would only be available via a share, and not /mnt/disk5. That makes copying what would have been on drive 5 to the new Crucial drive quite a challenge unless there's some shortcut I don't know about.

I got tired of waiting so I stopped my array and took the drive I was trying to rebuild to out of the array. I figured I'd just restart with drive missing and copy from the emulated drive to the external drive. Well, a few minutes later and I'm still at "Starting...". Under /mnt none of the array drives have appeared yet. One CPU of four keeps frequently pegging at 100% although I can't figure out what the array is up to with all of that CPU effort. Some more diags for ya. terminus-diagnostics-20250303-1603.zip

I paused the rebuild, but from the read/write speeds I see, it's still actually going on even though it's supposed to be paused. I have the Crucial drive plugged into an external USB enclosure, and I'm trying to fill it from the emulated content of drive 5 like this: cp -R /mnt/disk5/* /mnt/disks/TerminusBU2 But the same slow read speeds are still happening, the slow write speed to the drive that was being rebuilt continues, and content is being copied to the external drive just as slowly. New diags attached just in case that matters. terminus-diagnostics-20250303-1541.zip

I think the worse thing about these drives is not the actual performance, but that when something doesn't work well that no one will consider anything else other than it's a cheap drive. I have a new Crucial brand drive I be willing to give a try, but I'm guessing that if it's just a little bit smaller than the drive it would be replacing (4,000,787,030,016 bytes vs 4,096,805,658,624) that Unraid won't consider using it for rebuilding a drive that had been a even a little bigger. I'd probably have to copy files from an emulated drive 5 to the slightly-smaller drive as an unassigned drive, then swap that drive into position in the array using a new config that keeps all drives as-is, and then rebuild my parity drives instead.

My array currently consists of 2 8TB HHDs for parity and 12 4.1 TB SSDs. Yes, I know support for SSDs is only considered "experimental". But before anyone harasses me about that, I would dearly hope that they would consider there's nothing about using SSDs alone that should explain performance THIS BAD. I've definitely seen these drives show much faster read and write speeds. Functioning in the array the read speeds are always high, as high my 2.5 GB wired network allows at around 200-250 MB/s. Write speeds are slower in-array, but still typically in the 50-100 MB/s range. One of the same SSDs used as an "unassigned disk" allows writing at up to 150-250 MB/s. I recently took all of the disks out of the array configuration, gave them a good TRIM which they can't get while in-array, then reconstructed the array and rebuilt parity. That was a bit on the slow side (it was my first experiment with HDDs for parity along with SSDs for data), but still took less than a day. I've tried some recommended disk settings like this: ...but to no avail. It seems to be that there's got to be some crazy kind of bottleneck here, but what that might be I don't know. Blaming individual drive performance doesn't seem to cut it as an explanation for what's happening. terminus-diagnostics-20250302-1523.zip

Hence the "(yes, I'm aware that support for all-SSD is itself considered only experimental)" in my OP. 😁

Except in this case the 4TB data drives are SSDs, only the parity drives are now HDDs. (Using HDDs for parity, instead of those also being SSDs, is what I've been experimenting with.)

It looks like the answer is yes, the last 4TB will go a lot faster than the first 4. Oddly enough, the speed picked up quite a bit a little before the very end of the data drives was reached, as if the last dregs of real data could somehow be read and processed faster. When progress hit 51.2% all reading from the data drives stopped. The parity drives are now being zeroed out at a fairly steady 180-190 MB/sec. Estimated remaining time is a little over five and a half hours -- I think that's a fairly good, reliable estimate because the value isn't wildly bouncing around as it often does. The first 4TB of parity took over one full day to be completed. That perhaps could be really be more like a little under 24 hours if you subtract a 2-3 hours when I paused the rebuild (I'm not sure if the paused time gets subtracted from the elapsed time or not). But either way, yes, when there's no real work to do reading and computer parity, it looks like the parity building process can run at steady, fast pace, about as quickly as the drives will accept data.

I wasn't expecting it to give up. I'm wondering if the rebuild process will do the last half a lot faster, since there's nothing to read from other drives and nothing to compute. As the rebuild is at 49% right now, I should know fairly soon. The data drives are slightly larger than half the size of the parity drives, so it will be just a little past 50% when there's nothing left to read from the data drives anymore.

...and I'm wondering what will happen once the 4TB mark is reached. I'll have an answer later tonight or sometime tomorrow from real-life experience, so that's why I'm bringing this question up in the lounge. It's just idle conversational curiosity while I wait. Once 4TB is reach, all that's really left to do is "zero out" (I don't know if zero bits are actually used) the last half of each parity drive, right? Should I expect the parity rebuilding to race through this process of dealing with the last 4TB? Or will Unraid poke along as if it were still doing all of the work needed to build meaningful parity? What I'm up to at the moment is the experiment of taking what had been an all-SSD array (yes, I'm aware that support for all-SSD is itself considered only experimental) and replacing the parity drives with HDDs, figuring they have to deal with the highest level of write activity. Also, spin up delays and head seeking delays when writing data are of little importance to me. Fast reading without worry about mechanical delays (especially head contention) for video streaming is my primary concern. Since the HDDs are so much cheaper than SSDs I figured I might as well make my parity drives 8TB now so that I can easily add larger SSDs, or replace existing 4TB drives with larger SSDs, in the future. I'm also curious, if for some reason my array crashed while building parity, but did so somewhere beyond the 4TB point, if Unraid would be smart enough to know it had all of the parity it needs to emulate and rebuild any of my 4TB data drives, or if Unraid won't consider the parity drives to be at all valid and useful until the very last byte of their 8TB capacity has been filled.

I'm not using Plex, Docker, any VMs or anything else. I'm using Unraid as a simple file server, nothing more. I can only verify that the next time this situation arises, but it seems highly unlikely given how my Unraid array is used. Nope, not installed.

What you're saying might help a performance problem that I haven't noticed and wasn't asking about. I'll worry about that another time. I have no Docker apps or VMs anyway. Even if there are files I should keep on the cache drive, my concern is specifically about files that should NOT hang around on the cache drive longer than necessary, and about the odd and inconsistent reporting of space consumed by files that are no longer be on the cache drive anymore.

For the most part the cache works fine for me. I'm not having a major problem with it filling up and not emptying, not being used at all, etc. But every now and then a few files seems to get stuck in the cache and the mover won't move them out. I click the Move button and nothing happens. Today that happened to me with five MKV files totaling around 113 GB stuck in the cache. These files were not open or in use by any process. Instead of manually moving these files out of the cache to some /mnt/diskXX disk, I tried something new. Since copies of the files were readily available elsewhere (in my main video storage array -- this was a backup array I was trying to clean up) I simply deleted the files from the share where they resided (that is, I accessed the files via the share path, not directly by the cache drive path). This emptied the cache drive, as expected, but under Main/Pool Devices, the cache drive showed up still having 113 GB in use. Refreshing my browser didn't fix that, nor waiting a few minutes and refreshing again. I next rsynced the deleted files back into place. Displayed cache usage went up to 226 GB, as if two copies of everything were now there. Clicking the "Move" button finally worked,. These fresh file copies were moved off the cache drive into array storage. The Shares tab no longer showed any warnings about share contents not being fully protected. But the cache drive was now showing 113 GB in use again. I'm guessing this is just some weird bookkeeping error. Taking the array offline and starting it up again cleared that up, with the cache drive now showing only 4.3 MB of usage (essentially empty). trantor-diagnostics-20250108-1806.zip

After rebooting, why do I have to remove and re-add NFS shares to mount them? I have a pair of of shares that look like this: MOUNT is always grayed out and inoperable after a reboot. I have to remove one of the two shares, then re-add that share, before either share can be mounted. Beyond that, even after mounting one of the shares, they seem to time out after a while, so I have to unmount and remount (at least not remove and re-add) the shares to use them. These shares are provided by a separate Unraid server. Both arrays are running Unraid 6.12.10.

After a reboot, I have a pair of of shares that look like this: MOUNT is always grayed out and inoperable. I have to remove one of the two shares, then re-add that share, before either share can be mounted. Beyond that, even after mounting one of the shares, they seem to time out after a while, so I have to unmount and remount (at least not remove and re-add) the shares to use them. These shares are provided by a separate Unraid server. Both are running Unraid 6.12.10.

Actually, I just found a plugin called "Unbalanced". Looks pretty straight-forward, with a nice GUI for performing the task.

I have a couple of shares for Time Machine backups, and it occurs to me it might be convenient if all parts of each backup were all on a single drive.

Perhaps I'm just using the wrong terminology. The array was writing (presumably zeros) two all three new drives, whether that's called preclearing or not. It's all done now, everything is fine. I was just curious about the sudden drop in speed.

It wasn't my decision. I plugged in the new drives, started up, and that process started automatically. As for "it's normal for writes to start much faster", the cache isn't 1 TB in size. I could see cache being an issue for a few MB or GB, but it sure as hell wasn't cache that made the initial writes run at 450 MB/sec for the first whole TB.

This isn't a crucial issue for me, just a matter of curiosity. Today I added three new 4 TB SSDs to my array. They were preclearing at a nice clip, almost 450 MB/s, up until about 1 TB was cleared. At that pace the whole job was going to be done in less than three hours. Then things slowed down quite a bit, however, generally clearing at a speed from 50-100 MB/s. I was about to say the current speed was 85, but it dropped to less than 30 sometime after I started typing this post, and has since climbed back to around 65. I'm about 42% done now, with the wildly-fluctuating completion time estimated at around 11 hours from now. The array isn't busy doing anything else. It's idle apart from the preclear operation. I did just notice that my version of Unassigned Devices Preclear is 2023.05.20, with an update to 2024.04.23a available. Maybe that has something to do with the big speed drop? I have no intention of updating that plugin while preclear is already running, however. Could this just be the normal behavior of the SSD, maybe a response to the SSD warming up from running so steadily for a while?

I'm a little confused about what you're referring to as "my way", so I'll just try to be more clear about what worked for me: I created a file, /boot/config/my_profile (could be /boot/config/anything_you_want) which contained the extra stuff I wanted to append to /etc/profile -- just the extra stuff, not the full contents of that file. The only extras I wanted to add for myself were aliases, but you can add any other set-up commands you like. I then edited the existing /boot/config/go file to add this extra line: cat /boot/config/my_profile >> /etc/profile This causes my desired changes to be appended to /etc/profile at start-up.

Rebooting my array did the trick. I'm not sure if this might be why my array was clinging to these drives until a reboot, but just before checking out all of the various drives, I'd gone into by computer's BIOS and enabled hot-swapping for all on-board SATA ports so I could add and remove drives without a power cycle each time.

You mean for the third time today? 😁

I have three historical devices which accumulated while looking at some old drives, figuring out which one I'd previously used as a parity drive. Now that I found the drive I was looking for (to recycle it as a cache drive), I have no need to keep info about the other drives. But each drive has a black X, not a red X next to it. Clicking the black X does nothing (the mouse cursor doesn't change passing over the X either). How can I get rid of these? The array being stopped or started doesn't make a difference for this. trantor-diagnostics-20231018-1754.zip

Ah-hah! RECONSTRUCT WRITE!!! That's the secret to getting great performance out of my SSDs. By switching to reconstruct write for generating parity instead of auto (which must have always auto-selected read/modify/write), things are going MUCH faster. Along with upgrading my network to 2.5G, I'm getting nearly 250MB/s, ten times the speed I started with. No need to dis the cheap Fikwot drives either. Use the right settings, and they work great.