Multiple isues at once: new cache drive in RAID1, Docker not starting, disk emulation

[Karyudo]

Last night, I accidentally filled up my cache SSD (500GB). Today, I spun down the array and added an additional cache HDD (4TB).

 

On startup, I also notice Disk 1 is being emulated. I start the array in maintenance mode, run a check, and don't see any SMART errors, so unassign and re-assign the drive to replace itself. Rebuild starts.

 

Then I see that Docker isn't running. And the cache still seems to be 500GB. Do some searching: must be the default RAID1. But I can't see where to change this to RAID0?

 

I want to do three things:

1. Set the cache up so it's 4.5 TB across the SDD and HDD, not worrying about redundancy (I back up the appdata at least weekly).

2. Get Docker running ASAP.

3. Rebuild the previously-emulated drive (I don't mind re-starting this, since I've got two parity drives).

 

How should I proceed?

shinagawa-diagnostics-20180902-1510.zip

Edited September 2, 2018 by Karyudo
Added diagnostics zip file

[JonathanM]

2 hours ago, Karyudo said:

How should I proceed?

Download the diagnostics zip file and attach it to your post.

[Karyudo]

Done! (It's above, with my first post.)

 

I also now see another drive is reporting problems. This has got to be cable or controller related... right? (Even if that's the case, I'm not sure what I can cancel/shut down without making anything worse.)

Edited September 2, 2018 by Karyudo

[Karyudo]

Even to my untrained eye, this does NOT look right. 600,000,000+ errors?!

[trurl]

Since you have dual parity you should be able to rebuild disk1 and disk8 after you get your hardware problems taken care of. Check ALL disk connections, SATA and power, both ends.

 

You might just disable docker service until you get your array stable and deal with cache questions later. I am wondering why you think you need so much cache space that you are willing to mix SSD and HDD into a pool. I would think the 500GB SSD would be plenty if you have carefully considered how you are using cache and have things configured correctly. Looks like you have a couple of user shares that you created set to cache-prefer. Are you sure you know what that setting does? Lots of people confused about that setting wind up filling up their cache.

[Karyudo]

Thanks for your reply.

 

My first issue is, should I cancel the current parity operation (it seems to be stalled at 13-something hours remaining, with errors climbing)? Seems like the answer is "yes," but I've seen other threads where people have done what seemed logical, only to be told, "you should have done X before cancelling."

 

I know (now) that I don't have the cache set up correctly. I don't know why not: some shares were set to not use the cache, but files still ended up there regardless. I don't know how to fix that. I also know that a 500GB cache isn't big enough for the 1TB torrent I want to nab. Most of the other drives I have are shingled, so I want to keep torrent traffic on SSDs and "regular" HDDs. That's what's recommended, isn't it?

[trurl]

1 hour ago, Karyudo said:

I don't know why not: some shares were set to not use the cache, but files still ended up there regardless. I don't know how to fix that.

Any share that you set to cache-no will not write new files to cache. If the share already had files on cache they will stay there.

 

Any share you have set to cache-prefer will write new files to cache if there is space, and it will also move files from array to cache. This is what often trips people up and causes them to fill up their cache.

 

Anyway, let's deal with your cache issues later and get your array stable. Go to Settings - Docker and disable the docker service. Do the same for Settings- VM Manager if you are using it. This will take dockers and VMs out of the equation until we can get your cache straightened out.

 

Stop the current rebuild. It isn't going to be reliable anyway since you are getting so many read errors on another disk. Then shutdown and

1 hour ago, trurl said:

Check ALL disk connections, SATA and power, both ends.

*edit to add*

Also, if you have a disk controller card, reseat it.

 

Then, reboot and post another screenshot and new diagnostics.

[Karyudo]

On it. Thanks!

[trurl]

3 minutes ago, trurl said:

Also, if you have a disk controller card, reseat it.

Just in case you missed my additional edit above.

[Karyudo]

I did miss your edit; thanks for the additional post. I was already going to reseat the one PCI-E controller card I'm using, anyway, since I can feel it's running a bit hot. I'm also going to check which drives are on the card. If it's Disks 1, 7, and 8, then I think I know what the problem might be....

[Karyudo]

Reseated card; checked connections. Everything looks OK (but of course it looked OK before, too...).

 

BTW, Disks 7 and 8 were on the PCI-E controller card; Disk 1 was on a mobo SATA port.

shinagawa-diagnostics-20180902-2132.zip

[John_M]

Ignore this for now and work with trurl to get your main array stable. When you're ready, see this thread for an answer to why the second disk in your cache pool isn't being used:

 

 

and this post to convert it to a different RAID profile. You mention RAID 0 but that would only give you 1 TB of usable storage and waste 3.5 TB. The profile you really want is "single" which simply concatenates the two devices:

 

 

 

[Karyudo]

Consider it ignored (for now...)! I did previously read JB's FAQ entry on adding a cache drive. That's how I sort of know about RAID1 and RAID0. But "single" sounds like what I really want. Or maybe I want a non-array, non-shingled drive for torrents? "Yarr... I don't know what I'm doing."

 

Now that my hand has been forced anyway (with a drive rebuild or two looming), I think perhaps I'll look into adding a larger SSD rather than an HDD that's way bigger than I need, or maybe just redundancy at the size I've already got (if the torrent thing can be solved a better way).

[John_M]

What I would do is use the SSD for cache, as it was originally, and mount the HDD outside the array using the Unassigned Devices plugin. That way you have fast storage in the cache and a separate large capacity temporary store for your torrents. When you've accumulated all the pieces from the torrents you can transfer the files to the array.

[Karyudo]

That's where I was leaning, too—although I didn't reference "Unassigned Devices" explicitly. I've also got to get any files that should be only on the array off the cache drive. But that's a problem to be tackled once the array is fixed.

[Karyudo]

Sooo... what's the recommended next step? ("Are you there, @trurl? It's me, Margaret.")

 

Obviously a rebuild of some drives is coming, but I'm intimidated by what I don't know may be lurking in the diagnostic zip, and I don't understand the nuance of Disk 1 being emulated and Disk 8 being disabled and emulated.

 

I'm guessing Disk 1 can be rebuilt directly, because unRAID has no technical problems with the drive, just the data on it (orange triangle). I'm further guessing that Disk 8 will need to be checked somehow, and then returned to the array, and then rebuilt, because unRAID has identified a possible issue with the drive and the data (red 'x'). Is the recipe "rebuild D1, fix/replace D8, rebuild D8," or "fix/replace D8, rebuild D1 and D8 simultaneously"—or something else?

 

I'm also guessing there is no advantage to trying to recover Disk 8 first (it has only 250 GB of user data on it), because a rebuild is a volume-level reconstruction from parity of "blank space" along with user data?

[trurl]

Disk1 is invalid because the rebuild wasn't completed. Disk8 is disabled because a rebuild hasn't started on it yet. SMART for both disks looks OK. Since you have dual parity, both disks contents are emulated and they can both be rebuilt. Starting the array now should begin rebuild with the current disk assignments.

 

If you have spare disks you could also use those for rebuild and keep the originals in reserve in case there is a problem.

 

Do you have backups of any important and irreplaceable files?

[Karyudo]

Thanks for guiding me through this, trurl.

 

RE: important and irreplaceable

Umm... sort of. I've got my photos in the cloud somewhere, and most of my other personal files on a (much older, inferior) WHS box. But let's say I wanted to make backups now. Am I able to, before the rebuild starts? Or is it too late?

RE: Spare disks

I've got one 8TB Seagate SMT drive that I haven't shucked yet (but will now), which maybe I should use for Disk 1? Unless the previous rebuild would have already nuked the existing data on the iffy drive (and thus render it not worth keeping as a backup)?

I've got another 8TB Seagate SMT drive that previously had a red 'x', but that I replaced without checking whether it was a real problem or not. How could I check it now?

 

RE: Disk status

Do I have to do anything to Disk 8 before starting the array? Or will that red 'x' go away and the rebuild will start automatically?

Edited September 3, 2018 by Karyudo
Add detail on spare Disk 1

[trurl]

4 minutes ago, Karyudo said:

But let's say I wanted to make backups now. Am I able to, before the rebuild starts? Or is it too late?

The data from the emulated disks can be read from the parity calculation, so it should be possible to copy their data. I recommend not writing to the array at this point though, so you should copy anything you want to another system over the network, or else to an Unassigned Device.

 

7 minutes ago, Karyudo said:

the previous rebuild would have already nuked the existing data on the iffy drive (and thus render it not worth keeping as a backup)?

Possibly the failed rebuild would have made that original bad already. Can't know without trying to read it.

 

9 minutes ago, Karyudo said:

I've got another 8TB Seagate SMT drive that previously had a red 'x', but that I replaced without checking whether it was a real problem or not. How could I check it now?

If you plug it in you should be able to get its SMART from Unassigned Devices.

 

10 minutes ago, Karyudo said:

Do I have to do anything to Disk 8 before starting the array? Or will that red 'x' go away and the rebuild will start automatically?

Starting array will begin both rebuilds. Disk8 will go from disabled (red x) to invalid (yellow triangle like disk1) until rebuild is complete. If rebuild completes both disk will become green.

[Karyudo]

OK, here's what I'm going to do, pretty much in order:

 

1. Shuck new 8TB drive, and plan to use it as new Disk 1. (As an aside, where did the Preclear plugin go? When I added a drive recently, I didn't see any way to pre-clear it like I remember being able to do in the past. I'm assuming it's no longer recommended? Or what did I miss?)

 

2. Plug in older 8TB drive, to check SMART info in Unassigned Devices. (I assume this will be possible without starting the array?)

 

3. Mount a 4 TB drive in Unassigned Devices, to copy important data from emulated array. (Can I remove the drive I erroneously added to the cache pool somehow?)

 

4. Start array. Rebuild should start. While rebuilding, try to copy important data from emulated array to Unassigned Device data drive.

 

That about covers it, right?

[trurl]

Unassign both disks then start the array so you can copy the emulated data before beginning the rebuild.

 

I will be away for a few hours now.

[trurl]

OK, back sooner than I thought.

 

I wouldn't worry about preclear, especially if you are keeping the originals. Rebuild followed by a non-correcting parity check is a pretty good test of a new drive. It won't necessarily weed out infant mortality like multiple days of multiple preclear cycles, but nothing will ever guarantee a disk won't fail tomorrow anyway.

 

If you unassign disk1 and disk8, then you should be able to start the array and read from the emulated disks before rebuilding. Then you can copy the emulated data. Then assign the new disks and start to rebuild.

 

See here for instructions on removing cache2:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/#comment-480418

 

[Karyudo]

I've got the array up, without Disk 1 or 8, but with an Unassigned Devices drive. (That makes 13 devices, apparently, so I had to spring for a Pro license upgrade, but that's another story. Money well spent, really.)

 

I now have no idea how to "see" the Unassigned Device, or copy to it. I've still got Docker turned off, so Krusader is out....

 

I spoke too soon! After posting, I hit F5 again in my Explorer window, and sure enough now the UD drive appears.

Edited September 3, 2018 by Karyudo
Unassigned Device drive appeared!

[trurl]

1 hour ago, Karyudo said:

I spoke too soon! After posting, I hit F5 again in my Explorer window, and sure enough now the UD drive appears.

Are you talking about Internet Explorer, or Windows Explorer?

 

If you use Windows Explorer to copy to the Unassigned Device it is going to be getting the data across the network to Windows, then back across the network to the Unassigned Device. It would be better just to copy it to Windows in that case and skip the round trip.

 

I always use mc (Midnight Commander) to work directly with the disks on the server. It is a builtin textGUI. Have you ever used the command line with unRAID?

 

 

[Karyudo]

I was talking Windows Explorer (like everyone else, I haven't used IE for years...).

 

I guess I'd better figure out how to use Midnight Commander, because you're right: it's stupid to drag the files to a Windows machine, only to put them back on an Unassigned Device under unRAID. Is there a good primer in a thread someplace here? 

 

I used to be OK at MS-DOS, but I'm only just recently getting to know my way around Linux (Mint), and haven't done any CL stuff under unRAID.