Short version of the problem:

 

I'm running unRAID6 and everything on disk 6 somehow disappeared in Window 10 file explorer. I can see the folders and files through unRAID web interface or list them through telnet,  so all the contents are definitely there. Everything else about the RAID is running perfectly fine.

 

I tried New Permissions tool, or chmod command through telnet and none works.

 

 

Long version of the problem:

 

I'm upgrading the server with 8TB Seagate Archive drives. I already replaced parity and disk1 and disk 2. I want to consolidate the disks so I started copying disk 6 to disk 2. I used the cp command in telnet as recommended by the Lime-tech wiki, and all was going fine for about 15-20 mins. Then all of the sudden the copy slowed from 60MB/s to a few KB/s and kept at this snail speed forever.

 

I hit ctrl-C in telnet window to break up the copy process. The server behaves kind of weird for being very slow on everything. Eventually after several rounds of shut-down /reboot everything looks fine, except that on disk 6 I can only see those couple folders that were already copied to disk 2. Everything else somehow disappeared.

 

I googled and tried command chmod * 776 and that didn't help. I run New Permissons too and that didn't help either. Somehow in the process all the files on disk 6 disappeared, including the few copied ones that were previously visible.

 

I have two user accounts, root and test. Only test has a password and it's the account I used to log into unRAID server in window 10 file explorer. I login to telnet by root account.

 

Again, all the files are on still disk6, as I can see them either on unRAID web interface or through telnet. How to make them visible in windows?

 

I knew very little about unix command. Any helps will be greatly appreciated! 

 

Firstly, post your diagnostics (tools - diagnostics)

 

Secondly, install Fix Common Problems and run a scan.  And see what it says if anything.

... except that on disk 6 I can only see those couple folders that were already copied to disk 2. Everything else somehow disappeared.

I predict FCP and diagnostics will show he has a user share named disk6, with its files on disk2.

... except that on disk 6 I can only see those couple folders that were already copied to disk 2. Everything else somehow disappeared.

I predict FCP and diagnostics will show he has a user share named disk6, with its files on disk2.

Same reason I asked for both 

... except that on disk 6 I can only see those couple folders that were already copied to disk 2. Everything else somehow disappeared.

I predict FCP and diagnostics will show he has a user share named disk6, with its files on disk2.

 

Man you and Squid are the genius! Yes just looked at disk2 and there is a folder "disk6", and after deleting it the real disk6 pops up with all the folders there!

 

Here is the how I forgot to delete "disk6" folder on disk2: I tried to delete it after I broke the copying process but the server response was terribly slow. I began the shutdown/reboot whatever then I just forgot the folder.

 

By the way, anyone else having problems using cp command to copy files to Seagate archive drive? On both 8TB disk1 and disk2 drives I tried cp command first and I encountered exactly same problem: copying was fast and smooth at 60MB/s for a while then that came to a abrupt stall. I wonder it has sth to do with way archive drive handles writing. Somehow all was copied to its 32GB small file cache first and once the cache gets filled up its writing speed drops to near zero?

 

 

By the way, anyone else having problems using cp command to copy files to Seagate archive drive? On both 8TB disk1 and disk2 drives I tried cp command first and I encountered exactly same problem: copying was fast and smooth at 60MB/s for a while then that came to a abrupt stall. I wonder it has sth to do with way archive drive handles writing. Somehow all was copied to its 32GB small file cache first and once the cache gets filled up its writing speed drops to near zero?

This thread http://lime-technology.com/forum/index.php?topic=39526.0 is all about shingled drives

Thanks! I read that one before I decided to pull the trigger on 8TB drive. So far installed three of them and everything has been fine except for the cp copying problem. But I don't know whether same thing may happen when I use cp command to copy to a regular drive.

 

 

...  Yes just looked at disk2 and there is a folder "disk6", and after deleting it the real disk6 pops up with all the folders there!

 

Be sure you think about (and understand) how you managed to create a "disk6" share on a different disk.  It's NEVER a good idea to use a share name that matches a built-in name (e.g. "diskx", "flash", etc.)

 

 

... copying was fast and smooth at 60MB/s for a while then that came to a abrupt stall. I wonder it has sth to do with way archive drive handles writing.

 

As I outlined in the thread on the shingled drives (referenced by Squid above), if you do a lot of small copies that result in the persistent cache being full, the performance will indeed drop a standstill ... it will seem as if the drive has completely stalled (it should eventually pick back up, but depending on how the OS interprets the stall it could abort the operation).    These drives work very well for use cases where the persistent cache doesn't "hit the wall" -- but definitely are not a good choice if that's not the case.

 

The disk6 folder was automatically made by cp command. Next time I'll do sth different.

 

As to stalled copying process, I was copying big video files in 10GB size range, so normally there is absolutely noway for those files to fill the persistent cache. I just wonder whether cp copy does sth different that makes the cache full even when copying big files?

 

So far those 8TB archive drives have been working great in normal operations. On parity syncing or disk rebuilding I got 60MB/s to 100MB/s speed. Love those drives so far. Reading and writing through windows are fast too.

You should also consider turning off disk shares. If you had not expected to see a disk6 share from Windows it would have been obvious that something had gone wrong. More importantly, mixing disk shares and user shares when moving or copying files can lead to data loss. Search for User Share Copy Bug.

 

I don't use disk shares. You can get pretty good control of what gets stored where with proper use of user share settings. And if you have other users it is especially important they not be given an opportunity to mix disk and user shares.

... As to stalled copying process, I was copying big video files in 10GB size range, so normally there is absolutely noway for those files to fill the persistent cache. I just wonder whether cp copy does sth different that makes the cache full even when copying big files?

 

Normally the Archive drives will recognize copies that are larger than the shingled tracks and simply copy those files directly to a set of shingled tracks without any involvement of the persistent cache -- so very large files should copy just fine.    Not sure what cp might be doing that would cause this mitigation to not work as well as it can ... but that may indeed be what's happening.

 

Were you by chance copying so much data that the "residual" small bits at the end of these large files might have resulting in a full persistent cache because it never had a chance to empty the cache?

 

Running two instances of cp simultaneously would break the conditions needed for directly writing to the shingled bands. In that case it would only take two of those ~10 GB files to fill the persistent cache.

Running two instances of cp simultaneously would break the conditions needed for directly writing to the shingled bands. In that case it would only take two of those ~10 GB files to fill the persistent cache.

 

That's true => the same as copying two different files to the same drive on any PC at the same time -- the drive heads will "thrash" between the two files and the net result is a total time that's much longer than if you just did one file at a time.  If that's what's happening here, then that absolutely explains why the persistent cache is "hitting the wall".

 

Just to clarify each time I was running a single cp command. There were no multiple copy processes running concurrently.

 

In each case the copy stalled after copying only a dozen files, most of them are video files in GBs.

 

From this discussion it seems possible that for some reason the cp command is not compatible with the the drive? Again I have no problems copying TB s of data to 8TB drives through Windows explorer.

From this discussion it seems possible that for some reason the cp command is not compatible with the the drive?

 

No, I don't think that is true. Any chance there was something else writing to the disk at the same time? The Mover, CA Appdata backup, some downloading utility or other Docker are all possibilities.

 

I wonder if the Dynamix File Integrity plugin could have this effect if it started writing checksums for files previously written to the disk. I'm not sure how exactly extended attributes are stored, but clearly they involve writes and if they are concurrent with your cp operation that would surely involve the persistent cache. What are your thoughts on this, Gary?

 

I'm pretty sure I don't have the Dynamix File Integrity plugin. I attached the diagnosis file here in case it is helpful.

 

As far as I know nothing else was writing into 8TB disks when I was doing cp copying through telnet. I replaced parity with 8TB first. After rebuilding the parity disk I replaced disk1 with 8TB. Few hours after RAID rebuilding (took about 2 days) I did the cp copying disk10 to disk2 , and got the problem as I described. I eventually used a windows 10 PC to copy disk10, disk9 and disk8 to disk1 and removed these three disks. The copy speed averaged around 25MB/s and everything went fine.

 

I then replaced disk2 with 8TB drive, and I waited about a day after RAID finished rebuilding. I tried the cp command again to copy disk6 to disk2 and got the same problem.

 

Now I just finished copying disk6 to disk2 through windows. All was fine. It took 19hours to copy 2TB data, not too bad at all.

 

tower-diagnostics-20170128-2141.zip

Clearly cp is doing something that's resulting in the Archive drive not recognizing that you're copying a large, contiguous file that can be copied directly to the shingled tracks; and is thus resulting in the persistent cache being filled and the drive's write speed dropping to a virtual stall.    Not sure what that might be (could be a bunch of small blocks with timing such that the drive isn't recognizing that they're contiguous, etc.) ... but in any event that's clearly what's happening.  Since Windows Explorer doesn't have this problem, I'd simply use it and avoid the problem altogether.

 

Thanks for the comments. I was just curious of what was going on with cp copy to archive drives and it's satisfying to learn that maybe I guessed it right...

 

As I said I'm having no problem using windows to copy so this is really not an issue to me at all. CP command would be faster for copying a whole raid disk to 8TB, but I only need to do this when I'm reducing number of drives and right now the process is already done.

I have frequently used rsync to copy files to the 8GB shingled drives and not experienced the symptoms you describe. I prefer using rsync as you can interrupt it and then continue.

I may give rsync a try tonight, as I have one last 500GB folder to move to 8TB disk2.

 

Right now the server is rebuilding parity after I removed disk 6. The speed is at 128MB/s, very impressive.