Moving from an older Windows 10 system to Unraid. I'm terrified.

[csimpson]

20 minutes ago, Frank1940 said:

ls -al "/mnt/disks/Expansion/"

Get a screenshot of the GUI terminal after you run the following command:

 There's some garbage files in there that came with the seagate drive that I'll be getting rid of.  But it's mainly the directories 

Archive
Family
Finances
Media 
Other

That are going to stay.
 

 

Edited May 8 by csimpson

[Frank1940]

Try asking for assistance in the Unassigned Devices plugin support forum...

[csimpson]

Okay, thanks.  I've posted over at 

 

 

[Frank1940]

14 hours ago, csimpson said:

 

rsync -av --delete "/mnt/user/Shared/" "/mnt/disks/Expansion/"

 

 

It looks like this is a backup for data on your server.  I would like to point out that the use of the    --delete   switch is a topic of discussion for backup.  The problem is that if a file is accidentally deleted on the source, it will be deleted on the backup the next time you run this script.  If you don't use the switch the file will always be there in the backup when you 'discover' that you really need that file and it is missing from the source.  

 

I will concede that there are instances where deleting files is really what is needed but it may not be the best choice in all situations. 

[csimpson]

27 minutes ago, Frank1940 said:

 

It looks like this is a backup for data on your server.  I would like to point out that the use of the    --delete   switch is a topic of discussion for backup.  The problem is that if a file is accidentally deleted on the source, it will be deleted on the backup the next time you run this script.  If you don't use the switch the file will always be there in the backup when you 'discover' that you really need that file and it is missing from the source.  

 

I will concede that there are instances where deleting files is really what is needed but it may not be the best choice in all situations. 

That's why I really like Apple's Time Machine.  It allows you to go to a date and see a snapshot of the system and files that were there while doing an incremental backup in a GREAT UI.

However, with a backup drive on Unraid, let's say I change the structure of my media or delete some tv shows on my main array. WIthout doing an actual "sync" it would keep those files in their place and before I know it, the 14TB hard drive may be filled with duplicate files from simply being shuffled around. I used to need to delete tv shows and movies often so that may not be as much of an issue going from 4TB to 16TB, but it's moreso about having a duplicate of the data at a certain time.

 

In fact, on my old system, that's why I had two drives. One drive was a mirror of the other, but only nightly so if I deleted something off my main drive, I had 24 hours to recover it. And then my external drive was whenever I plugged it in.

Apple's Time Machine fills the drive and then removes older files as it needs to.  Best of both worlds. A Time Machine like app for Unraid would be...  Awesome.

[Frank1940]

2 hours ago, csimpson said:

A Time Machine like app for Unraid would be...

 

As I think about it, a BASH script could be written to do this.   (I couldn't do it but I bet there are some Unraid users who could.  It would not be trivial task.  The Unraid Preclear plugins and Dockers all still use a BASH script that was written about fifteen years ago!) 

 

A bit of trivial:   Many long years ago, I use to work for Western Electric.  Bell Lab engineers use to pride themselves on what things they could do with Shell scripts.  I once heard that the all of the Bell company telephone directories (those massive books you use to find on your front porch every summer) were generated using a Shell script. 

[csimpson]

So this is becoming a very unpleasant experience.

I got RSYNC working, but then realized that all of the files copied from my backup drive were recreated with new dates.  So everything on the unraid system lost their dated created.  Now, I'm going to one of my older backup sync drives which is correct, and am trying to delete some of the folders that are important with original dates and recreate them.

However, when trying to delete a folder in Krusader called "Family" I can't delete it because there's a file called .fuse_hidden[nnnnn...] sitting in there.  I can't delete the file either.

Jumping out to Terminal:

 

root@fileserver:/mnt/user/Shared# rmdir Family
rmdir: failed to remove 'Family': Directory not empty

Can't remove it in Terminal either.  I'm not a Linux person, but I did grow up with DOS 6.2 so Command line stuff doesn't scare me. But this is getting highly frustrating with my important files.

[csimpson]

Okay,  so stopping the array and then shutting down te docker apps and restarting them seemed to eliminate that .fusehidden2345345 file.

Now I need to think of a good way that I can delete around 500,000 photos and videos since 1988 all of which show they were created on May 1, 2024. From there I need to "move" the files from my external drive back over to maintain the accurate dates. 

But using Krusader, "move" actually removes them off the external drive.

<banging my head>

[csimpson]

Okay, it looks like RSYNC might get me to the finish line.  It's going to be a slow, painful race of replacing existing files with proper date stamped ones...  But I think I can do it after doing some tests...

[_cjd_]

I've been using Duplicati to make backups of critical data and it does provide similar "timeline" capabilities as Time Machine... not nearly as slick an interface but it gets the job done. At the moment all "critical" data also requires fast access so is on RAID 10 SSD, not the parity protected array, but there's nothing to prevent writing such data to a single array drive and using Duplicati to replicate to another array drive (that's one of the things I particularly like about the default Unraid array setup).

I also have a backup Unraid server (scheduled rsync) and external backups of things as well.

 

You're taking the time to verify results, which is something a lot of people forget to do. For the rest - keep calm, keep learning. Test, verify, and leave yourself an exit strategy. And do more work in terminal.

 

One thing I learned about rsync is that it's weird (or at least, not obvious) about some things - I had to go back and add -H (linked files become actual in the target otherwise) and -S for sparse files as my backup was a couple TB larger than source... so even when it works, there's always something new to learn.

 

My biggest panic so far was updating the encryption key on my array. My oversight was forgetting how add-on scripts work in Unraid - I thought I was setting up a non-destructive addition of a new key, but it was a replace. Even still, verified everything and all good... till I went to start the array. Tried everything I could think, no luck. Finally decided to risk a reboot and everything started up perfectly. No idea why it needed a reboot. When you start doing this kind of stuff, you *will* make mistakes as you learn. I've had to jiggle cables to fix disk speed issues and early on, bad memory causing all sorts of weird; when setting up my backup server, two hard drives up in (literal) smoke (fortunately NOT the brand new 16TB drives... I was using stuff I had lying around to test the automated rsync before buying new hardware) - probably the power supply (it worked fine with one spinner, but wouldn't even spin up two; was fine with 3 ssd). This is why you keep backups. Next time it'll be a NEW problem. I've also done more stuff without problems than stuff I've had problems doing.

[csimpson]

I'll take a look at Duplicati.  I'm using RSYNC with moderate success.  I'm copying all the data back over to the array from my external drive as it's the source of truth.  Going to take a couple days, but it seems like the "modified dates" are mostly correct.  Files are correct and most folder are, but tooehrs are decided that they're created today. I need to determine if that's content that matters for dates. 

Krusader is proving to be less than reliable. However, Midnight Control has solved some of the problems Krusader created such as a large folder deletion that Krusader couldn't finish the job and delete the folders. It did some, but gave up on others and just sortof did a half assed job and figured I wouldn't notice the mess. 

You said "My oversight was forgetting how add-on scripts work in Unraid" and that's the issue for me as well.

I'm not wanting to turn this into a thing I don't think...  I like to tinker, but not with family photos and important documents. I'm looking for GUI and UX friendly solutions so that when I can come back in time, I'll have a recollection of what I needed to do or what happened.  I'm not going to remember...
 

rsync -avz --info=progress2 --no-perms --no-owner --no-group --delete "/mnt/disks/Seagate_Backup_Plus_Drive/Media/Home Videos" "/mnt/user/Shared/Media/"

But a year from now when I come back I'll be like...

Edited May 10 by csimpson

[csimpson]

I'm absolutely livid.

 

I just lost EVERYTHING off my shares. I was finishing the last RSYNC and the following command deleted everything.
 

rsync -avz --info=progress2 --no-perms --no-owner --no-group --delete "/mnt/disks/Seagate_Backup_Plus_Drive/Family/" "/mnt/user/Shared/"

I realize I needed to have the trailing /Family/ in the destination. I'm so angry right now. Now I need to spend another week to copy all the data over.  This is why I prefer GUI's. A single character in the wrong spot can mess up everything. I know I can do a dry run, but with 5,000,000 files, I don't see how RSYNC can give a summary.

I'm so disappointed.

Edited May 11 by csimpson

[csimpson]

Okay, I've taken the parity drive out of the array to re-transfer all the data again. Hopefully, once completed(again), I can turn on the party drive and it'll run a parity sync. With my experience so far, UNRAID will have an issue with this and tell me there are errors all over the place and I'll need to wipe the parity drive and do all of that over again. And maybe it won't let me.  Maybe it'll tell me there are errors forever.

Sorry, I'm really frustrated with how vulnerable all of my data and files have been with UNRAID.  I want it to work, but in practice, this has been a very, very painful process and I've been feeling very exposed.

Edited May 11 by csimpson

[_cjd_]

First things first:

I think Krusader will do what you want it to.

I just installed it (Krusader) to do a test, because the timestamp change you mentioned recalled an observation I had with rsync. It maintains timestamp correctly, copying from an Unassigned NTFS mount into the array. HOWEVER - this is only reflected when it completes everything. While the copy is active, the timestamp is the ~now. For a large copy, it may reflect the wrong timestamp for quite some time.

 

Second:

Unraid is not the reason for your troubles getting files copied. An unfamiliar system is - it happens to be Unraid, but fundamentally this is a *nix challenge, and particularly a command-line thing.

GUI usually bakes in assumptions - and you are familiar with those baked into the Windows File Explorer. Command line has none of those. It can be frustratingly precise.

Setting up rsync for my backup Unraid I made a number of similar oversights getting the path exactly right. Correct and move on.

You can also do things in smaller pieces, or at least do a test on your command over a smaller data-set to verify it's doing what you want before taking on the large copy.

[csimpson]

Yes, I know it's a "me" issue. It's unfamiliar and I'm trying to make it work. My old setup was familiar and worked perfectly for many, many years but I wanted to try something new with proposed advantages and with less overhead.  Stable, so far, it is not.
 

I did use Krusader to change a folder name on my external drive. I think that messed up everything on the external drive that I'm copying from. And then RSYNC looked at my --delete parameter and because I had a path wrong, it deleted everything. Once again, my mistake. But I don't want to use Command Line and RSYNC for this very reason, but the other tools aren't working well enough.

Edited May 11 by csimpson

[csimpson]

1 hour ago, _cjd_ said:

First things first:

I think Krusader will do what you want it to.

I just installed it (Krusader) to do a test, because the timestamp change you mentioned recalled an observation I had with rsync. It maintains timestamp correctly, copying from an Unassigned NTFS mount into the array. HOWEVER - this is only reflected when it completes everything. While the copy is active, the timestamp is the ~now. For a large copy, it may reflect the wrong timestamp for quite some time.

 

Second:

Unraid is not the reason for your troubles getting files copied. An unfamiliar system is - it happens to be Unraid, but fundamentally this is a *nix challenge, and particularly a command-line thing.

GUI usually bakes in assumptions - and you are familiar with those baked into the Windows File Explorer. Command line has none of those. It can be frustratingly precise.

Setting up rsync for my backup Unraid I made a number of similar oversights getting the path exactly right. Correct and move on.

You can also do things in smaller pieces, or at least do a test on your command over a smaller data-set to verify it's doing what you want before taking on the large copy.

Interesting observation with Krusader.  I found files were somewhat correct, but folders were not. Though, I've tried a bunch of variations and have forgotten which does what and what affects the time properly.  I know RSYCN was doing a better job until I accidentally blew away the entire directory with it. I saw in a video (setting up Plex) that move maintains the original creation date, etc and copy does not.

Edited May 11 by csimpson

[_cjd_]

42 minutes ago, csimpson said:

Interesting observation with Krusader.  I found files were somewhat correct, but folders were not. Though, I've tried a bunch of variations and have forgotten which does what and what affects the time properly.  I know RSYCN was doing a better job until I accidentally blew away the entire directory with it. I saw in a video (setting up Plex) that move maintains the original creation date, etc and copy does not.

So... it's not JUST a "you" problem - you're definitely not alone in the struggle trying to do something in a totally different way than you're used to doing it. Doesn't change the frustration in the moment whatsoever, but it can help when you take that deep breath to re-orient and plan next steps.

 

I did a (multi level) directory copy with a simple drag & drop in Krusader. I had to tell it I wanted Copy. Timestamp was correct on each folder as well as individual files.

 

rsync is amazing when you have it sorted properly. That's a "when" with a bit of a learning curve, and internet "just do this" don't always help you understand the specifics which may actually matter. I couldn't automate my Unraid backup without it; the backup server is powered off most of the time, and its ability to not just transfer every bit over the network when not required is quite nice or it'd be days of backup (hard drives being the throughput limitation).

 

I don't know what krusader does under the covers, to be honest. mv vs cp vs rsync and they all work differently. You definitely do NOT want to be moving files. But rsync is the right tool for your particular needs if you're working in terminal.

 

Comfort with command-line and reading man(ual) pages is worth doing, regardless - even in Windows. As a server, stability is as much down to hardware choices and setup as anything. Unraid builds in some tricks Windows doesn't offer without significantly more setup work (if it's even an option), without a doubt. If you'd gone with another *nix NAS option you'd probably be dealing with similar struggles. Windows is better at glossing over the details for a lot of things the "average" user is likely to do, but it also limits quite a bit of what you can do as well.

[csimpson]

I appreciate the patience. And yes, you're right. In the moment, when dealing with significant data loss it's frustrating with a small oversight. In Windows, I used Syncback.  I had it configured and it ran perfectly every time.  It's why I'm able to restore it successfully. Then, this morning I was trying to copy back to the share but that inadvertent removal of the share name in the sync destination had the files going to the root. But when looking on my finder, through Krusader, Midnight Control or command line, I couldn't see anything and had assumed it was something that Krusader did.

I've got the RSYNC commands all stored in a document now where I can copy/paste. I'm also adding --dry-run to the syncs. I did many tests before this morning when I woke up and executed the final sync.  However, I removed the share name for whatever reason and everything was gone.  Sloppy Saturday morning oopsie. I was wondering why it was taking so long but it's not until folders started to disappear I clued in!

Edited May 11 by csimpson

[Frank1940]

I have a couple of .txt files where I store the rsnyc command lines that I use.  I copy-and-paste them.   (No typing commands in real time on the command line--- too easy to make a simple typing error!!!   That is also why many Unix users write Shell Scripts!)  When I am testing things (i.e., particularly various rsnyc commands out, I use a very small subset of my data and make sure that things work right before I use it on the 'real' data'.   I also include notes of things like the trailing slash problem/issue and when-and-why to use it in my .txt files. 

[csimpson]

So when this transfer is done and I turn the parity drive back on, it'll need to do a new parity check.  When it does, will it need to rebuild the parity automatically, or will it give me a bunch of errors that need to be fixed?

[Frank1940]

1 hour ago, csimpson said:

So when this transfer is done and I turn the parity drive back on, it'll need to do a new parity check.  When it does, will it need to rebuild the parity automatically, or will it give me a bunch of errors that need to be fixed?

 

It should simply do the parity build on the parity drive.  It will take about two hours per TB of the parity drive size.  There should be zero errors.  IF there are errors, you have other issues.  If you should have issues, please start a new thread in the General Support section of the forum and be sure to include a Diagnostics file. 

[csimpson]

11 hours ago, Frank1940 said:

 

It should simply do the parity build on the parity drive.  It will take about two hours per TB of the parity drive size.  There should be zero errors.  IF there are errors, you have other issues.  If you should have issues, please start a new thread in the General Support section of the forum and be sure to include a Diagnostics file. 

Okay, great. It's running the parity check now.  Will be done later today.  Hopefully it's all good.

[csimpson]

Okay, so an update.  Everything is done and humming along.  Other than a weird issue with my mac which indicated it couldn't connect to the share until I rebooted Unraid, I'm really enjoying it now that it's up and running.  I've got the cache drive installed and the array drives are spinning down after an hour of no use.  Idle draw according to my UPS is 45W (56W with the array drives and parity drive spun up) which I'm comfortable with.  Plex is working, I've installed a few Dynamix plugins and have Prometheun and Grafana up and running (wish I could have the stats on the Unraiid dashboard rather than needing to log into the docker). AppData is backing up nightly to the Array but not shared. RSYNC successfully syncs with my external 14TB drive. 

So all in all, the setup process was an uphill battle due to the various challenges I faced, but this is a wonderful community so far and I was able to move past those issues.  No data was lost in the end and nearly all files that I needed to be properly time stamped are now correct.

 

Next step is to create some automated user scripts that spin up and sync my files to my external drive and possibly use Duplicate for backups on my network share. However, I'm trying to figure out how to back up into a folder called "backups" on my share and it's not completely intuitive. Seems it wants other locations. I also haven't looked into it too much.

All in all, I've installed the following plugins:
- Appdata

- Community Applications

- Dynamix Auto Fan Control

- Dynamix File Integrity

- Dynamix File Manager

- Dynamix S3 Sleep

- Dynamix System Buttons

- Dynamix System Information

- Dynamix System Statistics

- Dynamix System Temperature

- File Activity

- Fix Common Problems

- Parity Check Tuning

- Recycle Bin

- Tips and Tweaks

- Unassigned Devices

- Unassigned Devices Plus

- Unassigned Devices Preclear

- User Scripts

The following Dockers are running:
- binhex-krusader

- disquetv

- duplicati

- Grafana

- Plex-Media-Server

- prometheus

Any others that will help or recommended?

 

Thank-you all!

Edited May 16 by csimpson