Defragment my data


tsawind

Recommended Posts

TsA here, so first I want to say this community has been very helpful.

 

I finally got my dual headed, 4k gaming, NAS system online and stable. (for the most part)

2 - 8tb wd red, one in parity, one as a disk1 in the protected array. Formatted as xfs

2 - unassigned 500gb SSD (formatted NTFS direct passthrough to windows 10 pro VM)

2 - 8tb external drives for manual backups offsite.  One being a backup for the backup.

 


1. How does UnRaid physically write files to disks through a windows VM?
2. My question is, how can I divide my brand new array into physical sections, kind-of like physical partitions, so as I start building it I can manual fix some of the fragmentation issues with my old files. (i.e. 2TB longtermstorage, 4TB movies, 1TB sharedworkstation, 1TB cache)
3. Should I not be so worried about the physical fragmentation of the data. (I want to be prepared for a data recovery company if, god forbid, catastrophic failure)

Thank you,

TsA
zrNl777.png

Edited by tsawind
Added picture
Link to comment
  • tsawind changed the title to Defragment my data

Update on progress, woke up with morning and sytem was turned off, haven't found where to look to find any logs of why it turned off.  Machine had array running and 2 VMs of windows 10 pro running. Re-sources have been split 50-50 to VMs. (7gb ram, 4 cores 8 threads 1 500gb SSD). I suspect something with the VMs cause the crash, maybe I need to learn how to dedicate one core to unraid, 4 to primary VM, and 3 to second VM?

I started the array and now it is parity checking with the 2 8TB wd drive.  I was planning to start writing data, but now I am hesitant, concerned about stability and parity check will take like 12+- hours.
I am going to reboot into a baremetal, clean install of window 10 pro and start doing writing data onto one of the WD easystore 8tb NFTS external drives. I have maybe about 3 TB of data to try to recover and go through, about 500gb is what I would consider "critical".  (Wedding pictures, tax statements ect).
I'm hopeing this wont hurt anything to click cancel parity check, I really don't want this to be a normal thing or that parity drive really probably isn't for me.

Link to comment
5 minutes ago, tsawind said:

Update on progress, woke up with morning and sytem was turned off, haven't found where to look to find any logs of why it turned off.  Machine had array running and 2 VMs of windows 10 pro running. Re-sources have been split 50-50 to VMs. (7gb ram, 4 cores 8 threads 1 500gb SSD). I suspect something with the VMs cause the crash, maybe I need to learn how to dedicate one core to unraid, 4 to primary VM, and 3 to second VM?

I started the array and now it is parity checking with the 2 8TB wd drive.  I was planning to start writing data, but now I am hesitant, concerned about stability and parity check will take like 12+- hours.
I am going to reboot into a baremetal, clean install of window 10 pro and start doing writing data onto one of the WD easystore 8tb NFTS external drives. I have maybe about 3 TB of data to try to recover and go through, about 500gb is what I would consider "critical".  (Wedding pictures, tax statements ect).
I'm hopeing this wont hurt anything to click cancel parity check, I really don't want this to be a normal thing or that parity drive really probably isn't for me.


The fact that when you started it back up, a parity check started means that there was an unclean shutdown. The parity check is to verify that the parity drive and the data drives are still "equal". Why would you not want not want to check that?

 

Link to comment

Thank you for the fast responses.

No I do not have an UPS yet, but I haven't had power failures at this house in months. (underground electric service)

And yes I understand parity checking from an unclean shutdown, that is a good thing.  The fact is there should have been no read/writes on the array between when I went to bed (parity valid) and woke up.  Also these are 2 brand new drives.  I mean Ill recheck parity that isn't an issue, issue is I don't want to start writing data during a parity check, I don't even know if unraid will let me do that sort of thing.

Link to comment

Will this hurt anything to cancel the parity check? I want to reboot to baremetal and start recovering data.
I will go buy a UPS and setup a video camera to monitor the screens I guess.  Any other suggestions while I have your attention?

Good to meet you squid, I have heard good things.

 

Link to comment
14 minutes ago, tsawind said:

I mean Ill recheck parity that isn't an issue, issue is I don't want to start writing data during a parity check, I don't even know if unraid will let me do that sort of thing.

You can always use the server normally during a parity check or rebuild.  Just reads / writes / parity check speed slows down.

 

So, did it come up as an unclean shutdown?  99.9% of the time on a crash, the computer will NOT shutdown.  If you didn't have a power failure, then I'd suggest a bad power supply, possibly CPU shutdown the system due to an overheat situation.  There is also a thread here from a few years ago that was determined to be a cat playing with the power switch.

 

8 minutes ago, tsawind said:

I have heard good things.

Don't believe a word of it.  Lies, more lies, and damned lies

Link to comment

The cat thing is funny, because it actually was on my list of things to check.  I haven't re-covered (taped hard plastic over it) my power button on top of computer since the rebuild.  I am trying to recover data some 10yr old hard drives ect, very risky, which is why I am asking. I have not plugged drives into power in months (part of why I am building a NAS). I will just deal with unraid later, im canceling parity and rebooting to a clean windows 10 install on a SSD that I know is stable.  Ill check these forums later today.

Wish me luck that I don't have to start calling data recovery centers today!
 

Thank you,
TsA

Edited by tsawind
Link to comment

BTW figured I would mention my luck and how I got here.  Every time a order new hardware this happens to me.
 

Needed an upgrade, gaming rig was 10yrs old and still running strong.

Ordered a 1070ti
Bought 4 new sticks of DDR2 and some used fans
Installed a fan onto motherboard and shorted it out. (my fault)

Succession of deaths an suicides to follow:

1 motherboard murdered.

like 1 or 2 fans dead.

2 fans committed suicide in front of me. (both were 120mm)
1 Seagate 250gb drive committed suicide, just started clicking, no recovery.

1 Seagate 250gb drive spun up, read all data and copied onto 1.5tb WD green.  Format attempted and it committed suicide.

1 WD 750gb spun up, read almost all data (bad sectors) and copied onto 1.5tb WD green.  Format attempted and it committed suicide.
Other miscellaneous murders, suicides, and deaths from old age have been witnessed during rebuild.

Now I am lookin at this 1.5tb "DATE: 06 MAR 2010" drive that is full, a 500gb "Date: 10301" (I'm guessing full) that I haven't dared plug in, and a dozen USB flash drives and camera flash drives on my desk in front of me. I also have some really old misc. 80gb drives to check through and trash during this process.

I'd say I'm a bit nervous ATM and I believe that is the definition of Fragmented.  So an uknown, unclean shutdown, of a brand new NAS is a bit concerning to me ATM.

Wishing myself luck,
TsA

Link to comment

Update:
I believe I have some form of mental disability (like autism), just to put things into perspective.

I Just had an AHAH moment, Huge breakthrough, like party time.  I just learned about M-Disks.

I am ordering drives and disks to incorporate into the system, not really too concerned about extreme redundancy in the array if I can use M-Disks.  They theoretically will last 1,000+- years.  They come in 4.7 GB DVD, 25,50,and 100GB Blu-ray.  They are one time write and exactly what I have been looking for, for literally 50+ hours in the past month. And realistically been seeking this data medium for years.

Couldn't talk myself into turning off the parity check, it is at 25%.

M-DISC Is THE BEST solution I have come up with for redundancies. MAJOR MAJOR AHAH MOMENT.

Edited by tsawind
Link to comment
4 hours ago, tsawind said:

What should I be looking for in a UPS, (minimum specs)

apcupsd compatible, which means almost any APC branded unit, many Cyberpower work except for the turn off after shutdown option.

 

As far as capacity, that's up to your budget. Almost any except for the very smallest units will probably have enough battery to allow for a clean shutdown IF your windows 10 VM's cooperate with a commanded shutdown. If your VM decides to be stubborn and stay running, you could end up with a crashed VM when unraid forces the issue, or your batteries run out. The stock unraid UPS control software (apcupsd) has clients for windows as well, so you can install UPS monitoring in your VM's. When a power outage occurs, you can program your VM's to shut down after a few (2-5) minutes on battery, so when unraid shuts down your VM's are already off.

 

Your goal for maximum equipment longevity is to use the batteries to less than 50% discharge, which means maximum usable runtime of around 10 to 15 minutes for mid to upper level consumer UPS. Server grade UPS can add battery capacity for more runtime.

 

Do you have a method to measure the ACTUAL power draw of your system at full load? Idle power draw is useless, as everything spins up and works hard during shutdown.

Link to comment
2 hours ago, jonathanm said:

If your VM decides to be stubborn and stay running, you could end up with a crashed VM when unraid forces the issue, or your batteries run out.

Install the VM client software on the virtio iso disk and then tell unRaid to hibernate the VMs during a shutdown.  You'll never have a problem

Link to comment
1 hour ago, Squid said:

Install the VM client software on the virtio iso disk and then tell unRaid to hibernate the VMs during a shutdown.  You'll never have a problem

Does that initiate windows hibernation, or the KVM save / restore hibernation?

 

If it calls for windows to hibernate, I don't think windows will act on it if it's in the middle of an update install.

Link to comment

I don't care about the VMs, im talking about the array where the important stuff is.  (I dont want to have to do parity for 20 hours if the power spikes)  This is a 2 headed gaming setup, each head has its own SSD passthrough.  I made full backups for each install after I did all the quirks like turning off windows search, power setting ect ect.  These backups are saved on the UnRaid array, so if a VM starts to fail or get slow or ANYTHING, I just bleep boop bleep, reformat SSD, pre-setup "clean" install.  So as far as im concerned, if UPS hits 75% battery from 100% it can force close the VMs, keeping the router and UnRaid array online as long as possible.


Looking for at least 30 minutes power for UnRaid before it shuts down cluster.
Do I just need a wall plug from amazon that detects watts drawn by system? Then calculate largest number drawn during a stress test then divide by 30 minutes I assume?

Sounds like apcupsd compatible (whatever that is) is a requirement.
Buy a name brand UPS.

Ideally system will restart and array will online when power comes back on.

 

Thanks, TsA

Link to comment

If you are looking for 30mminutes runtime on UPS then you should probably size the UPS to handle at least twice that.    This is to allow for the fact that batteries degrade and also to give time for a tidy shutdown if that limit is reached.    If you are measuring the power draw as part of trying to size the UPS make sure that you have all your drives spun up at the time as that is the worst case.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.