JorgeB

How can I use ddrescue to recover data from a failing disk? It can happen due to a variety of reasons, like a disk failing while parity is invalid or two disks failing with single parity, a user having a failing disk with pending sectors and no way to rebuild it using parity, for those cases you can use ddrescue to salvage as much data as possible. To install ddrescue install the the NerdTools plugin then go to Settings -> NerdTools and install ddrescue. You need an extra disk (same size or larger than the failing disk) to clone the old disk to, using the console/SSH type: ddrescue -f /dev/sdX /dev/sdY /boot/ddrescue.log Both source and destination disks can't be mounted, replace X with source disk, Y with destination, always triple check these, if the wrong disk is used as destination it will be overwritten deleting all data. If this is not the first time you use ddrescue make sure you use a different log file (/boot/ddrescue.log) (or delete the existing one) or ddrescue will resume the previous run and possibly not do anything. It's also possible to use an array disk as destination, though only if it's the same size as the original, but to maintain parity you can only clone the partition, so the existing array disk needs to be a formatted unRAID disk already in any filesystem, still to maintain parity you need to use the md# device and the array needs to be started in maintenance mode, i.e., not accessible during the copy, by using the command: ddrescue -f /dev/sdX1 /dev/md# /boot/ddrescue.log Replace X with source disk (note de 1 in the source disk identifier), # with destination disk number, recommend enabling turbo write first or it will take much longer. Example output during the 1st pass: GNU ddrescue 1.22 ipos: 926889 MB, non-trimmed: 1695 kB, current rate: 95092 kB/s opos: 926889 MB, non-scraped: 0 B, average rate: 79236 kB/s non-tried: 1074 GB, bad-sector: 0 B, error rate: 0 B/s rescued: 925804 MB, bad areas: 0, run time: 3h 14m 44s pct rescued: 46.28%, read errors: 54, remaining time: 3h 18m time since last successful read: 0s Copying non-tried blocks... Pass 1 (forwards) After copying all the good blocks ddrescue will retry the bad blocks, forwards and backwards, this last part can take some time depending on how bad the disk is, example: GNU ddrescue 1.22 ipos: 17878 MB, non-trimmed: 0 B, current rate: 0 B/s opos: 17878 MB, non-scraped: 362496 B, average rate: 74898 kB/s non-tried: 0 B, bad-sector: 93696 B, error rate: 102 B/s rescued: 2000 GB, bad areas: 101, run time: 7h 25m 8s pct rescued: 99.99%, read errors: 260, remaining time: 25m time since last successful read: 10s Scraping failed blocks... (forwards) After the clone is complete you can mount the destination disk manually or using for example the UD plugin (if the cloned disk is unmountable run the appropriate filesystem repair tool, it might also be a good idea to run a filesystem check even if it mounts OK) and copy the recovered data to the array, some files will likely be corrupt and if you have checksums or are using btrfs you can easily find out which ones, if not see below. If you don't have checksums for your files (or use btrfs) there's a way you can still check which files were affected: Create a temporary text file with a text string not present on your data, e.g.: printf "unRAID " >~/fill.txt Then fill the bad blocks on the destination disk with that string: ddrescue -f --fill=- ~/fill.txt /dev/sdY /boot/ddrescue.log Replace Y with the cloned disk (not the original) and use the existing ddrescue mapfile. Finally mount the disk, manually or for example using the UD plugin and search for that string: find /mnt/path/to/disk -type f -exec grep -l "unRAID" '{}' ';' Replace /path/to/disk with the correct mount point, all files containing the string "unRAID" will be output and those are your corrupt files, this will take some time as all files on the disks will be scanned, output is only displayed in the end, and if there's no output then the bad sectors were in areas without any files.

It actually is "unassigned" not "no device" on 6.2.4. Do you mind posting a screenshot, is yours different from this?

See if this helps: http://lime-technology.com/forum/index.php?topic=53689.msg517002#msg517002

How do I create a vdisk snapshot on a btrfs device? There are two methods of creating instant copy/snapshot with btrfs, if it's a single file like a vdisk use the first one, if you want to snapshot an entire folder use the 2nd, e.g., I have all my VMs in the same folder/subvolume so with a single snapshot they are all backed up instantly. Method 1 - Creating a reflink copy: Simple way for making an instant copy of a vdisk creating a reflink copy, which is essentially a file-level snapshot: cp --reflink /path/vdisk1.img /path/backup1.img Requirements: Both the source and destination file must use the same BTRFS volume, can be a single device or pool. Copy-on-write on (enable by default) Method 2 - Snapshot: Btrfs can only snapshot a subvolume, so first thing we need to do is create one, example below uses the cache device, it can also be done on an unassigned device adjusting the paths. btrfs subvolume create /mnt/cache/VMs You can use any name you want for the subvolume, I use one for all my VMs. The subvolume will look like a normal folder, so if it's a new VM create a new folder inside the subvolume with the vdisk (e.g: /mnt/cache/VMs/Win10/vdisk1.img), you can also move an existing vdisk there and edit the VM template. Now you can create a snapshot at any time (including with the VM running, but although this works it's probably not recommended since the backup will be in a crash consistent state), to do that use: btrfs subvolume snapshot /mnt/cache/VMs /mnt/cache/VMs_backup If at any time you wanted to go back to an earlier snapshot, stop the VM, and move the snapshot or edit the VM and change the vdisk location to a snapshot. To replace the vdisk with a snapshot: mv /mnt/cache/VMs_backup/Win10/vdisk1.img /mnt/cache/VMs/Win10/vdisk1.img or edit the VM and change vdisk path to confirm this is the one you want before moving it: e.g., change from /mnt/cache/VMs/Win10/vdisk1.img to /mnt/cache/VMs_backup/Win10/vdisk1.img Boot the VM, confirm this is the snapshot you want, shutdown and move it to the original location using the mv command above. Using btrfs send/receive to make incremental backups: Snapshots are very nice but they are not really a backup, because if there's a problem with the device (or even serious filesystem corruption) you'll lose your VMs and snapshots, using btrfs send/receive you can make very fast copies (except for the initial one) of you snapshots to another btrfs device (it can be an array device or an unassigned device). Send/receive only works with read only snapshots, so they need to be created with -r option, e.g.: btrfs subvolume snapshot -r /mnt/cache/VMs /mnt/cache/VMs_backup Run sync to ensure that the snapshot has been written to disk: sync Now you can use send/receive to make a backup of the initial snapshot: btrfs send /source/path | btrfs receive /destination/path e.g.: btrfs send /mnt/cache/VMs_backup | btrfs receive /mnt/disk1 no need to create destination subvolume, it will be automatically created. Now for the incremental backups, say it's been some time since the initial snapshot so you'll do a new one: btrfs subvolume snapshot -r /mnt/cache/VMs /mnt/cache/VMs_backup_01-Jan-2017 Run sync to ensure that the snapshot has been written to disk: sync Now we'll use both the initial and current snapshots for btrfs to send only the new data to the destination: btrfs send -p /mnt/cache/VMs_backup /mnt/cache/VMs_backup_01-Jan-2017 | btrfs receive /mnt/disk1 A VMs_backup_01-Jan-2017 will be created on the destination but much faster than the initial copy, e.g., my incremental backup of 8 VMs took less than a minute. A few extra observations: To list all subvolumes/snapshots use: btrfs subvolume list /mnt/cache You can also delete older unneeded volumes/snapshots, e.g: btrfs subvolume delete /mnt/cache/VMs_backup To change a snapshot from read only to read/write use btrfs property, e.g.: btrfs property set /mnt/cache/VMs_backup_01012017 ro false A snapshot only uses the space required for the changes made to a subvolume, so you can have a 30GB vdisk and 10 snapshots using only a few MB of space if your vdisk is mostly static.

Many ways of doing it, see the FAQ, fastest depends on how much used space it has and if you have or not an extra SATA port.

You'll never get twice the speed from one, there's always some overhead. I'm using a 24 SSD RAID10 pool and get between 800 and 900MB/s writes and 1GB/s reads over 10GbE.

For some time now I've been trimming a raid10 pool without any issues, raid0 should be no different.

With only 1 data disk + parity unRAID works like RAID1, much faster. Transfer rate data is for both disks, so about 135MB/s each.

Thanks for the detailed info, I hope to see trim support on the array in the future, the reason I stopped using my SSD only array was the deteriorating write performance I was getting because of the lack of trim, never more than 100 to 150MB/s and sometimes much less, like 50MB/s writes. I did use it a lot and never got a sync error, when I get the chance I'll check the type of trim each different model used. I do still use a couple of SSDs on my main array, for these I only care about the read speed, still trim support would be always nice for endurance, also never got a single sync error, and I been using them for about 6 months, just checked and they are DZ_TRIM.

What are the requirements for the second parity disk? (v6.2 and above only) Same requirements as the other parity, the size of the parity2 disk must be equal to or larger than the largest data drive, parity1 size is irrelevant.

I tried to edit the wiki myself but can't, I always get I'm not logged in, this is what I wanted to edit: on step 12: "Go back to the Main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it (click on Unassigned???)" - that should be changed to (click on "no device"). And more importantly, after step 11 there should be a warning about the new config, something like this: "Doing a new config will reset the file system type for all disks to auto, while usually this is not a problem, especially with the latest unRAID, in some circumstances can lead to unmountable disk(s), if that happens you just need to select the correct file system for that disk(s), if in doubt ask for help."

No, trim doesn't work on array devices, parity is maintained (but write performance takes a hit). True, but they can be worth using for the read speeds when using 10GbE, I get 500MB/s read speed from my array SSDs. Like I said, depends on your use case, but they can be a valid choice.

Depends on your use case, I had an SSD only array for some time, ended up moving them to a RAID10 cache pool due to the low write speed on the array, tried various SSDs but but never got sustained writes above 100-150MB/s, I still use a couple of SSDs on my main server due to low power and read speed (write speed is not important here because I use a NVMe device for cache)

It *should* work without any issues, especially since it uses a LSI chipset.

For v6.2 users using P parity only you could make small changes to 4 steps to make it work, but it involves doing a new config and that can make it much more prone to a user making a mistake: 10. Stop the array; we are now going to swap their assignments; when stopped, click on Tools -> New Config -> on "Retain current configuration:" select "All", check "Yes I want to do this", click Apply, click Done; go back to the main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it; 11. Click on the dropdown for the other drive (e.g. Disk 10), the one being converted, and reassign it as the physical drive that was just added, the new drive that was empty; 12. Click on the dropdown for the slot of the new drive (e.g. Disk 11) and reassign it to the physical drive that was being converted (e.g. Disk 10); you have just swapped the 2 drives, which is fine as they are identical (except for file system format) 13. (as it was) 14. You should see all array disks with a blue icon, a warning that the parity disk will be erased, and a check box "Parity is already valid"; important click it before the Start button to start the array; it should start up without issue and look almost identical to what it looked like before the swap, with no parity check needed; however the XFS disk is now online and its files are now being shared as they normally would; check it all if you are in doubt I checked and I believe the procedure is correct but in case you decide to use it it should be double checked by someone else.

Yes.

Anyone tried this: https://ransomfree.cybereason.com/ Free and supposed to stop 99% of ransomware strains.

That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight. How much RAM your machine have ? This happens only when some big files are moving.. I think it has to do with low RAM...4GB ? what ya think ? Now I have 8 but at the time I used it I had 4GB.

That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight.

You can't format a disk with the plugin and then use it on the array, it will appear as unmountable.

+1

They do get pretty hot, max operating temperature is 55C, there's no sensor, just make sure you have some airflow around it.

You don't need to wait, v6.3-rc already has them.

That was the parity build (sync), now you can do a non correcting check to make sure all is ok.

You can, you'll need to let parity2 sync first, the you can do a non correcting check, at any time in the future you can unassign it and go back to single parity.