Clarification question

[ICVDOK]

Hello Unraid community , I've just installed unraid to test it on a old qnap where I'm try to understand the concept.

 

The device has 4 disks 2x3TB and 2x8TB. I would like to setup the 2x3TB disks with max speed and redundancy (it was a RAID 1) with the old QNAP OS config. Remainig 2x8TB as single disks without any redundancy or fault tolerance. Means 2 disks completely separated.

 

What's the best configuration to achieve this setup?

 

Many thanks

[itimpi]

You could put the 2x3TB into a ZFS raidz1 pool for maximum speed and the 2x8TB drives into the main array without a parity drive.   The 2x8TB could alternatively be put into single device pools of their own.

 

Perhaps a bit more detail of your intended Use Case would help pin down the best option?

[ICVDOK]

Hello, many thanks for the clarification, let me elaborate a bit.

This NAS is used to duplicate data from the primary one (again QNAP for the moment, new DIY is coming...), mainly used to host pictures, movies and music (Office files are on Azure).

I had a very bad experience last summer where a disk on my primary NAS fail and the rebuild process fail as well. Recent backup and a professional QNAP support fortunately mitigate the dataloss but, my backup strategy is changed, no more HW RAID for home lab, just duplicate the data across multiple disks. (explain the 2x8TB st alone) that can be attached everywhere and readed (it is true with Unraid?). An external disk rotation to be moved outside my home.

 

2X3TB should be used to host nfs volumes for my docker servers (and then duplicated twice on the 2x8TB) , reason why the performances are relevant.

I hope that this explain a bit better the requirements.

 

In the meantime I've purchased the first license.

Thanks again for the support.

[itimpi]

If the 2 x 8TB drives are primarily for backup will you want to be able to remove either or both of them easily at any time?   If so then it might make sense to use the Unassigned Devices plugin to handle them.   That would give you more choices on the file system format for these drives which might be relevant if you want them to be easily read on non-Linux systems.

 

In terms of backing up from the 2x3TB to the 8TB drives you could do this via a script handled by the User Scripts plugin scheduled to run at whatever frequency you think is sensible.

[ICVDOK]

OK, I follow your suggestion and here is the result (maybe) 🙂

 

 

The warning message on backups folders should be because no parity is available (it can be deactivated?) . The amount of the service free space is wrong as well as the warning because is on RAID 1. There is something wrong I did?

 

In the meantime let me check the unassigned device plugin but, in case of failure, important to me to be able to access to the data, maybe also from another unraid NAS. If it's possible, I'm fine with this.

 

[ICVDOK]

Quote

That would give you more choices on the file system format for these drives which might be relevant if you want them to be easily read on non-Linux systems.

 

Note: Access from Linux system mounting the drive in caso of failure is possible with this config?

[itimpi]

1 hour ago, ICVDOK said:

 

Note: Access from Linux system mounting the drive in caso of failure is possible with this config?

 

Yes as you can use any standard Linux file system with Unassigned Devices.    You can also use other file systems such as NTFS (for Windows) and HPFS (for Mac).

 

When the drives are part of the main array then the options become XFS, ZFS and BTRFS which are easily mountable in any Linux system but not so easily mounted on Windows/Mac systems.

[itimpi]

1 hour ago, ICVDOK said:

The warning message on backups folders should be because no parity is available (it can be deactivated?) . The amount of the service free space is wrong as well as the warning because is on RAID 1. There is something wrong I did?

 

Each drive in the main array is an independent file system so they are not RAID1.  You could only have them as RAID1 if they were in a pool.

 

The warnings about backup1 and backup2 can NOT be deactivated (at least in current Unraid releases) as without parity drives in the main array are not protected.

 

 

 

[ICVDOK]

Great, looks OK for my requirements. In the meantime I was able to activate LACP network bound, another little progress. Very last question, there is some kind of repica service to be able to get or pull files from QNAP?

 

 

Rsync would be the best choice probably but I don't know if it's available as native service.

 

[bmartino1]

As of writing, I'm not aware of a docker/plugin for Qnap integrations.

I recommend downloading the User Script and nerd tools plugin.

 

Optional for screen backgroud tasks

 

with nerd tools install screen to use terminal user scripts.

 

Rsync is built-in to unraid.

https://medium.com/couchdrop-io/what-is-sftp-and-how-are-ftp-sftp-scp-and-rsync-different-252a664b57f4#:~:text=Rsync vs SFTP,designed for secure file transfers.

 

we will now make a cron script to do a rsync:

/Settings/Userscripts

 

add new script: 

In my example, I back up my local machine's appdata to my zfs pool:

I named this script Rsync-Cache-docker appdata backup 

 

click the gear and edit script to add terminal bash commands:

 

 

we can run a names screen to make sure it runs and stays in background. then we can go custom and set a rsyn timer on a cron .

 

Example script code:

#!/bin/bash
#cd /mnt/user
rsync -anv --exclude='lxddata' /mnt/user/appdata/ "/mnt/user/Program-Installers-PCRepair/Other OS/Docker-Rsync-Backup/"

 

your rsync command will be different I recommend using the sftp option to connect to your qnap and rsync to /mnt/user/*name of zfs poll or folder created in disk array

 

Other rsync example:

rsync --dry-run --itemize-changes --archive -e 'ssh -p 5533' root@tower:/mnt/user/system/ /tmp

 

 

Edited April 1 by bmartino1

[bmartino1]

it may be easier to use qnap and connect to a smb/cifs share in qnap to run a rsync command:

 

https://docs.qnap.com/operating-system/qes/2.2.x/en-us/GUID-48BD8578-9D7F-4EC0-B733-BAE05E9A3524.html

[ICVDOK]

Thanks for the suggestion, yesterday I did some checks. From UNRAID I was able to mount the QNAP using Unassigned Device plugin via NFS. It could be sufficient for data duplication.

I've also check the option suggested to access from QNAP to UNRAID via RSYNC but , I suspect, RSYNC on UNRAID isn't available out of the box (it's true?).

I've then installed a simply docker (axiom-data-science/rsync-server: A rsyncd/sshd server in Docker. You know, for moving files. (github.com)) with RSYNC server exposed and, finally I was able to connect to UNRAID via RSYNC from NAS.

 

Creted some replica jobs on QNAP and voilà (no scripting for the moment). Obviously there are still some improvements or fine tuning to do, but it works.

 

The performances are just sufficient (about 50Mbps) but considering the very old HW is OK.

 

Unless I miss something, I would prefer to have RSYNC as a native service on UNRAID. What's your view?

 

 

 

 

 

 

 

[bmartino1]

rsync is available out of the box just not a fancy web interface to manipulate it. User plugin and self written cron jobs is the function for that.

*tested on a test download flash drive, no plugins...
 

Due to a storm. I currently have my main unraid system off atm. Later I will post my finding. It may be another plugin that gave me the rsync command. But I have had it native to my system since 6.9...

https://linux.die.net/man/1/rsync

I often forget about the unattended driver plugin(I use and have it installed for lazurs mission to recover data) and thing that is a great way to access and connect to the qnap nas.

 

Yes there are multiple docker type systems to complete the rsync task. Other options and a local terminal command would be ideal to rsync/copy from one system mount to another.

atm I can only recommend my default list of plugins:

 

Plugins I recomend for a AMD ryzen build:

 

AMD VENDER RESET
*For amd 660xt / AMD onboard GPU fix
-Driver support

 

CA Auto Update Application
-Mainly for daily plugin updates
-Can auto update select dockers as well

 

CA Mover Tuning
-For parity or file management to move folders between array and poll devices

 

Compose Manger
-Docker Compose addon for unraid docer systems

 

Denyhost Pugin
-ssh and brute force detection

 

Disable Mitigations Settings
-Mainly to see if i have affected board against specter/meltdown
-can be used to disable the mitigation...

 

Dynamix File Manager
-Mainly form FCP needed adon.
-helps with gui side file management outside of terminal mc

 

Dynamix System Information
-The dxdiag for linux and hardware data to confim memory and settings

 

Enhanced Log Viewer
-to add color and help sort the log

 

FCP- Fix Common Problem
- Squid/Unraid help read log and misconfigured settings to prolong Unraids operations.

 

GPU Statistics
-Driver Support
-To monitor cpu temp by checking on onboard GPU
as i PCI pass thorugh my amd 6600xt

 

LXC
-adds the lxd like sytem to run light weight linux vm outside of docker used

 

NerdTool
-Mainly to install screen for terminal command to maintain a script running in background.

 

Parity Check Tuning
-to add more options and set other seting for disk array and its parity to keep a working copy

 

Dlandon Python 3 as some plugin need python

Radeon Top
-the AMD drive for amdgpu

 

snapshots
-for vm/zfs/btrfs/lxc
adds support and backup settigns and options

 

ssh plugin
-adds secoday sshd options in the gui.
-checks openssl version its running and sets other user ssh logins.

 

SwapFile
-adds a 2GB page file to a brtf partion to fix error with some dockers and swap

 

Tips and Tweaks
-adds good documentation to option in Unraid to assit in settings

 

Unassigned Devies And plus
-To add disk that are not apart of the arry to help move data in and out.

 

USB Manager
-to assit 3rd party device passed to VM outside of a running vm using a non passed usb port to pass device.

 

user Script
-if it works in termainl it can be bash scripted here.
-use to run cron jobs

 

ZFS plugins
ZFS Master for Unriad and znapzend for unriad
-additional zfs support to manage datasets

As of writing, while unraid has rsync command avilable there are no plans to make a tool or web gui to use rsync like qnap did.

You will have to script it your self.

In unraid web terminal, you should be able to run command

rsync --help

Edited April 2 by bmartino1