Majyk Oyster
-
Posts
26 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by Majyk Oyster
-
-
Hi folks,
Yesterday my beloved dog stepped on the switch of the power strip of my server.After scratching his ears and giving him a treat to show him who's boss, I restarted the server, and launched a parity check.
I was relived when I got this message :
Event: Unraid Parity-Check Subject: Notice [SAUCISSE] - Parity-Check finished (0 errors) Description: Duration: 1 day, 6 hours, 46 minutes, 53 seconds. Average speed: 144.4 MB/s Importance: warningTo my surprise I got this other message 5 minutes latter :
Event: Parity Check Tuning Subject: [SAUCISSE] Automatic Correcting Parity-Check finished (2414 errors) Description: Elapsed Time 2 day, 3 hr, 56 min, 1 sec, Runtime 2 day, 3 hr, 56 min, 1 sec, Increments 1, Average Speed 85.6 MB/s Importance: alertI fail to understand the difference in error number. Could you help me with that ?
Bonus question : how can I know where/in which files are these 2414 errors ? -
Hi folks
I noticed an error in my unraid logs, appearing every few minutes :May 3 20:03:04 Saucisse kernel: r8169 0000:03:00.0 eth0: Rx ERROR. status = 352cc31c May 3 20:09:00 Saucisse kernel: r8169 0000:03:00.0 eth0: Rx ERROR. status = 352cc37b May 3 20:14:56 Saucisse kernel: r8169 0000:03:00.0 eth0: Rx ERROR. status = 352cc1aa May 3 20:19:44 Saucisse kernel: r8169 0000:03:00.0 eth0: Rx ERROR. status = 352cc07dI tried searching for the few keywords in the messages, all I can tell is it's about the NIC of my server (Realtek r8169).
I have no idea what to do next, any idea ? 🥲
-
I just noticed the exact same message popping in my logs. Allow me to take a seat.
-
On 9/4/2022 at 7:28 PM, kuchta said:
There seems to be a pretty major bug here. I tried to run it and it told me it was going to clear the wrong disk. I looked at the code a little closer and it seems like you opted to continue looking for additional disks that were ready to be cleared instead of breaking out of the loop. Unfortunately, this means that if only one disk is found ready to be cleared, it will think it's the last one in the array since the previously identified disk isn't remembered. I "fixed" this by adding a break command at line 113 (after the "((found++))" command). Oddly, the script still didn't work for me even though it identified the correct device. I get an EOF error right when it is about to unmount the drive. I didn't see any errors in the code to cause this, so I'm wondering if it's a copy-and-paste issue or something. I've attached my log in case it's helpful. For reference, disk7 is the one I was trying to clear and disk8 is the one it incorrectly tried to clear first.
I don't need the script anymore as I just created my own script with the 2 action lines, but wanted to make sure I reported this issue so that no one accidentally clears the wrong drive.
On 9/15/2022 at 4:14 PM, inh said:This script looks good but it still loops through every drive, so it always wants to clear my highest numbered data drive. Very scary. This is on 6.10.3
Edit:
Here is my fixed version:
Thanks for your inputs. Seems like I had a brain fart.
I guess I debugged it when clearing the last disk of my array. I'm updating my previous post.
-
1
-
-
On 8/7/2022 at 2:03 PM, AverageMarcus said:
I made a small update to the `clear_array_drive` script to support setting `status=progress` for unRAID >= 6.10.
I did that and more changes a few does ago, you can see it on the previous page.
There's so many links pointing at RobJ's initial post, that would be nice if there was a way to mention updates.
-
1
-
-
11 hours ago, hernandito said:
Has anyone created a script that renews the LetsEncrypt certificate? Would love to implement.
thank you.
Do you need the certificate updated for the Unraid server or for a hosted VM/container ?
-
When I started replacing my old 3-6TB regular disks with 8 TB NAS ones, shucking was way cheaper. I paid 160€ for my first My Book in early 2020 on westerndigital.com. They're 50% more expensive today.
In the meantime it seams like the price difference between a WD My Book and a WD Red decreased a lot, not it's around 10%.
I bought my last 3 or 4 8TB drives used, for around 100€. Good thing is all of them except one were very lightly used (less then 100 hours). I'm pretty sure it's almost impossible to find used Reds that fresh. Not sure about the future of shucking, we'll see in a couple years when I need to hoard more data.
-
14 minutes ago, JonathanM said:
If the disk is working properly, it won't happen. However... the situations that call for permanent erasure are often when a disk is no longer working reliably. If the software based erasure isn't reliable, the procedure must be escalated to physical destruction. If you like playing with powerful magnets, orderly disassembly is recommended, followed by shattering the platters. If you don't have time for that, 5.65 or larger orifices through the platters is satisfying and quite reliable.
I actually have access to an actual certified disk shredder. I work for a managed security services provider, specialized in hosting critical data (for hospitals, health insurances, ministry of defense...). I haven't worked as a sys admin for a while, but my colleagues in the datacenters can turn my drives into coarse powder if need be.
-
1
-
-
I just watched Spaceinvader One video about the plugin, and I indeed saw that there was an option for multiple cycles (couldn't find it mentioned in the plugin thread).
I don't want to sound like I'm arguing for the sake of it, but I just noticed Squid mentioned on the Preclear plugin thread that it was incompatible with Unraid versions 6.9.0+ and should be uninstalled. So there's that.
As for the final check for the actual destruction of the data, the fact that "successful" shred commands could actually fail to destruct the data beyond recognition didn't occur to me. I'd have to look into that.
-
4 hours ago, JonathanM said:
The existing preclear plugin for Unassigned devices does a good job of erasing drives, with the added benefit of ensuring that the final pass actually was written correctly.
I checked the documentation on that plugin, but if it actually renders the previous data unrecoverable the info eluded me. Zeroing a device doesn't prevent a potentially malicious person from using data recovery tools to obtain potentially private data. The previous magnetic state of the device has to be randomly altered several times to obtain a satisfactory protection.
-
Not sure where to put that info, so there it is : newish (2021-) disks shucked from Western Digital My Book run hotter than older ones. Could be a matter of regular air Vs helium or something.
-
After removing devices from the array, I needed to shred any trace of previous data to sell the HDDs, so I made this.
#!/bin/bash # The purpose of this script is to remove every trace of the data that was once on a device (aka data shredding), so it can be discarded or sold. Multiple devices can be shredded simultaneously (not one after another). # How the script works : # - Using the Unassigned devices plugin, format and mount (if needed) the devices you want to shred. # - If any data remains on the devices, remove it completely. # - Create a "shred-me" folder on each device you want to shred. Any other device will be ignored. # - Launch the script via the "User Scripts" plugin (preferably in background) or via CLI. # - A list of the devices correctly prepared for shredding will be displayed. # - Devices to shred will be unmounted, and won't be remounted after shredding. # - Shredding will then start on all relevant devices simultaneously. Progression will be visible in real time for each device. # - A notification will be sent on completion, using Unraid notification system. # # Version 1.0 - This is a script by user Majyk Oyster on forums.unraid.net. Some elements of RobJ's Clear An unRaid Data Drive were used. ############# # Functions # ############# funDisplay () { echo -e "\n`date +%d/%m" "%T` : $1\n" logger -t$tag "$2" } funMail() { /usr/local/emhttp/webGui/scripts/notify -i normal -s "$1" -d "$2" } ############# # Variables # ############# # Sets the number of passes. 3 is standard, you can increase the value for even safer data removal. passes=3 scriptVersion="1.0" marker="shred-me" tag="shred_unassigned_device" started=0 found=0 wait=60 scriptDir=$(dirname "$0") userscriptstDir="/tmp/user.scripts" # Colors management if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running from User.Scripts GUI. Sets html colors. colorStart="<font color=blue>" colorFinish="</font>" else # running from CLI. Sets shell colors colorStart="\x1b[36;01m" colorFinish="\x1b[39;49;00m" fi ########## # Checks # ########## funDisplay "Shredder v$scriptVersion." funDisplay "The purpose of this script is to remove every trace of the data that was once on a device (aka data shredding), so it can be discarded or sold. Multiple devices can be shredded simultaneously." # checks if a disk is ready for shredding disks=`ls -d /mnt/disks/* 2> /dev/null` devicesToShred="" funDisplay "Checking all unassigned devices..." for disk in $disks; do itemsList=`ls -A $disk 2> /dev/null` itemsNumber=`echo "$itemsList" | wc -l` # test for marker and emptiness if [ $itemsNumber -eq 1 -a "$itemsList" == "$marker" ]; then itemsSize=`du -s $disk | awk '{print $1}'` if [ $itemsSize -eq 0 ]; then disksToShred="$disksToShred $disk" device=`df | grep "$disk" | awk '{print $1}' | sed 's/.$//'` devicesToShred="$devicesToShred $device" ((found++)) fi fi done # No drive or multiple drives found to clear if [ $found -eq 0 ] || [ -z "${devicesToShred// }" ]; then funDisplay "Did not find an empty drive ready and marked for shredding. To use this script, the drive must be completely empty first, no files or folders left on it. Then a single folder should be created on it with the name 'shred-me', exactly 8 characters, 7 lowercase and 1 hyphen. This script is only for safely removing all data from unassigned devices, typically after removing them from the array." exit else funDisplay "Device(s) found : $colorStart $devicesToShred $colorFinish" fi ############ # Warnings # ############ funDisplay "Found marked and empty drive(s) to shred. \t* Device(s) will be unmounted first. \t* Then random data will be written to the entire device(s) $passes time(s). \t* When complete, all data from device(s) will be cleanly erased." echo -e "\n Commands to be executed:" for diskToShred in $disksToShred; do echo -e "\t* $colorStart umount $diskToShred $colorFinish" done for deviceToShred in $devicesToShred; do echo -e "\t* $colorStart shred -vf $deviceToShred $colorFinish" done if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running in User.Scripts funDisplay "You have $wait seconds to cancel this script (click the red X, top right)\n" sleep $wait else #running from CLI funDisplay "Press ! to proceed. Any other key aborts, with no changes made. " ch="" read -n 1 ch echo -e -n "\r \r" if [ "$ch" != "!" ]; then exit fi fi ############# # Preparing # ############# funDisplay "Unmounting device(s)... \r" for diskToShred in $disksToShred; do if `umount $diskToShred`; then # unmount success funDisplay "Disk $diskToShred unmounted successfully." else # unmount failure funDisplay "Disk $diskToShred unmount failed. Exiting." exit fi done ############# # Shredding # ############# funDisplay "Shredding Device(s)..." startTime="`date +%c`" if [ ! -z `cat $userscriptstDir/tmpScripts/*/log.txt 2> /dev/null | grep "Shredding unassigned devices"`]; then logFile="ls $userscriptstDir/tmpScripts/*/log.txt" else logFile="/tmp/shredLog.txt" touch $logFile fi for deviceToShred in $devicesToShred; do shred -vf $deviceToShred | tee -a $logFile & done wait endTime="`date +%c`" ######## # Done # ######## funDisplay "Shredding is complete" funDisplay "Started @ $startTime\nFinished @ $endTime" funDisplay "Unless errors appeared, the device(s) is/are now shredded!" funMail "Shredding complete on device(s) $devicesToShred." "Shredding complete on following device(s): $devicesToShred Started @ $startTime Finished @ $endTime "Your devices to shred should look something like that before using the script (formated, mounted) :
Here's how the script looks like in User Scripts :
-
On 7/11/2022 at 2:27 AM, Darksurf said:
What is up with the zero drive script? It immediately gives up.
Check my post above, I think it might help.
-
Hi there.
I needed to use RobJ's "Clear An unRaid Data Drive" script, and noticed it wasn't compatible with Unraid 6.10 (version verification checked only 1 digit). So I dived into the script and found a few things I could update or do differently.
Since RobJ doesn't seem to be around anymore, I took the liberty to rewrite most of it.
Change log :
Version check adapted for Unraid 6.10 and up.
Check if unmount successful before clearing.
Send mail notification when clearing finished.
Script adapted for more then 9 drives.
Dead code removed.
Code simplified and optimized.
Ambiguous variables renamed.
Progression of the dd command sent to the GUI logError message more explicit if more than 1 drive is prepared for clearing.
#!/bin/bash # A script to clear an unRAID array drive. It first checks the drive is completely empty, # except for a marker indicating that the user desires to clear the drive. The marker is # that the drive is completely empty except for a single folder named 'clear-me'. # # Array must be started, and drive mounted. There's no other way to verify it's empty. # Without knowing which file system it's formatted with, I can't mount it. # # Quick way to prep drive: format with ReiserFS, then add 'clear-me' folder. # # 1.0first draft # 1.1add logging, improve comments # 1.2adapt for User.Scripts, extend wait to 60 seconds # 1.3add progress display; confirm by key (no wait) if standalone; fix logger # 1.4only add progress display if unRAID version >= 6.2 # 2.0 - This is an update/fork of RobJ script by user Majyk Oyster on forums.unraid.net. #Change log : #Version check adapted for Unraid 6.10 and up. #Check if unmount successful before clearing. #Send mail notification when clearing finished. #Script adapted for more then 9 drives. #Dead code removed. #Code simplified and optimized. #Ambiguous variables renamed. #Progression of the dd command sent to the GUI log # 2.1 - Fix to clear the found disk ############# # Functions # ############# funDisplay () { echo -e "\n`date +%d/%m" "%T` : $1\n" logger -t$tag "$2" } funMail() { /usr/local/emhttp/webGui/scripts/notify -i normal -s "$1" -d "$2" } ############# # Variables # ############# scriptVersion="2.0" marker="clear-me" tag="clear_array_drive" started=0 ddArg="" #used for dd command found=0 wait=60 scriptDir=$(dirname "$0") userscriptstDir="/tmp/user.scripts" # Colors management if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running from User.Scripts GUI. Sets html colors. colorStart="<font color=blue>" colorFinish="</font>" else # running from CLI. Sets shell colors colorStart="\x1b[36;01m" colorFinish="\x1b[39;49;00m" fi ########## # Checks # ########## funDisplay "Clear an unRAID array data drive v$scriptVersion." # check unRAID version unraidVersion=`cat /etc/unraid-version | cut -d '"' -f 2` majorVersion=`echo $unraidVersion | cut -d "." -f 1` minorVersion=`echo $unraidVersion | cut -d "." -f 2` if [ $majorVersion -eq 6 -a $minorVersion -ge 2 ]; then ddArg="status=progress" else funDisplay "This script was not validated for this version of Unraid ($majorVersion.$minorVersion.x)." if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running in User.Scripts funDisplay "You have $wait seconds to cancel this script (click the red X, top right)\n" sleep $wait else #running from CLI echo "Press ! to proceed. Any other key aborts, with no changes made. " ch="" read -n 1 ch echo -e -n "\r \r" if [ "$ch" != "!" ]; then exit fi fi fi # checks if array is started disks=`ls -d /mnt/* | grep disk | grep -v disks` drivesNumber=`echo "$disks" | wc -l` if [ $drivesNumber == 0 ]; then funDisplay "ERROR: Array must be started before using this script. Exiting." exit fi # checks if a disk is ready for clearing funDisplay "Checking all array data drives (may need to spin them up)..." for disk in $disks; do itemsList=`ls -A $disk 2> /dev/null` itemsNumber=`echo "$itemsList" | wc -l` # test for marker and emptiness if [ $itemsNumber -eq 1 -a "$itemsList" == "$marker" ]; then itemsSize=`du -s $disk | awk '{print $1}'` if [ $itemsSize -eq 0 ]; then foundDisk=$disk ((found++)) fi fi done # No drive or multiple drives found to clear if [ $found -eq 0 ]; then funDisplay "Checked $drivesNumber drives, did not find an empty drive ready and marked for clearing! To use this script, the drive must be completely empty first, no files or folders left on it. Then a single folder should be created on it with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen. This script is only for clearing unRAID data drives, in preparation for removing them from the array. It does not add a Preclear signature." exit elif [ $found -ge 2 ]; then funDisplay "Checked $drivesNumber drives, found multiple empty drives ready and marked for clearing! To use this script, the drive must be completely empty first, no files or folders left on it. Then a single folder should be created on it with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen. This script is only for clearing unRAID data drives, in preparation for removing them from the array. It does not add a Preclear signature." exit else disk=$foundDisk deviceNumber=`echo $disk | cut -d "/" -f 3 | cut -c 5- ` device="/dev/md$deviceNumber" funDisplay "Device found : $colorStart $device $colorFinish" fi ############ # Warnings # ############ # First, warn about the clearing, and give them a chance to abort funDisplay "Found a marked and empty drive to clear: $colorStart Disk $deviceNumber $colorFinish ( $device ) * Disk $disk will be unmounted first. * Then zeroes will be written to the entire drive. * Parity will be preserved throughout. * Clearing while updating Parity takes a VERY long time! * The progress of the clearing will not be visible until it's done! * When complete, Disk $disk will be ready for removal from array. * Commands to be executed: \t* $colorStart umount $disk $colorFinish \t* $colorStart dd bs=1M if=/dev/zero of=$device $ddArg $colorFinish" if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running in User.Scripts funDisplay "You have $wait seconds to cancel this script (click the red X, top right)\n" sleep $wait else #running from CLI funDisplay "Press ! to proceed. Any other key aborts, with no changes made. " ch="" read -n 1 ch echo -e -n "\r \r" if [ "$ch" != "!" ]; then exit fi fi ############# # Preparing # ############# funDisplay "Unmounting $disk ... \r" funDisplay "Unmounting Disk $disk (command: umount $disk ) ..." if `umount $disk`; then # unmount success funDisplay "Disk $disk unmounted successfully." else # unmount failure funDisplay "Disk $disk unmount failed. Exiting." exit fi ############ # Clearing # ############ funDisplay "Clearing Disk $disk (command: dd bs=1M if=/dev/zero of=$device $ddArg ) ..." startTime="`date +%c`" if [ ! -z `cat $userscriptstDir/tmpScripts/*/log.txt 2> /dev/null | grep "Clear an unRAID array data drive"`]; then logFile="ls $userscriptstDir/tmpScripts/*/log.txt" dd bs=1M if=/dev/zero of=$device $ddArg >> $logFile else dd bs=1M if=/dev/zero of=$device $ddArg fi endTime="`date +%c`" ######## # Done # ######## funDisplay "Clearing Disk $disk is complete" funDisplay "Started @ $startTime\nFinished @ $endTime" echo -e "A message saying \"error writing ... no space left\" is expected, NOT an error. Unless errors appeared, the drive is now cleared! Because the drive is now unmountable, the array should be stopped, and the drive removed (or reformatted)." funMail "Clearing complete on $device." "Clearing complete on $device.\nStarted @ $startTime\nFinished @ $endTime"-
1
-
2
-
-
tl;dr : If you entend to reuse a WD My Book enclosure that's less than 2 year old (approximatively), maybe don't do it.
I've looked around for the info, but couldn't find it, so I'll share.
I've shucked several WD My Book hard drives in the past, and found it useful to reuse a couple of the enclosures with older WD HDs or other brands.
The trick, because WD had the brilliant idea to put a hardware lock, is it break 2 pins on a Winbond chip.
So today I got 2 new 8TB My Book, and thought I'd do it again to house a 4TB HD. Not only did it now work, but the enclosure doesn't work anymore.
As far as I know, here's the info I got to tell the old and new versions appart :
Functioning enclosures after the mod were all "WDC_WD80EZAZ-11TDBA0".
They use an Asmedia controller.
The new, non reusable, enclosures are "WDC_WD80EDBZ-11B0ZA0".
They use a JMS579 controller.
-
You can use PSU Calculator to make sure, but both those PSU should be fine. Hard drives don't need that much power.
-
1 hour ago, trurl said:
Is this a move or a copy? Moving of course is also writing the source disk to delete the files.
Both seem to be just as fast while the data is written. I guess the extra work needed for a move is done once the transfer in completed, and the source file deleted.
1 hour ago, johnnie.black said:Is this data to data disk, or cache to data? Like mentioned data to data disk will always be slower.
It's data to data (/mnt/diskA > /mnt/diskB/). I just witnessed such a transfer at 80MB/s, so the aforementioned fluctuation is indeed confirmed.
-
As the parity is done rebuilding and the array is protected once again, I checked various things :
- Reads to the array from ethernet are stable at 120MB/s (which is pretty nice).
- Writes from ethernet to cached shares on the array are stable at 80MB/s.
- Writes from ethernet to non cached shares on the array are stable at 80MB/s.
- Disk to disk transfers within the array start around 120MB/S and slowly stabilize around 50MB/s.
I've double checked, the files indeed land on /mnt/cache/ or /mnt/diskX/ as they're supposed to. I'm not sure how to simulate the varying speed of parity reconstructing writes, depending on the disks reading data and the position of the data on the platters.
That makes me wonder : are my array disks too fast to allow my cache (SSD) to make any difference on array writes ? The wiki page might be a little outdated on that subject, since mentioned speed and sizes are those of 2011 hardware.
-
1 minute ago, itimpi said:
Did you read the link I gave earlier in the thread? Once you start getting complete disk rotations as part of a write operation that is very much a limiting factor.
Absolutely, every word of it. I only used the Read/Modify/Write mode for a few tests, my array is configured with turbo mode. Disk rotation wait shouldn't be an issue (?).
As I understand things, considering Diskspeed results and the way turbo mode works, the slowest writing speed would be around 60MB/s when reading from the end of my 3TB WD Green disk to reconstruct parity, but it also could be much better when writing past the "half" of my parity drive, which means reading only from faster disks. So I don't get why I never experienced better speeds since I activated parity. I could still be missing something though.
-
Thanks for the details, much appreciated. Seems like a wasted a bunch of time and a perfectly good parity then. 😁
The WD Greens can definitely max out that 60MB/s "hard limit", so I guess it's pointless replacing them while using parity. I'd still love to know exactly what's the limiting factor in all this, but that may be Unraid secret sauce.
On a side note, I shrunk my VM and docker.img, so now I have under 3,5GB used space on the cache, and it's also freed of all automated writes. That should let it breeze a little.
-
18 minutes ago, johnnie.black said:
Do you mean from one array drive to another? That will always be slow, and write mode will auto change from reconstruct to r/m/w.
Exactly, disk to disk within the array using MC or another CLI command.
I can't find any reference of what to expect performance-wise on average/enthusiast/non-pro hardware, so I have a hard time knowing what's normal and what could be improved. Troubleshooting something that's not broken can be hard sometimes. 🤓
-
1 hour ago, johnnie.black said:
With reconstruct write enable you should get write speeds around the max speed of the slowest disk in use at any point, so if you were writing to an empty disk max speed should be >100MB/s, if you were writing for example to one disk with about 2.8/2.9TB used speed should be around 60MB/s, since it would be limited by the 3TB WD green, other than that you can have one or more disks with slow sector zones, was the parity check speed always normal, without slow periods? If you have dynamix stats installed you can check the graph (before rebooting).
Thanks for the input. That's closer to what I first imagined.
I just installed Dynamix stats, so there's not much data to inspect right now. But I took a closer look at the parity sync (still running). It just went over 50% (4TB) while I was watching, which is where WD Greens are not used anymore, and only the Reds and the Ironwolf are left. The speed instantly went from 70MB/s to 160MB/s. During the first hours, the speed was around 110-120MB/s. The last time I made a parity sync, it ended with a 120MB/s average speed. So everything looks normal and coherent with the Diskspeed results so far, regarding parity sync at least.
Before :
After :
However, writing from a Red disk to another, using read/modify/write mode so the Greens don't slow everything done, shouldn't I get better performance than 40-60MB/s ? It didn't seem to make any difference when I tried yesterday. I did quite a bunch of tests with disk to disk transfers before removing the parity, I doubt they where all impacted with the WD Green end of platter speeds.
Replacing those WD Greens with more Reds (shucked white labels actually, don't tell WD) is very tempting indeed. I've added 3x8TB drives to the server in the last few weeks (and retired a pile of 1-3TB disks), so my computer budget is kinda burned right now. I might try to replace the Greens with old but functional 7200rpm drives that sit on my desk collecting dust, that could make things better I guess (unless I find it's pointless in the meantime).
1 hour ago, Squid said:@johnnie.black Not always Because the kernel does delayed writes to the physical device, if you have a few drives, and your allocation mode is set to "Most Free", the files all go to different drives.
After a bit of tinkering, I only use high-water now. Thanks for the input nonetheless !
-
Thanks for your replies.
I expected some overhead, but I had no idea how much since I have 0 experience with software raid-like solutions. I couldn't find any infos to roughly quantify real life performances. I'm still moving/sorting lots of data (disk to disk) to make my server nice and tidy, which would bypass the cache. That kind of transfer should diminish drastically soon.
I'm still somewhat confused : since the CPU, RAM and bandwidth are all far from saturation, what would the typical limiting factor ? Could I do anything to slightly improve things ?
32 minutes ago, trurl said:I know you aren't complaining about this, but Mover is intended for idle time. You might find it better to run mover less often and not cache some of your user shares. Do you need cache speed for all writes?
I cache very little, since most writes to my server are scheduled backups and queued downloads, so I am not waiting on them to complete anyway.
The only reason I set the mover hourly is because I used the small 120GB SSDs I had laying around, and they fill up pretty quickly. Ideally I would switch to 480GB-960GB SSDs and use the default nightly mover setting.
I never enabled the cache on shares containing huge files (20GB+). Disabling it for all shares used mainly for automated transfers is indeed something to consider, thanks for the input.
35 minutes ago, trurl said:Haven't looked at the diagnostics. Are you actually having any problems other than the expected slower parity writes?
Not that I can tell, the wiki and the forum helped me with every issue I had so far.
-
Hi folks,
I've been trying Unraid for the past month (1 hour remaining on my trial actually), and I'm quite pleased with it so far. Since I'm impatient, have proper backups and had my future parity drive stuck somewhere due to COVID-19, I first set everything up without parity. All was fine, transfer speed within the server was around 150-180 MB/s on average.
Last Monday, the parity drive finally arrived. I precleared it (I get it's not necessary anymore, point was avoiding premature failure issues), set the disk as the new parity, and let Unraid do it's job for the next 17 hours (for a 8TB drive). I then tuned everything up according to the wiki, Spaceinvader One amazing videos and this very forum.
Everything seemed to work fine, except files transfers between disks or to the shares slowed down a whole lot. They now start around 120 MB/s, and usually stabilize around 40-60 MB/s a few minutes later (using MC or any CLI tool).
Time: 0:00.08 ETA 0:07.05 (164.92 MB/s) Time: 0:00.15 ETA 0:10.05 (114.67 MB/s) Time: 0:00.24 ETA 0:11.30 (99.59 MB/s) Time: 0:01.22 ETA 0:13.19 (80.77 MB/s) Time: 0:07.25 ETA 0:09.21 (70.66 MB/s) ...
The rig :
Intel i5 3570k (4 cores)
Gigabyte H77N-WIFI
8 GB DDR3
Dell H200 controller (flashed in IT mode, fresh thermal paste and brand new Noctua fan. Heat sink is barely warm)
It seems more than capable for my needs, considering the minimum specs in the documentation. I barely use anything except the NAS features. I only run a very small Debian VM, to have a secured interface between Unraid and my distant servers. It does nothing 90% of the time, and was disabled during troubleshooting. I don't run any Docker container, except DiskSpeed during the benchmarks.
The disks :
Note : still waiting for an extra SATA power cable to plug my second SSD and set a RAID1 for cache. The mover runs hourly, so the limited capacity is sufficient for my needs.
Troubleshooting so far :
First I benchmarked the disks, which doesn't show anything special to my knowledge. No controller induced bottle-neck.
The 2 old WD Green drives are obviously slower, so I though that maybe reconstruct write for the parity wasn't optimal, since any write on the array makes the other disks read. Switching to read/modify/write made no difference, so I went back.
I re-wired everything in the server, checked the controller cooling, still no cookie.
The only issue I can see in the syslog is a seemingly random NTP drift, but I doubt it's relevant.
In a desperate move, I unassigned the parity drive, which immediately fixed the speed issue. Parity-Sync is running as I write to revert that bold move.
I've attached the diagnostics. Sorry the log is so short, I've rebooted the server a few dozens times in the last couple days. If necessary I'll let it run for a few days and upload it again.
Any help would be appreciated, I've no clue what to do next. 🤓
Trouble interpreting diverging error count between "Parity check" and "Automatic Correcting Parity-Check"
in General Support
Posted
Thanks for the replies ! 🍻
Here's what the history looks like :
I'm running another parity check right now, we'll see.
I have no idea why the elapsed time differ, looks like they run at the same time or something. Can't tell which would be right.
I only have backups of a small portion off the array, the critical stuff. I'm going to check this, and hope for the best for the media library.
I'm indeed using XFS, I'm looking into the File Integrity plugin right now.