Falcosc

Yes, either manually send the magic package or use this to wake your server on any request: https://github.com/nikp123/wake-on-arp Don't forget the g WOL flag in the sleep plugin setting

@Frank1940 Thank you so much, now it starts to make sense why there is a gap in the documentation. I did feel stupid because I did not find anything. But I couldn't find an obvious reason for that. And it didn't help that any parity check issue was related to hardware issues, and nobody did ask what happen if parity check errors occur on healthy hardware. OK, that means, we can use SMART if we don't believe bit rot Or maybe go even further, maybe SMART is able to detect bit-rot as well during read? My issue is that I don't know how powerful SMART actually is. Is there a checksum on HDD which makes it possible that the HDD can detect single bit errors during read? I don't care about multi bit errors, my data is not so important to talk about edge cases, for me, it is good enough to have a guideline how to handle something which is part of my monitoring. But I want to know if it is enough to trust SMART for usual use cases (people with single parity drives) If we can trust SMART, I would formulate the guide as following: Parity check found an error: no SMART counter did raise: unclean shutdown or Hardware is broken (ram, pci, controller, cables etc) SMART Counter did only increase on data disk: replace data disk SMART Counter did only increase on Parity Disk: replace parity disk SMART Counter did increase on multiple disks: we don't know which data is valid anymore new: this is very unlikely and should be investigated by the community either validate the data by using file system features or 3rd party apps replace all disks with warnings and restore backup Would you agree to this SMART based guide of how to deal with parity check results? Or did I make some wrong assumptions?

Yes, but how? We have Backup files to compare with the effected files. But how do I find which files are effected by the parity check result? It is not the solution to format the whole array and restore multiple terabytes of backup if you encounter your first parity check error. I want to know how to find out what needs to be done. How to check what disk is broken? If you know what disk is broken, then it is easy: Parity Disk is wrong (memory fault, unclean shutdown, other reasons): use parity check with correct Data disk is wrong: restore backup My question is: how to decide between parity check with correct bit or restoring backup. And if restore backup is the answer, do I need to restore the whole backup? Or just the broken file. I would like to read or create a documentation about how to deal with parity check results. - first of all check your system stability stuff (we don't need to talk about the first point, we have good pages for that) Then find the broken sector (this documentation is missing) - On File System A do this - On File System B do that - On File System C you can not do this and need the file integrity plugin Or is there a trick which I am missing which does bypass the whole file system analyses? Because it is strange that this important information how to determine what to be done after finding a parity check error is missing. I don't understand why there is an uppercase if. Could it be that modern hard disks are nearly flawless (flawless in terms of every issue is covered by smart) and you can skip all the file system checks and only do something if SMART did detect the error? If they were flawless, would the process look like this: Parity check found an error: - no SMART counter raises: parity data is broken, fix system and rebuild parity - SMART counters did raise: data disk is broken, replace disk and use the parity data, because parity is correct based on 0 SMART warnings on the parity drive I don't know how powerful SMART is, or if there are other things which tells us what to do on a parity error. Would be cool if HDD would have checksums on hardware level and if these are reliably reported in SMART. Or If I rephrase the "why if" question, why is the file integrity check optional? Is there a functionality somewhere which makes us comfortable to skip these things?

Yes, I know. But could we find the file which is using the sector on any data disk to validate the file on any data disk? 3 of 5 drives have a file we only need to validate the 3 files. If the 3 files are ok the error did happen on the drives which don't have data or on the parity drive it self. That means you can rebuild the parity. If the file is invalid, you have to fix it by fixing the cause of the issue (replacing hardware) and then you can fix the data and after the data is valid again rebuild the parity drive. But I don't know how to check all data drives for data corruption. We have this page: https://wiki.unraid.net/Check_Disk_Filesystem But it does only cover file system checks, is this enough? Or do some mentioned file systems don't support checksums for files? For me there is a gap between this file system documentation and how to treat parity check errors. I would like to know how to deal with monitoring tool results before I encounter them. For me it is pointless to monitor something if I don't know how to handle the result of parity check correctly

What is the best way to find out if either parity drive has to be overwritten or if the unlikely use case happens and the parity data is the correct one? I mean, yes, you can start a validation of all Data from all data disks, but this isn't very easy. And it does take a long time. Is there a way to find out which files are stored on all disk at a specific sector (we have the sector definition from the parity check result) It is cool that we have the parity check, I guess we are just missing documentation about how to handle parity check errors correctly. The coolest solution would be a selective data validation based on the invalid parity bit. Is that possible? That would be great and very fast. It maybe even useable if you don't have checksums because you may could extract the file and compare it with the backup. I don't know about the structure of the file systems so I don't know how to find data references pointing to the effected sector.

Why SMART? Why do I care about SMART? I don't like waiting for Data loss to get noticed, and I don't want to execute a Parity Check every day. So I like to monitor SMART to may detect something before the parity check gets executed. And for me as a private person, restoring backups is an inconvenient and slow process with a lot of manual tasks. It doesn't make sense to spend effort to make the backup restore more convenient. Or maybe it is just less interesting than spending time with SMART So I like to try to close the gap between the parity checks with smart monitoring to notice some issues before the be-monthly parity check. Feedback needed Did anybody already document or talked about this topic? I didn't find a procedure, so I thought about it and based on my limited knowledge I made some assumptions and created the following process. Could you give me feedback about these steps, and in best case point out where I had made wrong assumptions? That would be great. Maybe we can use the discussion result to share a documentation for people how want to add SMART data in their monitoring process. My Process check SMART notification (is a count change of some key fields part of email notifications?) SMART counter change is critical: New Pending Sectors or other critical indicators replace Disk immediately (replacement disk was precleaned in the past, that is good enough for this case, no time left for an additional preclean validation) meanwhile, preclean the broken disk to see if the counter recovers or if the count is stable to reuse the disk (I never had a pending sector disk which did pass the stress test) SMART counter change is a warning: new relocated sectors start a parity check nocorrect as stress test while the check is running, start preclean stress test on the replacement disk again to check if it is still healthy since the last preclean repeat the parity check if relocated sector does not change to get up to 3 time to get 3 read operations to all sectors Parity Check result: Relocated Sector Count did not change on all 3 parity checks (read only tests) don't replace the disk if it was the first time if the disk had that same issue in the past, replace it to start preclean write stress test on the problematic data disk Parity check result: Relocated Sector count did raise at any of the parity check runs disk has a persistent issue, replace it with a precleaned replacement disk Preclean of the replacement disk did create new relocated sectors select a different replacement disk and make one preclean run to validate it (even if it already was precleaned in the past) meanwhile, stress test the suspicious replacement disk with 5 preclean runs to check if it needs to be thrown away (because this takes long, you should select a different replacement disk to fix the array)

Do we have a guide which is telling something about how to fix partiy check errors? I think about some ways of identifying which data disk is causing the mismatch in the parity. Depending on which file system you use, the guide would look differently. I did already think about it and have one major issue: What if the error is caused by a bit flip in an area of a data disk which has no data in this sector? File based check will not find this issue, so all file checksums of all data disks would be OK and you would not know at which disk did cause the error.

I have the same question. How to detect what data is correct and how to correct? Maybe we can start easy: Is it able to extract both version of the mismatching data block for all data disks? And if you find tools to validate the result, how to decide in which way the array should be reconstructed? 2nd Question: why is there a "Correct" Flag implemented in the parity check? Who would want to blindly overwrite the parity drive data? https://wiki.unraid.net/Parity#Checking_parity I would guess that the process is to resolve the issue on the data disk and then rebuild the parity with the correct flag. But this would only be possible if the parity check does tell at which data disk the issue did Accor, which is not possible without checking the data block on all connected data disks. My conclusion: if we don't know how to correctly solve a parity check result (blindly changing parity drive data is not a solution) I don't understand what the point of the parity check is.

Beside the intended way, I did actually find another use-case for MAX LBA. I do buy broken HDDs which only have relocated sectors and no real issues. Then I do 5 full formatting runs to check if the number stays stable in a 48h stress test. But one of 3 Disks was actually broken. During the first run, I did discover the place where the sectors get relocated by monitoring the formatting progress and monitoring at which point in time new sector relocations did happen. I did add timestamps to windows format tool output with PowerShell write-output "exfat" "j" | format J: /FS:exfat | Foreach{if ($lastline -ne $_){"{0} - {1}" -f (Get-Date),$_}; $lastline = $_} And I did use cristaldisk info with option->Autorefresh: 1min At CrystalDiskInfo8_12_4\Smart\ST4000VN008-2DR166ZM4105DY\ReallocatedSectorsCount.csv you will find the sector count change timestamps By comparing the timestamps, you can calculate at which place on the Disk the errors happen. Then I did create a healthy partition in front of the error to be able to create a tiny 100gb partition directly over the broken part. Formatting the broken space is way faster, and then I did identify that the disk had real trouble with this section, and each run did create new relocations. After the 3rd run, it was clear that this part is physically broken. Be careful with formatting the broken part of the disk, the relocated sector count has a limit monitored in % in the smart table. It did go from 99% to 87% in a half hour. So remember to stop the process as soon as you feel confident that this is an unrecoverable problem. At the end of the story I did use max LBA to exclude the broken part of the disk since it was only effecting the very last 100GB. After doing this I have a 3,9TB HDD which is already at the 7th full formatting path. Using the windows tool to have one write 0 pass and 2 write random passes format J: /FS:exfat /P:3 I don't recommend putting more than one of these questionable disks in your array, but since I have backups, I am happy to experiment how unraid does handle real hard disk failures by raising my array risk factor with drives saved from landfill. I hope this does offset the power used for the 48hour stress test and all the unnecessary power on hours during unraid array feature experimentation sessions.

The cool thing about https://wiki.unraid.net/The_parity_swap_procedure is that you don't have to switch the parity drive today. You can just wait for the first failure and swap parity and data by copying the parity drive. Max LBA changes are supported, but you have to extract the count from the old drive and can not directly calculate it based on the displayed size because there is some overhead. And if you use Seatool GUI you do enter the index, so 1953525167 need to be used there Old drive: May 27 04:36:22 unraid emhttpd: ST1000VX_000-1ES162_Z4Y2FLNG-0:0 (sdc) 512 1953525168 new drive: May 27 08:40:26 unraid emhttpd: ST4000VN_008-2DR166_WDH1TJTV-0:0 (sdc) 512 1953525168 After rebuild, I will try the party swap procedure ... Success, the documentation does still work. But I didn't do data integrity tests, I can only tell that the array was green after the process.

Thanks, so this is the part which I thought has to be done manually by hand: Cool that you have a button for that usecase: I did just not think about the possibility to use the old parity drive as new data drive. I only concluded this after I was phrasing my response to MAM59. I will try this procedure with my test setup.

That is exactly what I need, I couldn't find that. And MAM59 did even tell us that it is impossible to do that. Do you have a link to the documentation? By the way, you guessed the size of my spare drive I have an array of 2TB drives with an 3TB Parity drive

Thanks for clarification, so instead of doing very inconvenient unsupported manual steps, it is just impossible to use a larger disk on a data drive fail. So anybody should have a spare disk on hand which is smaller or equal to the parity drive. I want to talk about the use case to convert an unfitting disk to align with that requirement. I treat max LBA just as a way for permanently converting a HDD to a smaller size Removing LBA is the same process as physical replacing the HDD, that should be clear I don't want to sell LBA for this, no I just need the permanent convention to the smaller size Maybe I will upgrade my parity drive in 5-10 years, but this thread was created to find a solution for the worst case scenario where a data drive does fail before I plan to upgrade the array size. My test rig is running. I'm glad that I did select two old 1 TB drives for testing because I forget that I need to wait for a full rebuild of the parity drive before the test array can be used. If your server setup is very easy to maintain, you could just use the larger spare drive to replace the parity disk today and use the old parity disk as spare for the data disks.

Thank you, I will test this on my rig and share the results. Why isn't this a common issue? I thought if you mix WD, Seagate or Toshiba with the same advertised size, I would expect real size differences in the kb or mb range. Or does unraid some clever under provisioning to avoid this mix vendor size difference issues?

Yes, I understood the issue of BIOS related limits. I would like to evaluate seagates solution which is written directly to the drive controller but don't have an unraid server on hand to test it directly in unraid. Could I do some test without unraid by checking how linux does see my max LBA seagate drive? I would need to know which parameters are used by unraid On my unraid server I don't have empty SATA ports to test it. I could use a USB to SATA controller to plug my too large seagate drive into my array, but my licence does not allow additional drives. And I don't have a spare testbench with 3 drives to test it outside of my unraid server. So I need some help with the evaluation of the seagate max LBA feature. I could use my main computer for testing unraid support for seagate max LBA, but I have some limits on my main rig: - is it possible to create an array with 2 drives? 1 data + 1 parity? - I only have one external SATA connector, so I would need to connect the 2nd drive via USB: are USB 3.0 disks supported? If both is supported by unraid, I could test HDD swap with to large drives using seagate seatool and I could share the results. If seagates feature is compatible, this would be great. Not only for my usecase, but also for use cases with mixed vendors where the drive size is only off by some megabytes

Thank you for your quick response. Cool, that sounds promising, because somebody already tested a drive assignment based on an altered max drive size. That makes me confident that this could work. Unfortunately, I don't have a gigabyte board. But if seagates solution is implemented similar like HPA, it would be more reliable because it is set on the hard disk and not on the bios I could test Seagate max LBA by manually test how the drive does response to my requests: Which values/parameters are queried by unraid?

I use multiple drives with different sizes in my array. My spare drive is larger than any other drive in my array. I don't want to upgrade my parity drive size because if the parity drive fails I don't want to be forced to buy an unnecessary large replacement drive. If any of my drives fail, I know about the issue that I can not use a replacement drive which is larger than the parity drive. Which means in case of a data drive failure, I would need to replace 2 drives: update the parity drive to allow larger data drives replace the broken data drive Because you can not replace the parity drive in a degraded array so easily, I want to know more about the issues around too large hard drives before I have to deal with this issue. So I have multiple questions: Is it possible to assign partitions instead of disks to work around the drive too large issue? I have a Seagate drive, is the max LBA feature supported by unraid? https://www.seagate.com/manuals/software/seatools-bootable/using-seatools/#set-max If you are not sure about max LBA support, which hard disk parameters are queried during the disk size check? I could try to verify if these parameters get altered after I define a max LBA via seatools

I was not sure about the state of my Disk because I did not remember if I hit acknowledge half a year ago, and it did degrade since then or if I did not hit acknowledge in the past. Because the UI does not tell you the stored values from the last acknowledgement, I did search for the config file. This topic was listed in the Google search, so I will provide the answer here: /boot/config/plugins/dynamix/monitor.ini this file tells you the old values and helps you to decide if your disk did degrade significantly or just had a hick up. I recommend making a copy of this file with the current timestamp because It does not tell you at which date these values got recorded and the extended smart error log does only save errors.

@TraumG kannst du mal schauen, ob du Ähnlichkeit zwischen unseren Systemen findest, um den Unraid Jungs mehr Daten zu geben? Dort im Bugticket findest du mein lspci Output. Hab den Hack als Workaround in den unraid Startup Scripten eingebaut. Falls du das auch machen magst, kann ich dir das auf Anfrage zusammenschreiben.

It was a bit tricky to remove the bottleneck because it already was tuned to process sequentially with one thread per disk. The issue was around expensive binary calls with a lot of syscalls like file stat, get file attribute, unnecessary subshells for each file and so on. After fixing these I found all the other issues like unwanted parallel IO caused by monitoring. I did also experiment with offloading the really needed but still expensive syscalls into asynchronous subshells. But XFS isn't good enough to handle asynchronous file metadata actions without hurting spinning disk IO too much. If XFS metadata were completely cached, then you would reach nearly peak disk read performance on small files (did tests without Metadata, which did achieve this) But check export does skip all these things because it doesn't need to read or write file attributes if the exported hash matches. For that reason, this process is the fastest one. It is basically directly using the hash list cli argument and only starts custom error handling in case the cli response stream contains verification errors.

Actually you can't see how fast blake3 is because a single thread can handle 4-6GB/s which is way beyond spinning disk performance. So, we configured to run single threaded to be more efficient. The main goal off introducing blake3 was to reduce CPU load. I did benchmark memory and thread settings to get the best small file performance. The biggest issue was the amount of syscalls in the old script. But we had other issues which did slow down spinning disk IO as well. - added blake3 hash support for 2-6 times more hash rate than blake2 - reduced CPU load up to 4 times at same data rate when using blake3 - improve build, verify and check speed for all hash methods on small files - fixed stacking find commands avoiding clean shutdowns while watching control page - fixed starting multiple build status checks while watching control page - added monitor background verify processes and all manual bunker script executions - fixed rare process status detection bug "Operation aborted" - fixed file name truncation if key is broken - inline help for disk table and commands more accessible - fixed multi-language support for buttons - added watch error and warning messages of running or finished progress - added Disk status icon for running build status check

It could be possible that we still need a broken disk2.tmp.end file in case the bug is not obvious. (I worry that the bug is not obvious because at least one test run with simulated corruptions didn't have this issue)

That is the reason why I would like to have the file. I test it once and didn't see any issue. Edit it as much as you like. In the best case, you can reproduce the issue with just normal characters and only 3-5 lines in your .end file

Yea, you can remove your personal stuff and save to see if the issue persists with your modified file

Please share your /tmp/*disk2* files and /var/tmp/*disk2* files. I did add errors to them, and it looks like there is at least on layout function which does not ignore this new content. Is a disk2 process running? Because you should not see any disk2 stuff if no disk 2 process is running. After a process is finished, reload will remove your disk2 progress info.