Jump to content

falconexe

Community Developer
 View Profile  See their activity

  • Posts

    789

  • Joined

  • Days Won

    15

 Content Type 

Posts posted by falconexe

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted

    13 minutes ago, Keexrean said:

    Ran both commands.

    Can confirm it's not what was actually messing things up on my side.

     

    The directory search returned nothing.

    The file search returned just old files from damages drives and/or NTFS partitions that dropped to raw, and none of these files were the ones making the dir command crash, I can even find them in the disk audit log I ran yesterday.

    No worries. At least you know that is not the issue.

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    26 minutes ago, falconexe said:

    Thanks trying this now...

    Yep that worked. Thanks.

     

    Echo Performing Audit on Disk: 1...
    DIR /s /b "\\%ServerName%\disk1" 2>NUL 1>"%OutputPath%\%CurrentDate%\%FileNameRaw%"
    DIR /s    "\\%ServerName%\disk1" 2>NUL 1>"%OutputPath%\%CurrentDate%\%FileNameMeta%"


    FOR /L %%i IN (2,1,%DataDisks%) DO (
    Echo Performing Audit on Disk: %%i...
    DIR /s /b "\\%ServerName%\disk%%i" 2>NUL 1>>"%OutputPath%\%CurrentDate%\%FileNameRaw%"
    DIR /s    "\\%ServerName%\disk%%i" 2>NUL 1>>"%OutputPath%\%CurrentDate%\%FileNameMeta%"
    )

     

    image.thumb.png.4d472e40c45e6aa8ed73827e559601de.png

     

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    1 hour ago, falconexe said:

    I was able to find these erroneous files and folders by going to each disk share and running the following command on the UNRAID terminal: find . -type f -ls |grep ' 1969 '

     

    So I forgot to mention that the "type" parameter can be set to the following:

     

    File:           find . -type f -ls |grep ' 1969 '

    Directory:  find . -type d -ls |grep ' 1969 '

     

    I have seen both files and folders have the missing creation date attribute where it defaults to 1969.

     

    Here is some more info on the subject: https://www.a2hosting.com/blog/whats-the-deal-with-12-31-1969/

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    15 minutes ago, Keexrean said:

    Well thanks but don't take too much of it as any kind of teaching, there is a LOOOOOT of bad habits there, from when I started spagehti writing batch as a teenager in the early 2000s.

     

    But yah, that's a decent preview of how logic works in most of my batches.

    Namely I'm using a lot what I label as "modules" in my code.

    Like using the same module to both do a full array detection, and also to just check if the manual set disk count is at least valid.

     

    Note though, I forgot to add a cache-exist detection I think, it doesn't check before asking you if you want to audit cache.
    I'll do it in the next iteration when a solution for the ">dir crashes to restricted files/directories" issue just pops in my head.

    No worries. I have been having some weird stuff with my scripts too. The Meta audit sometimes runs into 1969 files that are missing the creation date on the folder. When this is encountered it outputs "The parameter is incorrect." into my screen output. I spent many hours last night touching files to reapply these attributes.

     

    I was able to find these erroneous files and folders by going to each disk share and running the following command on the UNRAID terminal: find . -type f -ls |grep ' 1969 '

     

    Then you "touch" the files and it should fix the issue.

     

    Here is what it looks like when it happens in the script. You can see this exact output by just running the command in CMD natively, so it is not script related. Super freakin annoying...

     

    image.thumb.png.846c1ac4f84e2edc160e188bb113b8ea.png

     

    I had this completely fixed last night after fixing all of the files in question. I then re-ran the audit and I had clean screen output.

     

    Then this morning I ran it again and I have more of these lines in different disks. So not sure if it is always the "1969" issue. This does not happen on the RAW "DIR /s /b" command, only the META "DIR /s" command.

     

    Anywho, I'll keep trying to track the issue down. I hate intermittent issues! If anyone else is seeing this on their server, I would be very curious. Again the scripts work perfectly. It is the actual data throwing the exception.

     

    Worse case scenario, I would love to just suppress the error line on my output screen. Any thoughts on how to do this?

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    9 hours ago, Keexrean said:

    Oh the dynamic loop issue I solved before diving into the access denied skip!

     

    If you don't have any permission issues, the script should work fine for you already, but I'm in heated brainstorming on stackoverflow, so the following version of the script is to be considered temporary.

     

    I want it to work without having have a network user with GAWD PAWA on the network who can access really every. I have to find a way to make it handle&skip, of just avoid the a set list of folders or folder/subfolder couples like the plague, whereever they are, without hard coding them to a specific location.

     

    I supports cache audit toggle, as well as asking OR detecting the number of drives, and also supports adding the path to a arrayshare instead of relying on diskshares.

     

     

    UNRAID_Audit_All_Disks.bat 9.41 kB · 1 download

    Dude, this is impressive. It is way more crazy and complex than I could have imagined. You definitely taught me some new techniques. And I noticed your PowerShell injection. 

     

    I just finished running the audits using your script and got the EXACT same results down to number of lines in each file and the exact same size of output files down to the byte. So both of our scripts match up output-wise exactly. Peer Review Complete ha ha.

     

    Nice work!

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    Freaking Sweet. So after about 10 minutes of Google searching, I was able to solve the dynamic loop question. I can now dynamically tell the script the total number of data drives I have and it will loop through a single block of code until completed. Work smarter not harder! 😅

     

     

    Variable:

    SET DataDisks=28

     

    Code:

    Echo Performing Audit on Disk: 1...
    DIR /s /b \\%ServerName%\disk1>"%OutputPath%%FileNameRaw%"
    DIR /s     \\%ServerName%\disk1>"%OutputPath%%FileNameMeta%"
     

    FOR /L %%i IN (2,1,%DataDisks%) DO (
    Echo Performing Audit on Disk: %%i...
    DIR /s /b \\%ServerName%\disk%%i>>"%OutputPath%%FileNameRaw%"
    DIR /s     \\%ServerName%\disk%%i>>"%OutputPath%%FileNameMeta%"
    )

     

    Note: The IN (#,#,#) DO loop parameters map out to (Start, Step, End). So in this case, we start at disk2 since the script is now appending to the files disk1 created. We are then stepping by 1 disk at a time. And finally, we end at the variable value of 28 (my last disk).

     

     

    I Also Added Some New Bells and Whistles:

    • Dynamic Audit Output Folder Creation Based on Current Date in YYYYMMDD Format
      • Only Creates This Folder If It Does Not Already Exist
    • Audit Start/End Timestamps During Run-Time

     

     

    I have attached the FULLY DYNAMIC Version of the UNRAID DISK AUDIT script. I have also fully set all text contents to variables which will make @Keexrean happy.

     

    Anyone in the community should now be able to run this (On Windows) for any UNRAID server by simply changing out the variables at the top of the script.

     

     

    Prerequisites/Instructions:

    • Set Your Windows hosts File (Make a Backup First!)
      • Allows You to Call UNRAID Server by Name Instead of IP (For NET USE Command)
      • Location of hosts File in Windows 10: C:\Windows\System32\drivers\etc\hosts
      • Open hosts File with Notepad
      • Enter This At Bottom of hosts File (Change IP Address and Host Name Accordingly):

        • #Custom Defined Hosts

          192.168.#.#   YOURSERVERNAME

      • Save the New hosts File

    • Turn on Per Disk Shares in UNRAID (Do This For Every Data Drive)
      • I Set My Disk Share Type To:
        • Export: Yes (hidden)
        • Security: Secure
      • I Set My Disk Share Access To:
        • User: Read-only
    • Right Click and Edit the Batch Script
    • Set Variables Accordingly
    • Sit Back and Relax, This Will Take a While...

     

     

    Use Cases:

    • Run Regularly to Have an Exact Mapping of Every File on the Array
    • Run Prior to Parity Checks
    • Run Prior to Reboots/Shutdowns
    • If You Ever Experience Data Loss, At Least You Will Know Which Files Were On That Drive!

     

     

    ENJOY!

     

    Updated Code 2020-01-19

     

    UNRAID_Audit_All_Disks.bat

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    Yeah, I just performed a full audit on my appdata share without any issues. Natively my appdata folder is about 30GB in size. For the audit results, it produced txt files with hundreds of MBs and millions of records/lines inside each (Both Raw and Meta). I have different dockers than you, so I must have ones that work without throwing exceptions.

     

    Let me know if you figure it out. Sorry you are running into issues...🤷‍♂️

     

    I am very interested to see how you loop through a set number of disks via a single variable though. If you figure out that snippet of code, please send it my way. I can incorporate it back into my script and send it out to the masses if you are cool with that.

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted

    23 minutes ago, Keexrean said:

    Oh well, I knew about this option, but I'll concede to mess up some perms I set myself for the sake of that script, and I'll let you know ;)

     

     

    Edit: Dockers stopped, perms resetted, Dolphin's share still denying access. I feel like this one will tickle my pickle quickly and yeet off of my server.

    Hmm. I don't use that docker so I am stumped. Any chance you could skip the cache share for now and get the main disk script working fully dynamically?

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    17 minutes ago, Keexrean said:

    Mmmmh... when running on the cache it seems to stop listing folders after bumping in the dolphin docker folder and skips to the array.

    Any idea on that? You think it's because it's denied access maybe?

     

    Also yeah I see why you see cache as a share, but I see it more as a disk since I have a Raid1 cachepool and actually have some share that live solely on the cache, not the array (including appdata, system, a "CachedDomains" folder for some VMs, etc).

    Yeah I can see both sides. Yes, when it stops it most likely is a permissions issue. I've seen the same thing on my cache share script. Certain appdata folders can cause issues.

     

    For data disks, I usually run the "Docker Safe New Perms" script to reset permissions across the array. It is safe...and resolves similar issues, especially when uploading files to the array from OSX.

     

    image.thumb.png.fbd54e7a0ccf5fe1ded179597fb831c9.png

     

    image.thumb.png.35c96137d6115955241969a831b0fbcf.png

     

    You also might be running into issues if certain dockers are running and have file locks. I usually run my cache share script when all dockers are disabled and the array is stopped for that reason.

     

     

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted

    13 minutes ago, Keexrean said:

    While I'm janky-typing that, does it needs to also audit cache drives?

    In my mind, the cache is a SHARE, not an array data drive, so I have it in my Share script. However, technically it is a drive, so I could see it being in the Drive script.

     

    If it is not too hard, yeah throw it in the end of the script. Or you can make 2 versions, one with just true Data Drives, and one with Data and Cache. However you would like. Thanks again for taking some time to solve this!

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    14 minutes ago, Keexrean said:

    Also, thinking about it, I may have something that could help you ease the logging/managing of such a massive array.

    I have that in my SMB extra config: 

    
[Procyon_ArrayWholeView]
      path = /mnt
      comment = sysview
     browseable = yes
        valid users = rootshare
        write list = rootshare
        vfs objects =
        guest ok = no
        read only = no

    I just felt that it was more comfy than making disk shares, but that's imhu.

    Thanks! I coulda used that info a while back before getting carpal tunnel setting each of the 28 disks one-by-one... 🤣

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    13 minutes ago, falconexe said:

    OK, and here is the ALL DISKS script with dynamic variables. Change the variables up top as usual. You can remove the extra disks of your array is smaller as needed. I have 28 data disks, so the code goes that far. Let me know if you have any questions.

    UNRAID_Audit_All_Disks.bat 5.89 kB · 0 downloads

    @Keexrean OK so here is where I could use some help due to lack of time.

     

    I would like to code this fully dynamically with a loop that goes through the number of disks based on a "TotalDisks" variable. So instead of me repeating blocks of code (1 per disk), I would like this to use a SINGLE BLOCK of code, that loops once per disk.

     

    Variables Needed: 

    SET TotalDisks=28

     

    Code to Loop Through 28 Times:

     

    Echo Performing Audit on Disk: #...
    DIR  /s /b \\%ServerName%\disk#>"%OutputPath%%FileNameRaw%"
    DIR  /s \\%ServerName%\disk#>"%OutputPath%%FileNameMeta%"

     

    The trick is that on the second iteration of the loop and so on to the upper bound of the variable (2-28), the code would need to append the file like below (in RED):

     

    Echo Performing Audit on Disk: #...
    DIR  /s /b \\%ServerName%\disk#>>"%OutputPath%%FileNameRaw%"
    DIR  /s \\%ServerName%\disk#>>"%OutputPath%%FileNameMeta%"

     

    Also, the user feedback message would need to be dynamic and refer to the current disk number (Current Loop Iteration):

    Echo Performing Audit on Disk: #...

     

    If you OR ANYONE ELSE has this already written, or would like to solve this, I would be super appreciative! Otherwise, I'll need to find some time to write this myself. Either way, I'm sure we'll get it figured out and we can upload it to this thread to help some people out. Thanks!

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted

    17 minutes ago, Keexrean said:

    Do note in no mean I was "criticizing" your work or underrating what you can do, I was already convinced you were able to do the little adjustments I made to your script.

    It was more of just a little introduction as to not barge in like a basic scriptkiddie who just wanted to gather internet points, and made a small PSA

     

     

    None taken and no worries. I went ahead an made a "simple" dynamic version of the ALL SHARES script.

     

    I may have 1 thing that you can jump on if you have the desire in my ALL DISKS script. I don't have the time right now to do this PART myself, but if you have existing code and/or want to tackle it, by all means, it would be appreciated. I'll ping you once I post it and let you know what I need...

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted

    2 hours ago, falconexe said:

    Sure. Glad these worked for you and that they helped someone else. I'm working on the templates for the other 2 scripts now.

    @dabl Here is the ALL SHARES version of the script.  I made this one completely dynamic @Keexrean

     

    You will need to set each share name variable that you have on your server. If you go past the 3 I show here, just make a new numbered variable, copy and paste the code block that does the work, and change the variable name in that block. You will also want to set the other hard-coded variables for your server (just like before, but now they are at the top). Since we are using variables, you only have set them once.

     

    Example: If you wanted to add a 4th share for "Media" you would add the following variable up top and code in the middle:

     

    Variable:

    SET Share04=Media

     

    Code:

    Echo Performing Audit on Share: %Share04%...
    DIR  /s /b \\%ServerName%\%Share04%>>"%OutputPath%%FileNameRaw%"
    DIR  /s \\%ServerName%\%Share04%>>"%OutputPath%%FileNameMeta%"

     

    Note: The first share creates (and overwrites if they exists) the output files. Shares 2+ will APPEND to these files.

     

    This will take a very long time to run since it is documenting all contents on all shares you set. Let me know if this works for you.

    UNRAID_Audit_All_Shares.bat

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted

    1 hour ago, Keexrean said:

    Chiming in, but wouldn't it be easier to transpose to other servers and stuff if it was like that? (file attached)

     

    Batch is my go to for anything I can't feel arsed to do more than twice, and also ended up making some that write VBS scripts that it can then inject variables into and call to do things like download files from some locations to others, do version checking of packages I made myself and tons of stupid stuff like that...

    Yeah, some of my batch scripts almost look like viruses sometimes, dropping dozens of .vbs files, downloading archives, popping python scripts to decrypt/encrypt stuff, ect.

    Some nearing the 20megabytes mark... oof.

     

     

    All in all blabering, just saying that declaring a bunch of variables right away avoids having to track down through your scripts what you need to modify or not if it's all presented to you at the very top of it.

    UNRAID_Audit_Single_Disk.bat 1.92 kB · 1 download

    Sure you can do that. This was just a quick and dirty script I wrote for myself that I was sharing.  I was fine with Copy and Replace for the server name. It was never meant for the masses. I have some extremely complex and dynamic versions of these scripts that I will keep to myself. You and I are seasoned veterans in batch it appears, and most people area not. I wanted to keep it simple for everyone else. Anyone can take this to the next level and put in loops and dynamic variables at their pleasure. I welcome the sharing of code. If anyone improves upon my original work, please do share on this thread. I would love to see it. I'm just trying to help out... These scripts have saved my butt on numerous occasions. I would actually like to see UNRAID put in similar scripts natively that save off EVERY SINGLE FILE in a log before a parity check starts. That would be really cool. There are some plugins out there that you can have stuff kick off before and after a parity check, but they never worked consistently for me. As a result, I do this manual method with my batch scripts on Windows.

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted

    39 minutes ago, Froberg said:

     

    Sounds like my origin story, too, almost all of it. My computer was far shittier though and came with Windows ME. I remember one time fiddling with screen refresh rates, not knowing how they worked, and being unable to use the computer. My mom was NOT happy, lmao. Figured it out, because that's how you learn, but man was I worried because it was so expensive.

    My second one was an Athlon XP 1800 I believe. Massive upgrade!

     

    Ha Ha. I remember those days when the screen goes black and you basically crapped yourself because you were messing with refresh rates/unsupported resolutions. Good times.

    [SOLVED] Double Data Drive Failure During Parity Check (Dual Parity)

    in General Support

    Posted · Edited by falconexe

    So here is the first script (attached). It is dynamic in that it prompts the user for a disk number. Note disks 1-9 are single digit, while 10+ are double digits. Once you enter the disk you want, the script will pull both a raw file listing with paths, and a metadata listing with folder structure to your desired location. Once completed, the script will automatically open the completed audits.

     

    This is coded in Windows batch so you will need to right click and select "edit" on the batch file. You will want to change out "MassEffect" with the name of your UNRAID server along with your server details. You will also need to change your destination path accordingly. Let me know if you run into any issues.

     

    If this works for you, I have 2 more scripts that do this same work for the entire array with ALL DISKS, and ALL SHARES. If you are interested in these, please let me know.

     

    Note: This script requires that each disk be shared on your array. I keep mine shared, but hidden so anyone else on my network would not see/know they are there. I also have my security per disk set to Read Only.

     

    image.png.c774aa377a22c5ace48c364255f9d836.png

     

    image.thumb.png.7cfcaa69b0351350160c8e8628fbb49d.png

    image.thumb.png.783c2bc4396b2b592d6646066cf79c6b.png

    image.thumb.png.90a3975e969ed72d7e1a81c73c32680a.png

     

     

     

     

     

    UNRAID_Audit_Single_Disk.bat

×
×
  • Create New...