shelfactor

ANOTHER SMALL UPDATE: Just to show more ongoing iostat results. No read/write to the resilver drives, is kinda odd. 18.txt19.txt

SMALL UPDATE: After doing a few tasks, adding media and moving some backups. I noticed that the output of your zpool iostat was giving some interesting info. Looks like the resilver-ing drives were not being called upon, but other drives were. Attached are two outputs from when the tasks where touching data on the pool, the last is now when the pool is idle again + PDF output from MAIN tab that matches 17.txt. 15.txt16.txt17.txtserver_Main-19.pdf

Ok, I'll pack up my rack and head your way! We can have a LAN Party and test all sorts of things on 1PB+ pool !!! haha Quite honestly, I have let the pool grow so that my main share can all be within the same pool (for hardlinking). However, I have started to plan out a new layout, that will separate 'active' media from archived (which should not require hardlinking). 1) Once this resilver completes, I'll try the above command to remove a single mirror pair, as a good test for a large pool. I assume it's better to leave both drives in for the removal? Or does it not matter? Thinking about reducing the read/writes off one of the drives if possible. 2) If I remember correctly, the 'new config' is only for the array/pool layout, not other settings/apps/plugins etc. correct?

Resilver completion time % has been increasing much faster since running your command changes. Last night, I had to work on a couple tasks (adding media etc), so I took a few snapshots while performing those tasks > 12.txt Then just now, I checked it again, and it's back to much calmer performance, and has gained a decent % upgrade > 13.txt Still chugging along, I still find it quite odd how long this reilver process is; and am trying to decide the best way to perform the next step (replacing the remaining 16TB drives to 18TB). 1) Remove Mirror Pairs and re-add with 2x18TB -- Wondering if this will also help with spreading data across more drives. 2) Perform yet another ~1 month resilver just like this one 3) Replace only 4 or 6 at a time, to remove such a high resilver call on the pool I am wondering, if after this resilver completes, if performing any of the above options will run faster. Thinking back to an earlier thread of you and I discussing this very topic, after a scrub/resilver > well a subsequent scrub/resilver finish faster? 12.txt13.txt

For reference, these are the numbers before and after your command changes:  Here are some example outputs. Note, There was some pool activity popping up, but was brief. Edit: One more as the pool calmed a little 10.txt 7.txt8.txt9.txt

Got it, I think some of this is what I have learned from that linked article from earlier.. But definitely do not have a strong enough knowledge on how it works. But I'm reading up on it. I have updated with your commands, will wait a short bit of time and capture some outputs Thank you so much @JorgeB !! Current Progress prior to commands:

I decided to let a major download task occur, and ran the command during that data download. The drives near the bottom are the most empty of the drives, and not part of the resilver.. But this was just to get another look, with high speed download. EDIT Adding a few more examples, so really interesting wait times. I'm not sure how to read into it. 4.txt 5.txt 6.txt

Just getting to this. Current state and attached PDF; as well as a few outputs from the above command. Only 1 or 2 or the mirror pairs are moving data... The expected finish time is now climbing. See page 2 of this thread to see another result 3.txt 2.txt 1.txt

Very odd and sporadic read/writes now, see attached server_Main-14.pdf

@JorgeB Maybe at this point I should create a new topic to discuss this further, but I am still a bit confused on the tasks to rebuild mirrored sets. My 'resilver' has now slowed down tremendously. You can see the speeds of scanned and issued are much slower. I am getting much more random reads compared to writes. ** I read this article about why rebuilds could be slow (I didn't attempt anything from it, just read it). And I'm somewhat perplexed by how my pool reacts during this resilver. Note, this server is mainly a media server, so not a lot of 'hard-hitting' applications, in fact none really at all. -- However, right now, if I start downloading new content, the resilver will slow even further. I would expect that if a file wanting to be accessed (such as a movie that is located on one of the degraded drives) would become slow due to the drive being access. I am wondering if some of the tactics shown in that article can be used to identify why this process is taking soo much time. I understand that ZFS has the 'catalog' or whatever it's called that it refers to which files are where. But my pool is made up of Mirrored pairs.. With very static content, that are fairly large files. I can't conceive why a resilver takes 2-3 weeks, to replace 12 pairs. 12 x 18TB drives = 216 TB of data to be restored (theoretically on 16TB as that was the source size). I am confused on the issued amount during this process. Article: https://cr0x.net/en/zfs-resilver-rebuild-speedup/ server_Main-13.pdf

@JorgeB for records-sake. I was able to stop the array/pool move a bunch of drives around to isolate the resilvering pairs (I may not have done it to the best isolation, but decent enough)... I can confirm: 1) The resilver picked right back up 2) The resilver copy speed has increased by almost double 3) I got caught up in why the drives appeared in the same place on the MAIN tab, but realized that's just how unraid sees the slots, it has no care where they are plugged into as long as the system can see the drives as they were before. 4) Google Fiber is amazing (this is just a fun extra note, as I got 8g/8g fiber installed today!) Will be continuing to wait the resilver for now. REPEAT QUESTION for future clarification: -- When upgrading mirror pairs with larger drives, is it better to manually remove the pairs from the pool OR perform the two-stage process that I am doing, which will call for 2x scan/scrub of the pool. server_Main-9.pdf

@JorgeB Found cause for this issue and has created a bug report. Link to thread comment about what is causing it for me: https://forums.unraid.net/topic/199210-resilver-vs-remove-mirrors-time-to-scrubreplaceresilver-1month/#findComment-1626731 Marking Solved.

I assume your test pool is also set to mirrored pairs? As you have probably seen my pool has 72 mirrored pairs, but I did run into this 'full-scrub'-resilver prior with around half of the number of vdevs. These drives are installed on DS4246 Disk shelves, most of the pool is made up of Exos 18TB drives, in both x18 and x20 series. The drives that I am replacing are Exos x18 16TB drives, being replaced with x18 18TB drives. These drives are rated for read/write ~250mbps.. however I think that the lower speed for the resilver is related to the backplane in the DiskShelves. Earlier I updated firmware across all of the shelves, but I believe that since the drives are resilvering on the same backplane .... They are maxing out the backplane speed, not the drive speeds... This has me wondering, if I turn off my server... Then swap the target drives with drives from another diskshelf, (effectively moving the 12 pairs across the 6 disk shelves, so that only 2 pairs are resilvering per shelf).. I wonder if I could get some faster speeds. Question 1: Can I stop the server, and when it comes back online and the array starts... the resilvering resumes correct? NOT a restart? Question 2: When reading the status of the pool, it appears to me that only 431T of the full 1PB pool has been 'read/scanned/scrubbed' ... is that correct, or is the 'issued' number? I ask, as I am looking at the MAIN tab (updated attachment), the resilver/copy numbers we are talking about above, and ONLY that data transfer is occurring. Meaning Drive<>Drive copy to replace the mirrrors. I don't see any other activity (of decent speed) for any other drive. Note, I have basically all my services turned off, so there should be no data movement other than the resilver. BUT If the full pool 1PB of data hasnt been scanned yet, at what point does it start processing the rest of the data? server_Main-8.pdf

@JorgeB Small update today when checking.. Attached you can see the parallel symmetrical read/writes. edit: Adding second PDF, showing excellent speeds on the read/writes. This is similar to past experiences, once long ago when you and I discussed how ZFS will "scrub" the whole pool during a resilver. And once just before the recent command run... Meaning I think the resilver will start progressing faster as more of the pool as been read. ***** This does remind me, that long ago when we discussed the full pool scrub, We had the question: "If ZFS scrubs the whole pool during this resilver, and it completes... Then If I were to swap in another drive (starting a new resilver) would it then require scrubbing the whool pool again, or would it retain the recent scan information. server_Main-6.pdf

That is definitely very curious. I would be interested, if you do find out something later on (and if you still think back to this thread). But as always, thank you for all the help! Just sent you some well deserved beer money!

@JorgeB Sorry I am out of town this weekend, so my responses are a little slower than usual. I am watching the process right now, and there seems to be a lot of reads, and occasional writes. But nothing perfectly symmetrical like the last example we just looked at. Here is the current status, and then 3x PDFs of the drives read/writes. (me trying to capture the read/writes in varying states over the last few minutes. server_Main-5.pdf server_Main-4.pdf server_Main-3.pdf

I just checked again. And you’re right only that one pair is active currently, I’ll check it a couple times in the next hour to see if there is any further activity.

Gotcha. Yea, before I ran that command, I'm positive that I saw read/writes working on multiple pairs. Update for now + plus main-tab pdf attached again. server_Main-2.pdf

Sounds good, I’ll leave it for now. Is there a command that just shows resilver information? As in, feedback on if it’s running parallel etc. versus what I ran that restarted it?

@JorgeB What would you take be on my thoughts from the original post. (technically even outside this current issue). Would it had been smart to just 'remove' the mirror pairs (which i did have plenty of room to do so).. and then add new mirror pairs of 18TB? Would it be benefical, to stop the re-silvering now, and do that? Would it be safe...? Let me rephrase *** Original: Two drive mirrors of 16TB (Drives A & B) Current: One original 16TB and one 18TB (Drives A & C) Future: Two 18TB Drives (Drives C & D) --> If I stopped the array right now, removed drive C from resilvering with drive A, (so that only the original 16TB drives are active), then install all new 18TB mirror pairs (using the ones that were being resilvered to)... Then when the array comes online, manually run the command to remove each specific 16TB Mirror pair (which only a one of each is currently active). Thus moving the data to the other drives (any, it doesnt matted to me where) in the array which also includes the new 18TB mirror pairs. edit for current status: Watching the Main tab, it appears it's all READ activity currently.

The only confidence I had, was watching the Main tab read/writes, the drives that were replaced had exact matching reads to write (16tb to 18TB) combinations. Other than that, I have no concrete evidence that it was in parallel. My incorrect assumption was the command above would check that, but as I said, I read too fast and incorrectly assumed that. haha my fault for sure. For now, I saw the stats on the pool page, reset. Here are two screen shots one from yesterday, one from just after I ran your command. As well as your requests: June 01 June 02 (after command) June 02 (now, looks like the process has scanned at a much higher rate than earlier) Attached a 'print to pdf' of the Main tab, since it would take numerous screens (Let me know if you prefer screenshots, i can do that too) server_Main.pdfserver-diagnostics-20260602-1456.zip

Hey @JorgeB thanks for the reply. Unraid 7.3.1 Well, facepalm moment for me, as I read and typed too quickly. I thought your command would be to check if they were simultaneous, not to re-start the process. I was monitoring the read/writes and there were matching reads<>writes on the various pairs. Now, it just restarted the whole thing. :☹️ Back to 40 days ETA. Now I'm curious if the time will reduce quicker since the drives already have data, or will it start fresh again? I had a thread a couple days ago, which showed screenshots, I believe it was resilvering all drives simultaneously.. Similar to what it's showing now again:

This is related to a recent post of mine, where I have 12 mirror pairs of 16TB drives as part of a larger ZFS mirror set pool. I wanted to replace the 16TB drives with 18TB drives. I figured it would be easy to remove one drive from each mirror pair, and then select the new 18TB for each set.. It would proceed to copy the data from the original mirror to the new mirror pair drive. However, I forgot that 'resilvering' in this situation, would virtually scrub the whole (1.3PB) pool, while trying to resilver the drives. (I had this happen a long time ago for a single drive replacement). This resilver has been working for over 2 weeks now, with about 2 weeks remaining. I am wondering, if I should change the process: 1) Should I stop the resilver, and then remove the mirror pairs manually? Would this be safe, in that it would just copy the data to the still active pool devices (outside of these replacement sets)? 2) If I let this resilver finish, and then proceed with part 2 which is removing the original 16TB drive, and adding in the second 18TB.. I would expect the 1 month resilver process to occur again?? ** Why I think this is a terrible process, where the ZFS pool essentially scrubs the entire pool, to replace a single drive or multiple at one time, is that it hits the pool with 100% requests. so all the drives are being hit simultaneously, which hurts performance with all the containers I have running. I am open to any thoughts and suggestions! Thank you!

@JorgeB Thank you, since it appears the resilvering is moving along now, I will marked this as solved. I will come back if I run into any further issues.

@JorgeB I believe my red flag issue may be related to this similar thread... Maybe a bug in 7.3.1? https://forums.unraid.net/topic/199069-zfs-drive-status-cant-open/