Zero disk command runs at 400kb/s

[jkexbx]

The commands to zero a disk are:

umount /mnt/disk15

dd bs=1M if=/dev/zero of=/dev/md1 status=progress

 

When I run the dd command, the progress runs at 400kb/s with no way to kill it. I end up following the below steps to successfully zero a disk.

 

I have the dd command run fine after an umount in the past.

 

Does anyone know if there's a specific trick to it?

 

1. Run the umount command
2. The disk size changes to a random number
3. I then run the dd command, and progress is running at 400kb/s
4. I fail to kill the running dd using every method available
5. I stop the array, but it fails
6. I hard shutdown the server
7. It comes back up
8. I try to mount the array, but it's in some half broken state where VMs can start, but the GUI still allows you to modify disks
9. I then do another reboot through the GUI
10. Everything comes back normally
11. The disks I'm trying to zero show unmountable: wrong or no file system
12. I run dd again on them and it runs fine at a normal 80mb/s

tower-diagnostics-20231223-1749.zip

[itimpi]

Not sure when in the sequence of steps you outlined you took those diagnostics..   In the diagnostics there is writing going on to two drives (disk12 and disk17) which will kill performance of the clear (which is presumably to one of those drives).   Stop the writing to the other drive and things should speed up dramatically. 

[jkexbx]

Thanks for taking a look, but It's not relating to other drives activity. I can kill all dockers, VMs, and scripts, and it will still continue at the same speed.

 

There is some additional interaction going on that causes a consistently slow write speed of 400-500 kb/s. I believe it's related to the umount not working as expected.

 

If you need me to run through the sequence and download logs at a specific time I can. But I'm trying to avoid that without something to also try because the only way is to hard power down the server after running the first dd command. No matter what I do, I can't get the server to clean shutdown.

[Vr2Io]

I don't think unmount an array member disk was a correct step.

 

In this case, zero a member disk also cause parity update too. Slowdown also expected.

Edited December 23, 2023 by Vr2Io

[jkexbx]

Per the Unraid Docs it's how you zero a disk to remove it from an array. The script is broken in a couple of different ways, so I avoid that now.

 

Do you know if there's a new way to zero a disk?

 

Quote

Alternate Procedure steps for Linux proficient users

 

If you are happy to use the Linux Command line then you can replace steps 7 and 8 by performing the clearing commands yourself at a command prompt. (Clearing takes just as long though!) If you would rather do that than run the script in steps 7 and 8, then here are the 2 commands to perform:

 

umount /mnt/diskX

dd bs=1M if=/dev/zero of=/dev/mdX status=progress

 

(where X in both lines is the number of the data drive being removed) Important!!! It is VITAL you use the correct drive number, or you will wipe clean the wrong drive! That's why using the script is recommended, because it's designed to protect you from accidentally clearing the wrong drive.

 

It's supposed to cause a parity update. I'd expect it to run at 50 mb/s like it does after the hard reboot. The problem is something is happening with the umount to cause it to run at 400kb/s.

[trurl]

Simpler to just New Config without the disk and rebuild parity.

[Vr2Io]

4 hours ago, jkexbx said:

Per the Unraid Docs it's how you zero a disk to remove it from an array. The script is broken in a couple of different ways, so I avoid that now.

 

Do you know if there's a new way to zero a disk?

 

 

It's supposed to cause a parity update. I'd expect it to run at 50 mb/s like it does after the hard reboot. The problem is something is happening with the umount to cause it to run at 400kb/s.

Haven't notice official doc will umount the target disk, my bad.

 

May be best someone could try reproduce same problem or not.

 

It seems problem relate umount ( umount not complete ), some post also relate to this.

 

Anyway as official mention, start array in maintenance mode could improve zero disk performance but it also greatly avoid umount failure issue too.

 

Edited December 24, 2023 by Vr2Io

[jkexbx]

5 hours ago, trurl said:

Simpler to just New Config without the disk and rebuild parity.

Definitely simpler, but incredibly risky. If any data disks die during rebuild, you don't have a valid parity left. I'm trying to maintain parity while removing data disks.

[trurl]

I know why people do it.

 

But you should always have another copy of anything important and irreplaceable, parity is not a backup.

 

Of course, disks can go bad without warning, but some don't heed warnings until too late. Do you have Notifications setup to alert you immediately be email or other agent as soon as a problem is detected?

[jkexbx]

Yeah, all the above has been setup and tested previously. Just trying to avoid the headache of restoring data.

 

I just don't want to take any needless risk and it's surprising that removing a disk while keeping parity intact is such a difficult thing to do.

[itimpi]

24 minutes ago, jkexbx said:

Yeah, all the above has been setup and tested previously. Just trying to avoid the headache of restoring data.

 

I just don't want to take any needless risk and it's surprising that removing a disk while keeping parity intact is such a difficult thing to do.

 

It is reasonably easy to do via the CLI as long as you run the array in Maintenance mode while doing it.

 

It would be nice, though, to have a GUI way of forcing a Clear on the disk so it can be safely removed.

[jkexbx]

So I've had some interesting leanings, and found something workable.

 

1. Those commands as a pair are intended for the array to be mounted
   • If the array is in maintenance mode and you try to run umount /mnt/disk# you receive: no mount point specified
   • That means the only reason you would ever need umount /mnt/disk# is if the array was mounted
2. If you start the array in maintenance mode you can run just the zero command dd bs=1M if=/dev/zero of=/dev/md1 status=progress and the write speed is as expected
   • You can also exit the terminal and the command will stop after 10 minutes or so.
   • When the array is remounted, the disks show Unmountable: Unsupported or no file system
   • When you run the zero command on that unmountable disk then with the array running the zero runs at expected speed.
3. While the command is running, the scheduled parity check won't run.
4. All this means that when the array is mounted, and you run the umount /mnt/disk# something is happening that results in the disk almost, but not quite being unmounted.
   • The zero command does run, just slowly.
   • The zero command does not terminate
   • You cannot stop the array because the zero command is running without any way to kill it

 

My procedure that ended up being reasonable is:

1. Verify disks are empty
2. Stop array
3. Start array in maintenance mode
4. Run dd bs=1M if=/dev/zero of=/dev/md1 status=progress
5. Wait for first 20G of data to be zeroed
6. Close command window
7. Wait until disk activity goes to zero(around 50G of writes)
8. Repeat with next disk
9. Reboot (you could also stop array and then start, but for me that results in a graphical glitch)
10. Start array normally
11. See disks you're trying to zero as unmountable
12. Run dd bs=1M if=/dev/zero of=/dev/md1 status=progress on disks 1 by 1.
13. Remove disks from array
14. Reset configuration

What's funny about this is I still don't ever need to use the umount command. I've done at least a partial parity check and have 0 sync errors.

 

Running the zro command for a short while removes the drive formatting, so it's no longer mountable.

 

All with dual parity.

 

Of course you can always just keep it in maintenance mode, but if you're trying to zero a dozen disks like me that means your server is down for a week.

 

Edited January 6 by jkexbx