Jump to content
gfjardim

Preclear plugin

2901 posts in this topic Last Reply

Recommended Posts

Is there a chance you could add a DOD pre-clear as an option?
Apparently DoD does not have a method of choosing to erase non classified media, and classified media has classified methods of disposal.

We can use NIST to do so:

According to the 2014 NIST Special Publication 800-88 Rev. 1, Section 2.4 (p. 7): "For storage devices containing magnetic media, a single overwrite pass with a fixed pattern such as binary zeros typically hinders recovery of data even if state of the art laboratory techniques are applied to attempt to retrieve the data."[23] It recommends cryptographic erase as a more general mechanism.



So if you run an Erase and Clear the Disk operation, you will random fill and then zero fill your disk, making it virtually impossible to be recovered.

Enviado de meu SM-N970F usando o Tapatalk

Share this post


Link to post
11 minutes ago, gfjardim said:

Apparently DoD does not have a method of choosing to erase non classified media, and classified media has classified methods of disposal.

We can use NIST to do so:
 


According to the 2014 NIST Special Publication 800-88 Rev. 1, Section 2.4 (p. 7): "For storage devices containing magnetic media, a single overwrite pass with a fixed pattern such as binary zeros typically hinders recovery of data even if state of the art laboratory techniques are applied to attempt to retrieve the data."[23] It recommends cryptographic erase as a more general mechanism.
 



So if you run an Erase and Clear the Disk operation, you will random fill and then zero fill your disk, making it virtually impossible to be recovered.

Enviado de meu SM-N970F usando o Tapatalk
 

I'll just do that with 3 passes then.

Share this post


Link to post

just added a brand new 8TB drive to my server. I clicked unassigned devices so it would show up. It took a while for it to be listed. I went into unassigned devices and looked at the log and see this:

Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] 4096-byte physical blocks
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Write Protect is off
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Mode Sense: 7f 00 00 08
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Attached SCSI disk
Oct 7 11:53:04 AUBURN unassigned.devices: Error: shell_exec(/usr/sbin/hdparm -C /dev/sdn 2>/dev/null | /bin/grep -c standby) took longer than 10s!
Oct 7 11:53:27 AUBURN unassigned.devices: Error: shell_exec(/usr/sbin/hdparm -C /dev/sdn 2>/dev/null | /bin/grep -c standby) took longer than 10s!
Oct 7 11:53:27 AUBURN unassigned.devices: Error: shell_exec(/usr/sbin/smartctl -A /dev/sdn | /bin/awk 'BEGIN{t='*'} =='Temperature:'{t=;exit};==190||==194{t=0;exit} END{print t}') took longer than 10s!

 

When it did show up this is what is showing in the unassigned device area: I can't click format.

image.thumb.png.6ac2d67de4a60f38fd6f902f94afad63.png

 

I decided to go in and try to pre-clear the drive and when I select start pre-clear the process starts but just shows as starting.... I gave it about 30 minutes but still shows as starting. I stopped the pre-clear and status still showed as starting. I cleared the status so now it says Start Preclear.

 

I removed the drive and inserted it back in. I it recognized the drive and started preclear. Now it stuck on starting for 1 hour.

Attaching diagnostics.

 

Am I doing something wrong? Bad disk? Anything I should be checking?

auburn-diagnostics-20201007-1439.zip

Share this post


Link to post

@stepmback, I didn't see any error with preclear, but you do have a bad drive. It keeps disconnecting and reconnecting from the HBA, and perhaps the preclear session didn't started because it can't access the device after you start it.

 

 

 

 

Share this post


Link to post
22 hours ago, stepmback said:

just added a brand new 8TB drive to my server. I clicked unassigned devices so it would show up. It took a while for it to be listed. I went into unassigned devices and looked at the log and see this:

Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] 4096-byte physical blocks
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Write Protect is off
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Mode Sense: 7f 00 00 08
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 7 11:51:56 AUBURN kernel: sd 7:0:21:0: [sdn] Attached SCSI disk
Oct 7 11:53:04 AUBURN unassigned.devices: Error: shell_exec(/usr/sbin/hdparm -C /dev/sdn 2>/dev/null | /bin/grep -c standby) took longer than 10s!
Oct 7 11:53:27 AUBURN unassigned.devices: Error: shell_exec(/usr/sbin/hdparm -C /dev/sdn 2>/dev/null | /bin/grep -c standby) took longer than 10s!
Oct 7 11:53:27 AUBURN unassigned.devices: Error: shell_exec(/usr/sbin/smartctl -A /dev/sdn | /bin/awk 'BEGIN{t='*'} =='Temperature:'{t=;exit};==190||==194{t=0;exit} END{print t}') took longer than 10s!

 

When it did show up this is what is showing in the unassigned device area: I can't click format.

image.thumb.png.6ac2d67de4a60f38fd6f902f94afad63.png

 

I decided to go in and try to pre-clear the drive and when I select start pre-clear the process starts but just shows as starting.... I gave it about 30 minutes but still shows as starting. I stopped the pre-clear and status still showed as starting. I cleared the status so now it says Start Preclear.

 

I removed the drive and inserted it back in. I it recognized the drive and started preclear. Now it stuck on starting for 1 hour.

Attaching diagnostics.

 

Am I doing something wrong? Bad disk? Anything I should be checking?

auburn-diagnostics-20201007-1439.zip 156.75 kB · 1 download

Any insight into what may be going on? I think I just have a bad disk.

Share this post


Link to post
Any insight into what may be going on? I think I just have a bad disk.


Please take a look at the post right before yours.

Enviado de meu SM-N970F usando o Tapatalk

Share this post


Link to post

I ended up getting a replacement drive. Put it in and started preclear. No problems.. it is running.. I guess a bad drive.

 

gfjardim... do you know what disk is going bad?

 

 

 

Share this post


Link to post
On 2/5/2017 at 7:57 AM, gfjardim said:

In The Statistics Sender plugin, I added a Dismissed Statistics Reports under /Tools. If you dismissed a report and want to send it later, you can go there and submit it later.

This feature is not working for me. I have some reports stuck there that won't go away. My internet connection does not seem to be the issue as a new preclear report that pops up for sending has been sending successfully. I have no idea how to fix this or if I'm doing something wrong.

 

EDIT: It started working after a reboot.

Edited by DougCube

Share this post


Link to post

I am curious, is it possible to test the preclear signature to make sure it will work properly?

 

Let me explain, I have a USB dock I would like to use to pre-clear a drive but not sure if this will work when I plug it in via sata to attach it to the array.

 

I was hoping to be able to test this before hand before wasting the time. I could use a small laptop drive for testing if needed but was not sure on a way to do this that would not cause me to have to rebuild parity.

Share this post


Link to post
55 minutes ago, TexasUnraid said:

I am curious, is it possible to test the preclear signature to make sure it will work properly?

I can only speak for myself, but I'm sure others use the same methodology. I typically buy external USB drives and shuck them for use with my unRAID setups. Yes, I lose out on some warranty coverage (2 or 3 years vs 5 for the bare drives), but the cost savings can make up for that. For example, I picked up 4 x Seagate STEB16000402 16TB USB external drives just over a month ago. All 4 were precleared while still in the factory USB enclosure. When shucked and attached via SAS/SATA, the preclear signature was still present and valid.

 

So, as long as your USB enclosure/dock passes the drive through properly, you shouldn't have any issues. To confirm if it does, you should see the drive model number and serial number in the unRAID/Unassigned Devices webgui. Once the preclear completes, the webgui will show that the drive has a preclear signature.

 

If your USB dock alters the drive model/serial info, then I would suspect that the preclear signature may not be recognized when the drive is added to the array. The only way to be sure would be to try. Your thought on using a small drive to perform the test may be worth the effort.

 

Good luck!

 

Share this post


Link to post
4 minutes ago, AgentXXL said:

I can only speak for myself, but I'm sure others use the same methodology. I typically buy external USB drives and shuck them for use with my unRAID setups. Yes, I lose out on some warranty coverage (2 or 3 years vs 5 for the bare drives), but the cost savings can make up for that. For example, I picked up 4 x Seagate STEB16000402 16TB USB external drives just over a month ago. All 4 were precleared while still in the factory USB enclosure. When shucked and attached via SAS/SATA, the preclear signature was still present and valid.

 

So, as long as your USB enclosure/dock passes the drive through properly, you shouldn't have any issues. To confirm if it does, you should see the drive model number and serial number in the unRAID/Unassigned Devices webgui. Once the preclear completes, the webgui will show that the drive has a preclear signature.

 

If your USB dock alters the drive model/serial info, then I would suspect that the preclear signature may not be recognized when the drive is added to the array. The only way to be sure would be to try. Your thought on using a small drive to perform the test may be worth the effort.

 

Good luck!

 

Yep, same thing I am doing but when I used the enclosure it came with (WD elements) the pre-clear was not recognized and I had to do it again.

 

I also have a USB dock and was going to use it but didn't want the same thing to happen. Was not sure if there was a way to verify the signature without adding the drive to the array.

Share this post


Link to post
1 minute ago, TexasUnraid said:

Yep, same thing I am doing but when I used the enclosure it came with (WD elements) the pre-clear was not recognized and I had to do it again.

 

I also have a USB dock and was going to use it but didn't want the same thing to happen. Was not sure if there was a way to verify the signature without adding the drive to the array.

As for avoiding a parity rebuild if you try the test drive, you might be wiser to create another bootable unRAID USB key and use it on a second PC to do the preclear of the test drive in your USB dock. I and others keep a non-activated unRAID USB key handy to do preclears on another system. Once booted all you need to install is the Preclear plugin and Unassigned Devices. This standalone unRAID never needs to be licensed if you're only using it for preclears.

 

Once the drive has been precleared on your 2nd system, power down, disconnect it from your dock and attach it to your production unRAID system via SAS/SATA. If the signature is valid, it will show beside the drive info in the UD section of the webgui.

 

Share this post


Link to post
2 minutes ago, AgentXXL said:

As for avoiding a parity rebuild if you try the test drive, you might be wiser to create another bootable unRAID USB key and use it on a second PC to do the preclear of the test drive in your USB dock. I and others keep a non-activated unRAID USB key handy to do preclears on another system. Once booted all you need to install is the Preclear plugin and Unassigned Devices. This standalone unRAID never needs to be licensed if you're only using it for preclears.

 

Once the drive has been precleared on your 2nd system, power down, disconnect it from your dock and attach it to your production unRAID system via SAS/SATA. If the signature is valid, it will show beside the drive info in the UD section of the webgui.

 

Ah, didn't think about the UD section saying that it is valid. That should also work on the main system to let me know if the signature is valid. I can test it on a small 80gb drive that will not take long.

 

I don't really have a 2nd system I can sacrifice for a few days while this runs.

Share this post


Link to post
3 minutes ago, TexasUnraid said:

Ah, didn't think about the UD section saying that it is valid. That should also work on the main system to let me know if the signature is valid. I can test it on a small 80gb drive that will not take long.

 

I don't really have a 2nd system I can sacrifice for a few days while this runs.

Yes, your main unRAID can perform the same function. Note that when the preclear finishes, the signature may not show in the UD webgui until you restart your unRAID system. You may be able to just refresh the UD section of the webgui, but that sometimes doesn't seem to work. A restart is truly the best way to verify that the signature is present before you add the drive to your array. That's why I like using a 2nd system for the preclears - I use an older Dell laptop that has USB 3.0 on it so I can preclear at decent speed and I can verify the signature after a reboot without taking down my production unRAID system.

 

Share this post


Link to post
2 hours ago, TexasUnraid said:

I am curious, is it possible to test the preclear signature to make sure it will work properly?

 

Let me explain, I have a USB dock I would like to use to pre-clear a drive but not sure if this will work when I plug it in via sata to attach it to the array.

 

I was hoping to be able to test this before hand before wasting the time. I could use a small laptop drive for testing if needed but was not sure on a way to do this that would not cause me to have to rebuild parity.

Of course you can. In the Start Preclear menu, in Operation, just select Verify MBR Only or Verify All the Disk.

Share this post


Link to post
14 minutes ago, gfjardim said:

Of course you can. In the Start Preclear menu, in Operation, just select Verify MBR Only or Verify All the Disk.

Very nice, I thought I saw something like that at one point but then could not find it again when I had the issue last time.

Share this post


Link to post

I have been looking around and I can't find a definitive answer so hopefully someone can help me here.

 

I'm currently in the process of preclearing 2x 14TB & 2x 12TB drives via USB 3.0 before schuking them. I had scheduled them to do two cycles including pre and post-read, but each step is taking 20-24 hours, so a full cycle takes about 3 days. I'm currently on the first cycle for all drives and they are all at step 2 (zeroing) approximately 60 - 80% so far. I don't have nearly a week to wait just for these four drives, and then to top it off I've got 8 more drives to preclear. That would be three weeks before all the drives were ready for the array at this rate.

 

I mean, I do plan on setting up and starting the array with the first four drives once they are ready and adding the rest of them as they complete the preclear process. However I don't want to be waiting weeks on end before this is all set up and the array is complete. 

 

So I was wondering, is there a way to abort / end the current preclear process after the first cycle completes and still retain the preclear signature (I think it's called)? I know that there's no way to do it exactly as when it completes the first cycle it will automatically begin the second cycle, but at what point am I safe to cancel the process and still have the drives classed as precleared and ready to be formatted?

Share this post


Link to post
17 hours ago, VisualHudson said:

I have been looking around and I can't find a definitive answer so hopefully someone can help me here.

 

I'm currently in the process of preclearing 2x 14TB & 2x 12TB drives via USB 3.0 before schuking them. I had scheduled them to do two cycles including pre and post-read, but each step is taking 20-24 hours, so a full cycle takes about 3 days. I'm currently on the first cycle for all drives and they are all at step 2 (zeroing) approximately 60 - 80% so far. I don't have nearly a week to wait just for these four drives, and then to top it off I've got 8 more drives to preclear. That would be three weeks before all the drives were ready for the array at this rate.

 

I mean, I do plan on setting up and starting the array with the first four drives once they are ready and adding the rest of them as they complete the preclear process. However I don't want to be waiting weeks on end before this is all set up and the array is complete. 

 

So I was wondering, is there a way to abort / end the current preclear process after the first cycle completes and still retain the preclear signature (I think it's called)? I know that there's no way to do it exactly as when it completes the first cycle it will automatically begin the second cycle, but at what point am I safe to cancel the process and still have the drives classed as precleared and ready to be formatted?

If it's reading the disk (Pre or Post Reads) it's safe to stop the current preclear session and add it to the disk. If it's zeroing the disk, if you stop the session and add it to the array the disk will be cleared by Unraid again.

Share this post


Link to post
5 hours ago, gfjardim said:

If it's reading the disk (Pre or Post Reads) it's safe to stop the current preclear session and add it to the disk. If it's zeroing the disk, if you stop the session and add it to the array the disk will be cleared by Unraid again.

Because the disk will have already been zeroed by the preclear plugin, is that when you choose the option that "parity is still valid" or something that I've read online? Although I haven't tried to do this yet or seen this option myself.

 

So if I stop the preclear session in the pre or post-read step, and then create the array using those disks, I'm guessing it won't have the preclear signature? Whatever that is? What are the benefits / drawbacks of doing this and of the preclear signature?

 

All 4 of the drives are now in the post-read stage, two at about 85% and two at about 55% so if I'm going to follow your advice I need to end this process within the next day or so before it starts zeroing the drive again. 

Share this post


Link to post

I want to retrieve old pre-clear logs. Where are the logs stored on my server?

Share this post


Link to post

Every time I reboot my server, the Preclear plugin disappears as if it uninstalled itself (or something else uninstalled it). And I have to reinstall it again. Is this a known issue? Anybody else seeing this issue? It's not happening with any of my other plugins.

 

I'm on 6.9.0-beta30 with trial key.

Share this post


Link to post
Every time I reboot my server, the Preclear plugin disappears as if it uninstalled itself (or something else uninstalled it). And I have to reinstall it again. Is this a known issue? Anybody else seeing this issue? It's not happening with any of my other plugins.
 
I'm on 6.9.0-beta30 with trial key.
Please send me your Diagnostics file.

Enviado de meu SM-N970F usando o Tapatalk

Share this post


Link to post

Hi all, i have a question if any have the respond, it would be appreciated...

Q.   if you have 2 unraid server can you preclear your disk in one and put it in the other serve, will it be recognize or i will have to preclear again?

Share this post


Link to post
5 minutes ago, AndreB said:

can you preclear your disk in one and put it in the other serve, will it be recognize

This.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.