April 6, 201115 yr If all disks pass the 'new' test, then just start the array and proceed as normal. If any disks fail the 'new' MBR test and also fail the 'old' MBR test, then treat them as disks which are new to unRAID, and go ahead and format them before starting the array. However, if any disk fails the 'new' test, but passes the 'old' test, then treat it as a 'dirty' unRAID disk, flag it as such, and do not allow the array to start. The user would then have to intervene and decide whether to use preclear or mkmbr. There - solved! In most cases all would be business as usual but, if one of these 'dirty' disks is detected, it will not be modified (formatted) automatically, but left for the user to decide how to resolve. If the user is unsure about what to do they will, hopefully, seek help here in the forum. Of course, it may be simpler, if not so aesthetically pleasing, just to revert to the 'old' test. I disagree strongly with "There - solved!" part. Currently unRAID does not protect any part of the disk outside the first partition. It should not care what is outside the protected region, specifically the first partition, if the disk serial id's match the known recorded disk serial ids. The user should not be forced to do anything to an existing data disk if it is written to outside of the protected region. There is no reason or purpose to being forced to always have a pristine signature MBR.
April 6, 201115 yr If all disks pass the 'new' test, then just start the array and proceed as normal. If any disks fail the 'new' MBR test and also fail the 'old' MBR test, then treat them as disks which are new to unRAID, and go ahead and format them before starting the array. However, if any disk fails the 'new' test, but passes the 'old' test, then treat it as a 'dirty' unRAID disk, flag it as such, and do not allow the array to start. The user would then have to intervene and decide whether to use preclear or mkmbr. There - solved! In most cases all would be business as usual but, if one of these 'dirty' disks is detected, it will not be modified (formatted) automatically, but left for the user to decide how to resolve. If the user is unsure about what to do they will, hopefully, seek help here in the forum. Of course, it may be simpler, if not so aesthetically pleasing, just to revert to the 'old' test. I disagree strongly with "There - solved!" part. Currently unRAID does not protect any part of the disk outside the first partition. It should not care what is outside the protected region, specifically the first partition, if the disk serial id's match the known recorded disk serial ids. The user should not be forced to do anything to an existing data disk if it is written to outside of the protected region. There is no reason or purpose to being forced to always have a pristine signature MBR. But what's the benefit of letting a user keep stuff in their MBR? How would that ever be useful?
April 7, 201115 yr But what's the benefit of letting a user keep stuff in their MBR? How would that ever be useful? I think that BRIT's point is that, provided that the first partition on the disk is assigned to, and usable by, unRAID, it should be possible to do anything you wish to areas outside of that partition. Specifically, if there is a second partition on the disk, then the MBR will not look like a pristine unRAID MBR, and current betas will then treat that disk as unformatted. With my suggested changes, unRAID would refuse to start. I guess that this is an issue that should be discussed in the public forum, and a concensus reached. I can see arguments on both sides - if the drive contents can be manipulated outside of the unRAID environment, then that can be prejudicial to the data integrity ... but then this would still be the case if you dual-boot the machine from a disk which is outside of the unRAID array. Let the discussions begin .......
April 7, 201115 yr This idea that unRAID wouldn't care too much about the structure of the disk, and would just parity protect the first partition has come up several times over the years. Logically it really shouldn't matter whether the drive is RFS, NTFS, or extX, unRAID shouldn't really have to care to parity protect the first partition. But, on the other hand, this is Tom's playgound, and he might reasonably require adherance to specific disk structure and format to minimize support issues. There may also be some lower level I/O monitoring that is below the partition level that is used to provide high performance parity handling, and that that logic assumes a single partition extending to the end of the disk. Tom has on the roadmap allowing any format to the cache disk. I personally am fine with RFS and following his disk structure and formatting requirements for array disks. The expression "pick your battles" comes to mind.
April 7, 201115 yr The expression "pick your battles" comes to mind. Indeed! Perhaps we need an expert/novice setting to protect the innocent, but allow flexibility for the experienced? I agree with you - I have no requirement to run anything outside of the unRAID environment - I have other machines to do that. If I do require to boot an alternate system (for maintenance purposes?) then I will do that with a removable USB or eSATA drive.
April 7, 201115 yr Indeed! Perhaps we need an expert/novice setting to protect the innocent, but allow flexibility for the experienced? I'd rather have 3T drive support.
April 7, 201115 yr 3TB? I've only just managed to obtain a 2TB drive here ... and that was by special order! But, yet again, I find that I'm in complete agreement with you - let's have new, useful, features and keep unRAID in a strictly protected environment. We can already mount other drives outside of the unRAID array, we can install third party applications and utilities within the array - do we really need any more?
April 7, 201115 yr This idea that unRAID wouldn't care too much about the structure of the disk, and would just parity protect the first partition has come up several times over the years. Logically it really shouldn't matter whether the drive is RFS, NTFS, or extX, unRAID shouldn't really have to care to parity protect the first partition. But, on the other hand, this is Tom's playground, and he might reasonably require adherence to specific disk structure and format to minimize support issues. The issue has been described by Tom previously is not in protecting the first partition, but in re-constructing the new disk's MBR when a disk is replaced/upgraded. Currently, unRAID does NOT "protect" the MBR by parity since its format is exactly known for an existing disk and MUST be different for a replacement disk if the replacement disk is larger then the original (it cannot be re-constructed from parity when a disk is replaced with a larger one) Again, that "different" MBR is exactly known and can be programatically created based on the size of the disk. That all falls apart when you put other partitions, or non-standard partitions on the same physical disk. In those cases, the MBR cannot be re-created programatically based on the size of the disk. In order to allow arbitrary partitioning, there must be a process where the entire MBR and the first partition are able to be re-created. It could potentially be done by keeping a copy of each disk's MBR on the flash drive, but even that might be an issue of keeping it current with experienced users messing around with the partitioning, and will be complicated even more when we can no longer use MBR style partitions and need to use the new "GUID Partition Table" for disks larger than 2.2TB. I suspect that your desire to put any partitioning scheme in place you desire will not occur (if it occurs at all) until after 3TB drives are accommodated. Joe L.
April 7, 201115 yr The issue has been described by Tom previously is not in protecting the first partition, but in re-constructing the new disk's MBR when a disk is replaced/upgraded. .... That all makes perfect sense.
April 7, 201115 yr In any case, even if the MBR issue is handled in some way, if you were to create a smaller "first" partition, followed by an un-protected second partition it greatly complicates replacing a disk with a larger one. (Or rather, prohibits unRAID from automatically increasing the size of the existing partition to use the newly available space.) For that reason, I personally would not hold my breath for multiple partition support. I think I read on the roadmap the ability to have an un-protected disk in the array. (where unRAID would let it participate in the user-shares) That would allow the add-ons a place to put their temp files and still allow normal array expansion in the protected array. All that said, most of this discussion belongs in the roadmap forum and has nothing to do with 5.0beta6a. We really need to get back on topic. The relaxation of the beta6a warning and improved instructions for users migrating to it "should" keep the calls for help to a minimum. If users were to run the script I suggested, and could report on the results if something unexpected occurred, we should be fine. One new scenario occurred recently that resulted in an MBR - unknown on a cache drive was detected during an upgrade to 5.0beta6a. It was partitioned and formatted by the unRAID user outside of unRAID. Up until now, unRAID was happy to mount its first partition as the cache drive. Now unRAID is considering it MBR - unknown because of the vestiges of boot-loader code in the MBR from when the disk was used in a stand alone NAS. (It expected the MBR to be mostly zeros and did not expect to see the non-zero bytes of the boot loader.) Joe L.
April 7, 201115 yr I guess I don't follow what the importance of the other bits in the first sectors and why they have to be 0. Read the MBR and find a partition that goes from sector 63 or 64 to the end of the disk. What other checks are done on the data in the first 62 or 63 sectors to ensure it is a valid unRAID data? Peter
April 7, 201115 yr I think I read on the roadmap the ability to have an un-protected disk in the array. (where unRAID would let it participate in the user-shares) That would allow the add-ons a place to put their temp files and still allow normal array expansion in the protected array. Isn't that what the cache drive is for?
April 7, 201115 yr I guess I don't follow what the importance of the other bits in the first sectors and why they have to be 0. Read the MBR and find a partition that goes from sector 63 or 64 to the end of the disk. What other checks are done on the data in the first 62 or 63 sectors to ensure it is a valid unRAID data? Thank you. This is exactly my point. The rest of the MBR data should not matter one bit for a drive already in the array. That can be determined by checking drive serial ids. The rest of the MBR data only matters for drives not previously in the array. It should only be used by unRAID (emHttp) to determine if the disk is precleared so it can skip the lengthy clearing process. Yet because of the overly pedantic checks in 5.0beta6a, before each and every single array start I need to use mkmbr on drives unRAID knows were part of the array previously because of a 6 byte difference. My array only ever restarts when I install or patch the Linux Kernel which means a run of LILO which means the 6 byte difference in all drives' MBRs.
April 8, 201115 yr So, I recently upgraded from 4.7 to 5.0b6a. Things seem to have gone well by following the items in the release notes. I brought the array online, and ran the New Permissions script. I can see that the permissions on my files have been changed. However, only one of my 6-7 user shares is visible. The rest are absent. If I navigate to my drives, I see all of the folders present. My user shares are enabled in the settings. Any idea what the problem could be? Also, it appears the the existing user share does not want to unmount normally. If I use the webGUI it just sits there saying Retry unmounting user share(s)... over and over. I have to use the unMenu "Power Down" button to get it to turn off. I can't figure out what's going on... Edit: And after two reboots, they're somehow back. I don't know what the problem was, but it looks like it was upgrade process-related. I'll chime back in if they disappear on subsequent reboots, but everything appears normal now. I'm not sure what the second reboot did to make a difference.
April 8, 201115 yr So, I recently upgraded from 4.7 to 5.0b6a. Things seem to have gone well by following the items in the release notes. I brought the array online, and ran the New Permissions script. I can see that the permissions on my files have been changed. However, only one of my 6-7 user shares is visible. The rest are absent. If I navigate to my drives, I see all of the folders present. My user shares are enabled in the settings. Any idea what the problem could be? Also, it appears the the existing user share does not want to unmount normally. If I use the webGUI it just sits there saying Retry unmounting user share(s)... over and over. I have to use the unMenu "Power Down" button to get it to turn off. I can't figure out what's going on... Edit: And after two reboots, they're somehow back. I don't know what the problem was, but it looks like it was upgrade process-related. I'll chime back in if they disappear on subsequent reboots, but everything appears normal now. I'm not sure what the second reboot did to make a difference. Thanks Gizmo for posting your experience. Please post your syslogs (I believe that powerdown saves a copy to your flash prior to shutting down). The syslogs may have clues to help Tom figure out what went wrong with your upgrade. Glad things are working well now.
April 8, 201115 yr This idea that unRAID wouldn't care too much about the structure of the disk, and would just parity protect the first partition has come up several times over the years. Logically it really shouldn't matter whether the drive is RFS, NTFS, or extX, unRAID shouldn't really have to care to parity protect the first partition. But, on the other hand, this is Tom's playground, and he might reasonably require adherence to specific disk structure and format to minimize support issues. The issue has been described by Tom previously is not in protecting the first partition, but in re-constructing the new disk's MBR when a disk is replaced/upgraded. Currently, unRAID does NOT "protect" the MBR by parity since its format is exactly known for an existing disk and MUST be different for a replacement disk if the replacement disk is larger then the original (it cannot be re-constructed from parity when a disk is replaced with a larger one) Again, that "different" MBR is exactly known and can be programatically created based on the size of the disk. That all falls apart when you put other partitions, or non-standard partitions on the same physical disk. In those cases, the MBR cannot be re-created programatically based on the size of the disk. In order to allow arbitrary partitioning, there must be a process where the entire MBR and the first partition are able to be re-created. It could potentially be done by keeping a copy of each disk's MBR on the flash drive, but even that might be an issue of keeping it current with experienced users messing around with the partitioning, and will be complicated even more when we can no longer use MBR style partitions and need to use the new "GUID Partition Table" for disks larger than 2.2TB. I suspect that your desire to put any partitioning scheme in place you desire will not occur (if it occurs at all) until after 3TB drives are accommodated. Joe L. This does not seem at all an insurmountable problem. The following changes would be needed IMO. (Note that I'm not advocating this as a priority, but don't think it would be hard): 1 - Record the partion1 size to the super.dat file what a disk is added to the array. In that way unRAID would know the size of the partition should it ever need to rebuild the disk. 2 - unRAID would not repartition a disk on add, unless the disk lacked a partiton table. It would simply use the partition already created. 3 - Add a configurable paramter with option to either expand (when rebuilding) to full drive size of the replacement disk, or to restore to the exact same size partition size (default to expand). I think that's basically what would be needed to support partitioned disks in the array.
April 8, 201115 yr So, I recently upgraded from 4.7 to 5.0b6a. Things seem to have gone well by following the items in the release notes. I brought the array online, and ran the New Permissions script. I can see that the permissions on my files have been changed. However, only one of my 6-7 user shares is visible. The rest are absent. If I navigate to my drives, I see all of the folders present. My user shares are enabled in the settings. Any idea what the problem could be? Also, it appears the the existing user share does not want to unmount normally. If I use the webGUI it just sits there saying Retry unmounting user share(s)... over and over. I have to use the unMenu "Power Down" button to get it to turn off. I can't figure out what's going on... Edit: And after two reboots, they're somehow back. I don't know what the problem was, but it looks like it was upgrade process-related. I'll chime back in if they disappear on subsequent reboots, but everything appears normal now. I'm not sure what the second reboot did to make a difference. Thanks Gizmo for posting your experience. Please post your syslogs (I believe that powerdown saves a copy to your flash prior to shutting down). The syslogs may have clues to help Tom figure out what went wrong with your upgrade. Glad things are working well now. Well, I rebooted and my shares disappeared again. I haven't been able to link it to any particular item in the syslog, but I'm not an expert. I've attached a zip of all of my syslogs. Probably the most recent two will be the most helpful (the current syslog-2011-04-07 with only one share visible and syslog-20110407-234436 that had all shares visible). Also note that when my user shares were populated correctly, the array would unmount cleanly every time. This is not the case when I have missing user shares. When that happens, the webGUI simply reports umount retries. Edit: It's late and I have to get off to bed, but it just occurred to me what is probably happening. The user share that always shows up is Backup. This share is used with CrashPlan, which I have starting automatically in my Go script after 30 seconds. I started my array before CrashPlan was able to execute, and all my user shares show up. I wonder if it's somehow reserving the mount point, causing unRaid's automatic user share mount process to fail (because the location already exists) if I wait to start the array until after the CrashPlan engine has started. Since 5.0b6a can't automatically start the array, I'll just have to remember to start it manually when I reboot, or change CrashPlan so it runs manually. I'll do some more investigation tomorrow to see if this hunch is correct. Archive_2.zip
April 9, 201115 yr Well I thought I would be adventurous tonight and test an upgrade from 4.7 to the 5B6a on my main system. Of my seven assigned drives, 4 show MBR: unknown. <sigh> Attached is a bunch of info. The first disk, sda, is my parity drive. The other three are data drives. All 4 are attached via Sata ports on the motherboard. Three other data drives, connected via my RocketRaid 2300's are showing the 4k Alligned... Hopefully someone can help. If you need any other info, let me know! Shawn status.txt syslog.txt disk_mbr_debug.txt
April 9, 201115 yr Well I thought I would be adventurous tonight and test an upgrade from 4.7 to the 5B6a on my main system. Of my seven assigned drives, 4 show MBR: unknown. <sigh> Attached is a bunch of info. The first disk, sda, is my parity drive. The other three are data drives. All 4 are attached via Sata ports on the motherboard. Three other data drives, connected via my RocketRaid 2300's are showing the 4k Alligned... Hopefully someone can help. If you need any other info, let me know! Shawn This information is very valuable... many thanks. Tell us the motherboard/disk controller cards involved. Did any of these disks have an HPA that you subsequently removed? Joe L.
April 9, 201115 yr Well I thought I would be adventurous tonight and test an upgrade from 4.7 to the 5B6a on my main system. Of my seven assigned drives, 4 show MBR: unknown. <sigh> Attached is a bunch of info. The first disk, sda, is my parity drive. The other three are data drives. All 4 are attached via Sata ports on the motherboard. Three other data drives, connected via my RocketRaid 2300's are showing the 4k Alligned... Hopefully someone can help. If you need any other info, let me know! Shawn, this is from your syslog (twice as you have two cards): Apr 9 00:06:47 Tower kernel: sata_mv: Highpoint RocketRAID BIOS CORRUPTS DATA on all attached drives, regardless of if/how they are configured. BEWARE! Apr 9 00:06:47 Tower kernel: sata_mv: For data safety, do not use sectors 8-9 on "Legacy" drives, and avoid the final two gigabytes on all RocketRAID BIOS initialized drives.
April 9, 201115 yr Well I thought I would be adventurous tonight and test an upgrade from 4.7 to the 5B6a on my main system. Of my seven assigned drives, 4 show MBR: unknown. <sigh> Attached is a bunch of info. The first disk, sda, is my parity drive. The other three are data drives. All 4 are attached via Sata ports on the motherboard. Three other data drives, connected via my RocketRaid 2300's are showing the 4k Alligned... Hopefully someone can help. If you need any other info, let me know! Shawn, this is from your syslog (twice as you have two cards): Apr 9 00:06:47 Tower kernel: sata_mv: Highpoint RocketRAID BIOS CORRUPTS DATA on all attached drives, regardless of if/how they are configured. BEWARE! Apr 9 00:06:47 Tower kernel: sata_mv: For data safety, do not use sectors 8-9 on "Legacy" drives, and avoid the final two gigabytes on all RocketRAID BIOS initialized drives. Since the RocketRaid BIOS is not being used to initialize drives, and unRAID does not use sectors 8 and 9 on any disk, he is "probably" ok. Joe L.
April 9, 201115 yr Are parity "sync" when you start the array for first time after upgrade to 5b6a and regular parity "check" one and the same thing speed-wise.
April 9, 201115 yr This information is very valuable... many thanks. Tell us the motherboard/disk controller cards involved. Did any of these disks have an HPA that you subsequently removed? Joe L. This is a montherboard from an HP desktop - it is an Asus OEM m2n68-la. Here is the link at HP for the motherboard. - NVIDIA GeForce 6150SE nForce 430 I know for a fact the 2TB parity disk does not have HPA, was purchased brand new specifically for this task. The other three "might" but I don't think so. I typically don't have GB motherboards, but the most recent one in my gaming system is. It is possible that one of the other disks may have at some point been in that system, but even then, most likely not as the boot disk. And yes, I know about the possible issues with the RocketRaid, but as Joe indicated, I "should" be ok. But to be safe, I have a 20GB minumum free space on all disks hanging off those controllers. But, the MBR:Unknown disks are actually the ones connected to the onboard sata connectors. If you need me to gather any more info, let me know. System sitting here idling away. Shawn
April 9, 201115 yr Here is another unknown MBR after installing 5.0b6a over 5.0b4. It's a WD EARS20, connected to an Adaptec 1430SA, never used in an other system. But on earlier betas. root@Atlas:~# sfdisk -luS /dev/sdc Disk /dev/sdc: 243201 cylinders, 255 heads, 63 sectors/track Warning: The partition table looks like it was made for C/H/S=*/1/0 (instead of 243201/255/63). For this listing I'll assume that geometry. Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/sdc1 63 3907029167 3907029105 83 Linux /dev/sdc2 0 - 0 0 Empty /dev/sdc3 0 - 0 0 Empty /dev/sdc4 0 - 0 0 Empty root@Atlas:~# cat /sys/block/sdc/size 3907029168 root@Atlas:~# dd status=noxfer count=1 if=/dev/sdc | od -Ad -t x1 1+0 records in 1+0 records out 0000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * 0000432 00 00 00 00 00 00 00 00 3c a2 5f 38 00 00 00 00 0000448 00 00 83 00 00 00 3f 00 00 00 71 88 e0 e8 00 00 0000464 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * 0000496 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa 0000512
April 9, 201115 yr A quick analysis of what we've seen so far: 1) There are some disks which appear to have a genuine bootloader written in the first 446 bytes of the mbr. At some stage these disks have been made to be bootable. The first 446 bytes of the mbr are reserved for the bootloader. 2) There are some disks which genuinely have more than one partition (a subset of these will be disks which had (or have) HPA). 3) There are some disks which appear to have a 32 bit value written two bytes before the end of the bootloader area (in other words, the value is double-word aligned in the last possible position of the bootloader area). Now, the first two cases are explicable and not very surprising, or out of the ordinary. Something in the history of the disk has genuinely, and legitimately, put the data in those positions. The third case is the one which cannot, at the moment, be explained. beelyte is adamant that his affected disk has never been used in any environment other than unRAID (but including some earlier betas). However, it is attached to an add-on controller. The information from Whaler_99 is, perhaps, even more interesting, or strange. He has four disks with the spurious(?) 32 value at the end of the bootloader area (ignoring his sde, which will be his 1GB flash drive with genuine boot loader). All four disks are attached to mobo ports (the drives on add-on cards are not afflicted). There is a strange pattern (or similarity) in the values appearing on each of his four drives: 81114ede, 81114edc, 81114ed2 and 81114ed0 (assuming little-endian) on sda, sdb, sdc and sdd respectively. These four disks have three different sizes - the values of interest do not appear to be related to disk size. It may be interesting to note that the genuine bootloader on the flash drive has 6 bytes of data written in the very last six bytes of the bootloader area, following a number of null bytes. The burning question, of course, is how did these four bytes (at 440 - 443 of the mbr) come to be non-zero. Possibilities which come to mind are: 1) These disks did, once upon a time, have a boot loader (although beelyte would suggest not) but when unRAID (or preclear) initialised these drives it failed to zero these four bytes. 2) Some past version of unRAID (or preclear) spuriously wrote these values during disk initialisation. 3) Some BIOS (or system?) code has made the assumption that the full bootloader area is never used, and it can grab the last six bytes for its own purpose. Can anyone else come up with any further observations/analysis/possible explanations for what we're seeing? I'm stopping here, because it is after 2am, and I need to sleep).
Archived
This topic is now archived and is closed to further replies.