Lost a lot of data during unBALANCE and XFS upgrade. Please help explain why.

[volcs0]

Edit: SOLVED - See below

 

I just finished changing six ReiserFX drives to XFS by using unBALANCE to copy data around, serially formatting the drives in the process. It took about a week for my unRAID (8,4,3,3,3,3 = 24 Tb). The process went smoothly, and the unBALANCE screen showed the progress for each drive. Each drive completed to 100% copied, and I confirmed that the correct amount of data had been transferred each time. I used Copy and not Move, since I was told that was faster. I did not verify the copies, however. I triple-checked every move, every format, every thing I did. I went back through the unBALANCE reports and confirmed the right sequence and that everything was moved, according to the output. 

 

Yet, there is a large amount of data missing. Probably 1-2Tb, maybe more. I'm still sorting it out. So far, the only thing I can identify is TV shows, but it's easy to see those, because episodes are missing. A spot check of movies, music, and home videos look OK, but since I don't keep a master list of everything, I can't fully confirm. 

 

I'm not worried about the loss of content. The home videos are backed up on Crashplan, and there's nothing in Movies of TV that can't be acquired again.

 

But I'd just like to figure it out.

 

I know the most obvious explanation is that I made a mistake and screwed something up... forgot to tick a box in unBALANCE, formatting the wrong drive, etc. But I have reconfirmed the sequence as best I can, and it looks good. Specifically, the unBALANCE reports show that TV was selected for each copy. 

 

Other info: I used Fix Docker Perms twice during the process. Once at the outset and once when I was prompted to.

I kept all Dockers off. I don't use VMs. I prevented the Mover from running.

 

Again, this is merely an inconvenience. I know that unRAID is not a backup strategy, and I do not back up Movies/TV as a rule, only home videos. Going forward, maybe I'll try to at least keep a catalog of everything I have, in case I lost another chunk of data.

 

Other thoughts are appreciated.

 

Thanks.

 

Edit: SOLVED. Dumb mistake by me (see below). Added a disc to help with the file movement and didn't add that disc to the shares. So glad that (1) I didn't erase anything, and (2) it was figured out before I spent a lot of time recovering the files.

 

Thanks!!!

 

Edited May 11, 2020 by volcs0

[sota]

did you search the actual disk mount points and not the aggregated shares for the missing data?

[volcs0]

2 hours ago, sota said:

did you search the actual disk mount points and not the aggregated shares for the missing data?

OK. This is very embarrassing. But you were right to have me look more closely at the discs and not the mounted shares.

When I started the process, I added a disc to give me enough room to move the files around.

I had no idea that the new disc would not be added to the shares automatically.

So, there was Disc 6, filled with content (and with /TV, /Movies, etc.) but not being included in the shares. 

I stopped the array, added it to the global sharing settings, and now the files are back!!

 

Thanks so much for your help. You really saved me a lot of heartache (and bandwidth).

 

[itimpi]

If you do not explicitly specify any include/exclude disks at the Global Share level then all drives ARE automatically included.

 

[sota]

Glad it was something simple.

I had a feeling based on your description of being thorough in checking your unBALANCE logs and actual transactions, that it was something like this.

[volcs0]

9 hours ago, itimpi said:

If you do not explicitly specify any include/exclude disks at the Global Share level then all drives ARE automatically included.

 

But, when you add a new drive to the array, any existing shares will keep their existing drives. When you go into the share and select inclusion, the new drive is not available for inclusion. It is not until you (1) stop the array, and (2) go to global sharing and add the new disc, and (3) restart the array, that the new drive can be included in the share. Unless I'm doing it wrong.

 

 

[itimpi]

59 minutes ago, volcs0 said:

But, when you add a new drive to the array, any existing shares will keep their existing drives. When you go into the share and select inclusion, the new drive is not available for inclusion. It is not until you (1) stop the array, and (2) go to global sharing and add the new disc, and (3) restart the array, that the new drive can be included in the share. Unless I'm doing it wrong.

 

 

If you do not specify any Values for the include/exclude drives at the Global level, then all drives are available for use with User Shares.   It is only when you restrict the drives at the Global level that they are not automatically offered at the user Share level.