vdisk Disappeared

[JorgeB]

Well then it does look like the vidsk is gone, someone or something must have deleted it.

[3x3q]

Is it possible that the Mover may have (for some reason) tried to move it from cache to disk and, because VM was running, it messed up the moving process? The vdisk data does not seem corrupted, since it's still booted and running perfectly.

 

It's almost as if the path to the vdisk .img file has been forgotten by Unraid, and the phantom .img file isn't being overwritten/corrupted by new data because it's still in use.

[JorgeB]

Mover cannot move files that are in use.

[3x3q]

4 minutes ago, JorgeB said:

Mover cannot move files that are in use.

Right, that's my point. The .img data still appears to be on the cache disk as it should be, but the path reference to that data is gone.

 

Unless the Mover utility is a flawless piece of code, my suspicion is that Mover for some reason thought it had successfully moved the .img from cache to array and removed the original reference to it.

[JorgeB]

Mover should fail to move any file in use, and it reports that in the log 

[3x3q]

Thanks for your input!

 

So what do you suspect is happening to people's vdisks, if we were to attempt to solve this?

[JorgeB]

This is not a common issue, this thread you revived is from 2019, if the mover could cause this I would expect this issue to be much more common.

[Kamero4K]

I mean.. that happend a year ago.. I dont know if it was my fault or unraid really did shady things.. 

BUT I KNOW MOVER HAD STRANGE BEHAVIOUR.

Maybe i just dont know enough about the functionallity of unraid cache but i can only recommend : 

DO NOT JUSE CACHE 

DO NOT USE MOVER

I never lost data since then.. but i also didnt add any other drives to the array.. We will see.

I will upgrade my storage for sure in the future.. When something happens.. i will leave it here.

[avman]

In case it helps, I ran into this issue where the /mnt/user/domains folder was gone with the vdisk1.img file with it for a VM. This was after an unclean shutdown (I forgot the Unraid password, needed to unplug and reset). I reluctantly started to make a new VM to replace it.

Then, through the magic of a clean reboot, it reappeared! 

So, as usual, before you panic, did you turn it off and back on again?

 

[3x3q]

36 minutes ago, avman said:

In case it helps, I ran into this issue where the /mnt/user/domains folder was gone with the vdisk1.img file with it for a VM. This was after an unclean shutdown (I forgot the Unraid password, needed to unplug and reset). I reluctantly started to make a new VM to replace it.

Then, through the magic of a clean reboot, it reappeared! 

So, as usual, before you panic, did you turn it off and back on again?

 

My VM .img file did not re-appear and the 1.5T space reservation for the .img file on cache cleared immediately as I shut down the VM, just as I feared it would.

 

I'm sure a file recovery application would have been able to help in my case, since only the path to the file was removed, but the 1.5T size was just too for me to deal with.

[Denisio]

I had just the same problem, VMs were all still running fine!
I searched the whole array and found that all VMs were stored in a randome Subfolder of my Documents Share.

The day before they were all sitting in the VM Share and Main Folder... very strange

[trurl]

4 hours ago, Denisio said:

I had just the same problem, VMs were all still running fine!
I searched the whole array and found that all VMs were stored in a randome Subfolder of my Documents Share.

The day before they were all sitting in the VM Share and Main Folder... very strange

Not sure what you mean by "Main Folder".

 

Easy to drag and drop things in the wrong place. I usually don't even try. Lots of other ways to copy/paste without dragging.

[3x3q]

3 minutes ago, trurl said:

Easy to drag and drop things in the wrong place.

Are you suggesting UNRAID can allow a user to move or delete a vdisk image while it's in use?

[trurl]

Not suggesting anything of the kind. The user described something that can't happen without enough details to figure out what did happen.

[3x3q]

Everybody in this thread is describing an issue which "cannot happen" if you're under the assumption that the glorious UNRAID operating system is entirely without flaw.

 

I guess it's just much easier to chalk it up to user error though. But even then, in my case, how was I supposed to have removed record of my vdisk image while it was running?

[trurl]

That other user was saying it had been moved to a different user share. Unraid definitely would not do that by itself.

 

For those where it actually disappeared, removing, reformatting the disk containing the vdisk seems to be what happened with one user.

 

Possibly filesystem corruption could account for some.

 

And an incorrect path setting could result in a vdisk that wasn't persistent since it wasn't on any actual storage.

 

 

[PeteAsking]

A reddit user here has had the same issue occur:

 

 

 

It also happened after a power loss, so while filesystem corruption is possible it seems like something additional must be needed for the failure to occur.

 

I also agree that its possible the mover has something to do with the scenario as it is directly manipulating files, possible during or around the time of power loss.

 

In linux it is possible to delete a file in use as it does not actually remove the file, it just reduces the inode count by 1, and when the inode = 0 then the file is freed up from the filesystem. This can explain why user @3x3q was able to have a running VM with no file (but the space was not freed until the process holding the file was ended). If 3x3q had used the command "lsof | grep <name of missing file.qcow> most likely he would have been able to find the file and copy it to a new location using the "cp" command from /proc. Its possible this may have saved the VM if he had done this before turning off the VM.

 

It would be interesting to see if the disks used by people with the issues are similar.

 

Just adding this here as the number of reports is somewhat higher than random chance would suggest it should be.

 

P

[JorgeB]

The mover does not move open files, in part why it's so slow, because it checks if a file is open before attempting to move it, for every file.

[PeteAsking]

I understand the intention of the mover, but as nothing is perfect it may be possible for an unimagined scenario to allow a file to move that should not have been possible. Ruling this scenario out without rigorous and detailed mathematical proof it cannot be possible ever seems strange to me.

Edited March 3, 2023 by PeteAsking

[JorgeB]

I see your point, but unless you find a way of reproducing the issue, I don't see how LT can look at it.

[3x3q]

2 hours ago, PeteAsking said:

In linux it is possible to delete a file in use as it does not actually remove the file, it just reduces the inode count by 1, and when the inode = 0 then the file is freed up from the filesystem. This can explain why user @3x3q was able to have a running VM with no file (but the space was not freed until the process holding the file was ended). If 3x3q had used the command "lsof | grep <name of missing file.qcow> most likely he would have been able to find the file and copy it to a new location using the "cp" command from /proc. Its possible this may have saved the VM if he had done this before turning off the VM.

Thanks so much for posting this dude! This command may be truly helpful when someone encounters an issue similar to mine.

 

At least with this topic on these forums, I've noticed a real authoritative hesitancy to explore with an open mind potential bugs and edge cases in programs. Instead, users are regurgitated at with a debunking which assumes that "programmed behavior == intended behavior".

[PeteAsking]

Not sure but another scenario I am thinking of as it happens sometimes after a power failure is that something has happened between the state of the machine when it was powered on and after it got rebooted due to a power failure.

 

I am wondering if a common issue on the forum where a bad cable causes a disk to drop off TEMPORARILY could be something like this:

 

1) Unraid is functioning normally

2) A bad cable causes a disk to drop off the array temporarily (this could occur after the mover begins moving the file and is mid way through moving?).

3) A mover is invoked and it moves a file from a cache drive to the array that is now enumerated. The file is removed from the cache and the file is assumed to be on the array.

4) Unraid is rebooted due to some event (power failure, user intervention etc).

5) The disk comes back online as it was a temporary fault. Unraid starts the array? Unsure about this part. Possibly required manual intervention here?

6) The changes that were being enumerated on the parity disk are lost as now all disks are available so there there is no enumeration taking place. The file that was moved to the array no longer exists.

 

Thoughts?

Edited March 3, 2023 by PeteAsking

[JorgeB]

5 minutes ago, PeteAsking said:

3) A mover is invoked and it moves a file from a cache drive to the array that is now enumerated. The file is removed from the cache and the file is assumed to be on the array.

I assume you mean emulated disk, in that case the disk would stay emulated after a reboot, you'd need to manually rebuild the disk to re-enable it, that requires: stop array, unassign the disk, start array, stop array, re-assign the disk.

 

Also a question, why is the mover trying to move a vdisk? By default the domains share is set to cache=prefer, the user would need to manually change to cache=yes, and why would the user do that? Just asking because I don't see the use case, you usually want vdisks to remain on cache, not be moved to the array.

 

 

 

 

[PeteAsking]

4 minutes ago, JorgeB said:

the user would need to manually change to cache=yes, and why would the user do that? J

 

 

It is interesting that you say this because if you download the diagnostics of the reddit users post I posted earlier on the domains.cfg file it states:

shareUseCache="yes"

I imagine as opposed to "prefer"

 

Unsure why.

[PeteAsking]

14 minutes ago, JorgeB said:

I assume you mean emulated disk, in that case the disk would stay emulated after a reboot, you'd need to manually rebuild the disk to re-enable it, that requires: stop array, unassign the disk, start array, stop array, re-assign the disk.

 

 

 

 

 

What if the usb disk was disconnected during the time for any reason this happened so unraid could not write to the usb that there had been an array interruption? What happens then - ie all is perfect, usb becomes inaccessible (same thing that drops disk?), issue occurs then reboot?

Edited March 3, 2023 by PeteAsking