Dockers not seeing new files created within Unassigned Drive

[iPenguin02]

I have a VM running as my download pc saving all files to an unassigned drive. This drive acts as my storage before rename and sort them into the array. I mapped the unassigned drive to both my Emby and Nextcloud dockers. The current files in the drive appear after adding the drive. However, when new files are created in unassigned drive they don't show up in Emby or Nextcloud.

 

Both dockers are mapped to /mnt/disks/Storage

 

For the unassigned drive I have it 'Mounted' and 'Auto Mount' and 'Share' turned on.

 

I can't seem to figure out the issue. I tried unmounting then remounting the drive, but no luck.

Edited March 20, 2020 by iPenguin02

[itimpi]

What access mode have you set in the drive mappings in the docker template?    For UD drives you want it to be one of the ‘Slave’ modes.

 

[iPenguin02]

39 minutes ago, itimpi said:

What access mode have you set in the drive mappings in the docker template?    For UD drives you want it to be one of the ‘Slave’ modes.

 

It was Read/Write, but I tried both RW/Slave and RO/Slave, restarted the dockers and still doesn't show the new files. I tried readding the mapped drive as well. No luck.

 

I looked at the disk in the terminal and it doesn't show my new folders/files either. I did an ls to the folder below.

/mnt/disks/Storage/Downloads/Completed

Edited March 20, 2020 by iPenguin02

[Squid]

6 minutes ago, iPenguin02 said:

I looked at the disk in the terminal and it doesn't show my new folders/files either. I did an ls to the folder below.

 

Then how do you know that the file(s) are winding up on the unassigned device?

[iPenguin02]

6 minutes ago, Squid said:

Then how do you know that the file(s) are winding up on the unassigned device?

When I VNC into the VM, the files are in the drive which is the unassigned drive. Since its a windows vm its D:/Downloads/Completed

[trurl]

There is obviously a disconnect somewhere. If it isn't on the Unassigned Device, then your VM must not be looking at the Unassigned Device.

[Squid]

Or its a symlink or something within the VM.  Not sure how that gets handled over SMB.

[iPenguin02]

22 minutes ago, trurl said:

There is obviously a disconnect somewhere. If it isn't on the Unassigned Device, then your VM must not be looking at the Unassigned Device.

 

19 minutes ago, Squid said:

Or its a symlink or something within the VM.  Not sure how that gets handled over SMB.

 

I'm not sure either. I feel like its mapped correctly. I only have 1 unassigned drive.

 

The VM has a manual drive mapped to /dev/disk/by-id/ata-WDC_WD10EADS-11M2B3_WD-WMAV52309489

while my dockers are /mnt/disks/Storage/Downloads/Completed

[trurl]

6 minutes ago, iPenguin02 said:

The VM has a manual drive mapped to /dev/disk/by-id/ata-WDC_WD10EADS-11M2B3_WD-WMAV52309489

while my dockers are /mnt/disks/Storage/Downloads/Completed

That is not enough information for us to guess whether or not those are the same thing.

[iPenguin02]

6 minutes ago, trurl said:

That is not enough information for us to guess whether or not those are the same thing.

What other information would you need? I'm still new to Unraid and learning everyday so apologies.

[trurl]

I don't actually use any VMs, but are you sure you can even do it that way? Don't you at least have to specify a partition?

[trurl]

Just now, iPenguin02 said:

What other information would you need? I'm still new to Unraid and learning everyday so apologies.

For starters, the usual

 

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your NEXT post.

[JonathanM]

This sounds suspiciously like you passed the entire device to the VM, which now controls the content. If so, then unraid has no access to it.

 

If this device shows up in the VM's disk manager, it's not going to work like you want. It should have been mapped as a network drive in the windows VM.

[iPenguin02]

2 hours ago, trurl said:

For starters, the usual

 

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your NEXT post.

diagnostics-20200320-1550.zip

Attached

1 hour ago, jonathanm said:

This sounds suspiciously like you passed the entire device to the VM, which now controls the content. If so, then unraid has no access to it.

 

If this device shows up in the VM's disk manager, it's not going to work like you want. It should have been mapped as a network drive in the windows VM.

 

I did indeed pass the entire drive to the VM. I had no reason to map it as a drive. Is this the best practice? If so, I'll give this a try.

 

Drives purpose is for my downloads from qtorrent. any downsides to using a mapped drive to download vs passing the entire drive through?

[trurl]

If you pass the entire drive through, I think you would have to see if you can map it as a network share in Unassigned Devices, since Unraid can't use the drive if you give it to a VM.

 

Are you sure you even need a VM? There are a lot of dockers that people use for downloading.

[JonathanM]

Since you passed the drive through to the VM, it is, for all intents and purposes, in a separate computer from Unraid. If you want to access its content from Unraid, you will need to set it up as a network share IN THE VM, and connect to that share using Unassigned Devices.

 

Normal best practices for a VM would be to mount it in UD as a data drive, formatted to XFS, and if you want to write to it using another computer, be that a VM or physical machine, you would set UD to share it, and connect it as a network folder in that computer. The only exception I can think of is a gaming VM that refuses to allow content to reside on a network share.

 

I second trurl though, keeping a windows VM active for downloading seems like a waste of resources. A docker container can share CPU, RAM and disks with Unraid, giving and taking as needed. A VM keeps whatever resources its allocated hostage, as you've found with that disk.

[dlandon]

Your disk in mounting read only:

Mar 20 07:51:13 iPenguin-Server unassigned.devices: Mount drive command: /sbin/mount -t ntfs -o rw,auto,async,noatime,nodiratime,nodev,nosuid,umask=000 '/dev/sdd1' '/mnt/disks/Storage'
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Version 2017.3.23 integrated FUSE 27
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Mounted /dev/sdd1 (Read-Only, label "Storage", NTFS 3.1)
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Cmdline options: rw,noatime,nodiratime,nodev,nosuid,umask=000
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Mount options: rw,nodiratime,nodev,nosuid,allow_other,nonempty,noatime,ro,default_permissions,fsname=/dev/sdd1,blkdev,blksize=4096
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Global ownership and permissions enforced, configuration type 1
Mar 20 07:51:13 iPenguin-Server unassigned.devices: Mount warning: The disk contains an unclean file system (0, 0). Metadata kept in Windows cache, refused to mount. Falling back to read-only mount because the NTFS partition is in an unsafe state. Please resume and shutdown Windows fully (no hibernation or fast restarting.) 
Mar 20 07:51:13 iPenguin-Server unassigned.devices: Successfully mounted '/dev/sdd1' on '/mnt/disks/Storage'.
Mar 20 07:51:13 iPenguin-Server unassigned.devices: Adding SMB share 'Storage'.'

It can't be written to.

[iPenguin02]

7 hours ago, trurl said:

If you pass the entire drive through, I think you would have to see if you can map it as a network share in Unassigned Devices, since Unraid can't use the drive if you give it to a VM.

 

Are you sure you even need a VM? There are a lot of dockers that people use for downloading.

 

6 hours ago, jonathanm said:

Since you passed the drive through to the VM, it is, for all intents and purposes, in a separate computer from Unraid. If you want to access its content from Unraid, you will need to set it up as a network share IN THE VM, and connect to that share using Unassigned Devices.

 

Normal best practices for a VM would be to mount it in UD as a data drive, formatted to XFS, and if you want to write to it using another computer, be that a VM or physical machine, you would set UD to share it, and connect it as a network folder in that computer. The only exception I can think of is a gaming VM that refuses to allow content to reside on a network share.

 

I second trurl though, keeping a windows VM active for downloading seems like a waste of resources. A docker container can share CPU, RAM and disks with Unraid, giving and taking as needed. A VM keeps whatever resources its allocated hostage, as you've found with that disk.

Its mainly out of personal preference and habit. I came from using Windows Server and had things running using Hyper X. I lost all my data because of a single drive failure and other issues. I found Unraid and haven't looked back since. 

 

The VM serves as my downloader and my converter using Handbrake which I know have dockers for. The VM is always connected to NordVPN and is set to drop the connection if the VPN drops. I use VNC when I'm searching for torrents, I don't want my ISP snooping on my searches. I'm kinda picky when it comes to downloading, especially with my anime, thats why I haven't explored Sonarr or Radarr.

 

I'm open to suggestions. How do you guys search for torrents when using dockers? i'm interesting in knowing your methods since I'm still trying to piece everything together on how I should do things with Unraid.

 

6 hours ago, dlandon said:

Your disk in mounting read only:

Mar 20 07:51:13 iPenguin-Server unassigned.devices: Mount drive command: /sbin/mount -t ntfs -o rw,auto,async,noatime,nodiratime,nodev,nosuid,umask=000 '/dev/sdd1' '/mnt/disks/Storage'
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Version 2017.3.23 integrated FUSE 27
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Mounted /dev/sdd1 (Read-Only, label "Storage", NTFS 3.1)
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Cmdline options: rw,noatime,nodiratime,nodev,nosuid,umask=000
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Mount options: rw,nodiratime,nodev,nosuid,allow_other,nonempty,noatime,ro,default_permissions,fsname=/dev/sdd1,blkdev,blksize=4096
Mar 20 07:51:13 iPenguin-Server ntfs-3g[13565]: Global ownership and permissions enforced, configuration type 1
Mar 20 07:51:13 iPenguin-Server unassigned.devices: Mount warning: The disk contains an unclean file system (0, 0). Metadata kept in Windows cache, refused to mount. Falling back to read-only mount because the NTFS partition is in an unsafe state. Please resume and shutdown Windows fully (no hibernation or fast restarting.) 
Mar 20 07:51:13 iPenguin-Server unassigned.devices: Successfully mounted '/dev/sdd1' on '/mnt/disks/Storage'.
Mar 20 07:51:13 iPenguin-Server unassigned.devices: Adding SMB share 'Storage'.'

It can't be written to.

I think it is like jonathanm and trurl stated. because I have the drive passed through to the VM, the only way I can have it be seen by the docker is to mount it using UD. This places the drive in a read only state and I cannot toggle to off.

Edited March 21, 2020 by iPenguin02

[dlandon]

You don't want UD to mount a disk that is also passed through to a VM or Docker.  I'm not an expert on passing through disks to VM's and Dockers, but you can't pass it through to multiple systems.  Only one should manage the disk directly.  I think your best bet is to either pass it through to the VM and then SMB share it to the Dockers, or have UD mount it and share it SMB to all.

 

Maybe be a bit more specific on the VM and Dockers.  Windows VM, what Dockers?

[itimpi]

5 hours ago, iPenguin02 said:

I think it is like jonathanm and trurl stated. because I have the drive passed through to the VM, the only way I can have it be seen by the docker is to mount it using UD. This places the drive in a read only state and I cannot toggle to off.

If a drive has been passed through to a VM then it should NEVER also be mounted in Unassigned devices at the same time.   Doing so means that you have 2 different OS (Unraid and Windows) both thinking they have exclusive access to the drive and will not see any changes made by the other OS. This can easily lead to complete corruption of the contents.   The latest releases of UD have a pass-through option that you should set if passing the drive through to a VM as this will then stop you also trying to mount it at the same time.

[Squid]

You should definitely be using UD to mount an SMB share from the VM to access that drive.  Since everything is all stored locally, there's no performance hit.

[iPenguin02]

2 hours ago, Squid said:

You should definitely be using UD to mount an SMB share from the VM to access that drive.  Since everything is all stored locally, there's no performance hit.

How would one do this?

[Squid]

Share the drive / folder within the VM.  Within Unassigned devices, add a new SMB share

[iPenguin02]

4 minutes ago, Squid said:

Share the drive / folder within the VM.  Within Unassigned devices, add a new SMB share

I have the drive shared under the properties in the VM. I'm not familiar with adding a new SMB share using UD

[Squid]