Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

[wacko37]

10 hours ago, dlandon said:

You should report this to the ZFS Master plugin developer.  They will need to make a change to handle this situation.

 

Thanks for the speedy reply and as always a thorough explanation. TBH I have just realised how much continued work goes into maintaining such plugins/dockers. I can see you are always way ahead of the a-game (6.13) making sure the transition is a smooth 1. SO THANKYOU for all that you do...... can't say that enough.

 

10 hours ago, dlandon said:

There is an additional option in UD Settings to upgrade a ZFS disk in this situation.  That option shows in 6.13 and above, so a user can upgrade the pool if they want with the warning that it would not be backwards compatible.

 

The change to ZFS has been a step learning curve for myself, now I'm somewhat worried about this new transition with 6.13.

 

Will all existing Unraid ZFS pools be automatically upgraded with the new features when updating to 6.13, and any UD zfs disks would then require a manually update to be compatible. Will any data be lost during these upgrade?

[dlandon]

3 minutes ago, wacko37 said:

Will all existing Unraid ZFS pools be automatically upgraded with the new features when updating to 6.13, and any UD zfs disks would then require a manually update to be compatible. Will any data be lost during these upgrade?

Not sure yet how this will be handled with the Array, but I'm sure consideration will be given to prevent incompatible disk issues.  We've paused a bit on 6.13 and are concentrating on 6.12.5.

 

In UD your disk formatted in 6.12 will be mountable in 6.13 and all data preserved.  The issue is formatting a disk in 6.13.  Without the compatibility setting to legacy, the disk would not be mountable in 6.12.  Going forward is not any issue at all and disks formatted in earlier versions will always be mountable and work without any issues.  UD gives you the option to upgrade the disk if you choose and lets you know it will not be mountable in an earlier version of Unraid.  UD does not do an upgrade, you have to specifically do that.

 

The upgrade of a ZFS disk is only to enable some new features in ZFS.  Disks do not have to be upgraded to work in 6.13.  The issue is that an upgraded disk will not be backwards compatible and won't mount in 6.12.

 

The bottom line is you should never lose data, but an upgraded disk in 6.13 would not mount in 6.12.  It will always work in 6.13 and higher versions of UD.

[wacko37]

11 hours ago, dlandon said:

In order to keep backwards compatibility, UD now formats all ZFS disks with the compatibility set to legacy.  Because of this, the 'zpool status' shows different information that the ZFS Master plugin is probably not expecting.

 

Out of curiosity is there a way to downgrade UD to 2023-10.08 before the legacy update? This way I can continue until both plugins are compatible again/

[dlandon]

1 hour ago, wacko37 said:

 

Out of curiosity is there a way to downgrade UD to 2023-10.08 before the legacy update? This way I can continue until both plugins are compatible again/

Downgrading UD won't fix it.  It's the format on the disk.  While the disk is mouted go to a command line and run this command:

zpool upgrade tank

That will upgrade it to 6.12 and you'll be good to go.

[wacko37]

3 minutes ago, dlandon said:

zpool upgrade tank

That will upgrade it to 6.12 and you'll be good to go.

Absolute legend! - I'm back in business for now.....lol

[sustained-chop7918]

Good day,

 

New question:

 

Regarding Unassigned Devices to mount remote shares.

 

I have a Drobo that I backup files from Unraid to nightly. When I turn on the Unraid after powering it off or rebooting it, the shares don't automatically mount. The option is greyed out until I ping the IP of the Drobo.

 

Is there a way to run a startup script to ping the Drobo or run it before trying to mount the drives so they will seamlessly automount? 

 

This seems to be a weird situation and I couldn't find anyone else with the same issue.

 

Thanks!

[dlandon]

15 minutes ago, sustained-chop7918 said:

Good day,

 

New question:

 

Regarding Unassigned Devices to mount remote shares.

 

I have a Drobo that I backup files from Unraid to nightly. When I turn on the Unraid after powering it off or rebooting it, the shares don't automatically mount. The option is greyed out until I ping the IP of the Drobo.

 

Is there a way to run a startup script to ping the Drobo or run it before trying to mount the drives so they will seamlessly automount? 

 

This seems to be a weird situation and I couldn't find anyone else with the same issue.

 

Thanks!

UD does ping remote servers before trying to mount remote shares.  Your Unraid, network, or Drobo probably needs more time to settle.  Go to the UD Settings and set the 'Remote Share Mount Wait Time' to a larger value.  UD will give more time for the remote server to come on-line.  Click on the 'Help' icon and you'll se more of what the setting does.

 

If that doesn't work, post diagnostics.

[sustained-chop7918]

20 minutes ago, dlandon said:

UD does ping remote servers before trying to mount remote shares.  Your Unraid, network, or Drobo probably needs more time to settle.  Go to the UD Settings and set the 'Remote Share Mount Wait Time' to a larger value.  UD will give more time for the remote server to come on-line.  Click on the 'Help' icon and you'll se more of what the setting does.

 

If that doesn't work, post diagnostics.

I saw that. I suppose increasing to 60 seconds and seeing if that will work is worth a try  I'll need to wait until no one is using the system first.

[jademonkee]

On 3/29/2023 at 12:26 PM, jademonkee said:

Hi there. I assume that this is the best part of the forum to post this, but apologies if not.

I use UD to trigger a backup script when I plugin one of two USB hard drives. One I plug in every week, and another I do sporadically.

Every so often when I connect a disk and it kicks off the script, it begins to instead copy EVERYTHING.

It happened again yesterday when I plugged in the sporadic hard drive, so I cancelled the script through the UI's "Abort" button, but it seemed that rsync kept running - not knowing what else to do I rebooted the server, forcing a parity check on reboot 🙄

Today I ran my weekly backup and it ran fine. But then I plugged in the sporadic disk again, and it picked up where it left off yesterday, and continues to copy over everything. So that's about 1.5TB to be copied, which not only takes forever, but makes the disk get hot hot hot, so this is not ideal.

Any idea why this could be happening? Could it somehow (though I don't see how) be related to daylight savings starting last Sunday, and this being the first time I've attached the disk since then? (FWIW, the server time updated automatically, and I don't use the disk on any other machine, so I don't see how... but worth pondering). Or is there something silly in my script?

 

The script that fires is as follows (I nabbed from either this forum or somewhere else on the internet - apologies to the original author):

[script removed for space: see original post]

Thanks for your help!

So, the same thing happened again today: I plugged in my 'sporadic' backup disk and it started copying everything again. I went back to find my old post to see what the solution was, and I couldn't help but notice that I mentioned in my original post that this happened the first time plugging in the disk since daylight savings time started.

Well, this is actually the first time I've plugged in the disk since daylight savings time ended and here we are with it copying over everything again.

I note that it doesn't happen with my other backup disk, which is formatted in ext4.

Could this be related to the disk being exfat and the file timestamps changing relative to DST?

Is the solution to run my server as UTC, or is the solution to modify my script?

If it's running the server in UTC, what are the drawbacks? I'm assuming I can still choose timezones in Docker apps, yes (my Squeezeboxes are also my clocks)?

Thanks for your help and input.

 

EDIT: I note that this page: https://askubuntu.com/questions/1323668/one-volume-did-not-go-dst-with-the-rest-of-the-system

Mentions that you can mount disks using the option:

tz=UTC

How would I go about adding that option to UD for this disk?

Edited November 15, 2023 by jademonkee
Added possible tz=UTC solution

[AeroMaestro]

Hey unassigned folks.

 

I have an external HD.  Two partitions.  First one is APFS, second is HFS+.  I've successfully gotten Unassigned Devices Plus to work and setup a script so that when I plug in the drive it automatically backs up some of my Shares to the HFS+ partition.  

 

That worked once or twice, but now every time I plug in the external drive that HFS+ partition just mounts as "Read Only" and all the rsync commands in the script fail, showing "Read-only file system". 

I've been searching like crazy, but all I can find are lots and lots of posts in the Unraid forums with people having a similar issue, and the mods telling them to repost it here. 

Any advice? 

unraid-diagnostics-20231116-1122.zip

[dlandon]

1 hour ago, AeroMaestro said:

Hey unassigned folks.

 

I have an external HD.  Two partitions.  First one is APFS, second is HFS+.  I've successfully gotten Unassigned Devices Plus to work and setup a script so that when I plug in the drive it automatically backs up some of my Shares to the HFS+ partition.  

 

That worked once or twice, but now every time I plug in the external drive that HFS+ partition just mounts as "Read Only" and all the rsync commands in the script fail, showing "Read-only file system". 

I've been searching like crazy, but all I can find are lots and lots of posts in the Unraid forums with people having a similar issue, and the mods telling them to repost it here. 

Any advice? 

unraid-diagnostics-20231116-1122.zip 131.22 kB · 0 downloads

The hfs+ partition was not unmounted cleanly:

Nov 16 07:33:45 Unraid unassigned.devices: Mounting partition 'sdg1' at mountpoint '/mnt/disks/WD_easystore_264D-part1'...
Nov 16 07:33:45 Unraid unassigned.devices: Mount cmd: /usr/bin/apfs-fuse -o uid=99,gid=100,allow_other,vol=0 '/dev/sdg1' '/mnt/disks/WD_easystore_264D-part1'
Nov 16 07:33:45 Unraid unassigned.devices: Mounting partition 'sdg2' at mountpoint '/mnt/disks/Cold_Storage'...
Nov 16 07:33:45 Unraid unassigned.devices: Mount cmd: /sbin/mount -t 'hfsplus' -o force,rw,users,umask=000 '/dev/sdg2' '/mnt/disks/Cold_Storage'
Nov 16 07:33:45 Unraid kernel: hfsplus: Filesystem was not cleanly unmounted, running fsck.hfsplus is recommended.  mounting read-only.

That's why the partition was mounted read only.  Click on the file sysrtem check (check mark) next to the partition mount point and see if UD can fix the partition.

 

In the future, be sure to unmount the device properly from UD and the Mac you use it with.

[AeroMaestro]

3 hours ago, dlandon said:

Click on the file sysrtem check (check mark) next to the partition mount point and see if UD can fix the partition.

 

In the future, be sure to unmount the device properly from UD and the Mac you use it with.

 

Thanks a bunch.  I'm working remotely today, so I've only tried the checkmark so far, and the fsck gives me this:

FS: hfsplus

Executing file system check: /usr/sbin/fsck.hfsplus -l '/dev/sdg2' 2>&1
** /dev/sdg2 (NO WRITE)
** Checking HFS Plus volume.
** Checking Extents Overflow file.
** Checking Catalog file.
** Checking Catalog hierarchy.
** Checking Extended Attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume Cold Storage appears to be OK.

So UD+ wasn't able to fix the problem, but when I get home I'll plug it into the mac and properly unmount it. 

EDIT - Some time later - Yup.  That fixed it.  I guess my routine is going to require me to spend the extra five seconds every day to properly eject the drive from my mac before I plug it into the Unraid server.

Edited November 17, 2023 by AeroMaestro
Added a followup to say it's all good, man. Thanks so much.

[jademonkee]

On 11/15/2023 at 11:15 AM, jademonkee said:

 I note that this page: https://askubuntu.com/questions/1323668/one-volume-did-not-go-dst-with-the-rest-of-the-system

Mentions that you can mount disks using the option:

tz=UTC

How would I go about adding that option to UD for this disk?

Does anyone know how to mount disks with the tz=UTC option?

[dlandon]

4 hours ago, jademonkee said:

Does anyone know how to mount disks with the tz=UTC option?

What is it you are trying to accomplish?  I assume you are copying files with differing source and destination time zones.

 

Please provide some clarification:

• What is the issue with the time zones?  Where are you copying files from and to?
• What file copy command are you using?  Rsync?

[jademonkee]

31 minutes ago, dlandon said:

What is it you are trying to accomplish?  I assume you are copying files with differing source and destination time zones.

 

Please provide some clarification:

• What is the issue with the time zones?  Where are you copying files from and to?
• What file copy command are you using?  Rsync?

 

Hi, sorry, I originally quoted an earlier detailed post, which can be found here:

The summary is that I have an exfat disk that I keep music on and which is accessed by Windows and Mac computers (thus my preference for exfat). I sync it from my server using UD and rsync (the rsync script is in the original post).

I posted a while ago that it occasionally syncs the entire drive contents, rather than just what has changed, and I now realise it's because of daylight savings changing, and exfat not using unix time. At the time I thought maybe the drive was failing.

One solution is to mount the device using the option "tz-utc", so that the time stamps don't change when the UK moves over to DST (we're effectively on UTC otherwise), so I was wondering if it was possible to mount specific disks as UTC, and how to do it. I would think that the script would be too late for adding that mounting option, correct?

I'm open to other suggestions, though, including modifying the rsync command/script, or even moving to a format freely and reliably supported by Windows and Mac that also supports unix timestamps/DST.

The rsync command from the script is:

rsync -vrltD --delete --exclude '.Recycle.Bin' /mnt/user/mymusic $MOUNTPOINT 2>&1 >> $LOGFILE

Many thanks.

[dlandon]

3 minutes ago, jademonkee said:

The rsync command from the script is:

Add the following parameters to your rsync command:

-a --times

In this command:

• a (or --archive) is used to preserve various file attributes, including timestamps.
• --times ensures that modification times are synchronized between the source and destination.

[jademonkee]

4 minutes ago, dlandon said:

Add the following parameters to your rsync command:

-a --times

In this command:

• a (or --archive) is used to preserve various file attributes, including timestamps.
• --times ensures that modification times are synchronized between the source and destination.

Oh, I didn't realise that would work on exfat. Many thanks!

[dlandon]

6 minutes ago, jademonkee said:

Oh, I didn't realise that would work on exfat. Many thanks!

It should.

[jademonkee]

8 minutes ago, dlandon said:

It should.

Ah, I already specified -t in my command (vrltD --delete), so I don't think it'll fix it.

The issue is that the server's time changes for DST, but the timezone for the exfat isn't specified, so it thinks that every file has shifted by an hour.

If it isn't possible to specify the TZ for a specific disk in UD, I'll keep hunting around for an rsync solution.
A workaround is to specify "--modify-window=3601", though this means that if I modify something, sync it, then modify it again all within an hour, it won't sync those changes next time. I can probably keep this in mind, though.

Thanks.

[dlandon]

13 minutes ago, jademonkee said:

Ah, I already specified -t in my command (vrltD --delete), so I don't think it'll fix it.

The issue is that the server's time changes for DST, but the timezone for the exfat isn't specified, so it thinks that every file has shifted by an hour.

If it isn't possible to specify the TZ for a specific disk in UD, I'll keep hunting around for an rsync solution.
A workaround is to specify "--modify-window=3601", though this means that if I modify something, sync it, then modify it again all within an hour, it won't sync those changes next time. I can probably keep this in mind, though.

Thanks.

Add the archive option '-a'.

[jademonkee]

23 hours ago, dlandon said:

Add the archive option '-a'.

Ok, I've changed the command to:

rsync -av --times --delete --exclude '.Recycle.Bin' /mnt/disk1/mymusic $MOUNTPOINT 2>&1 >> $LOGFILE

And the log is filled with e.g.

rsync: [generator] chown "/mnt/disks/ADATA2TB/mymusic/Albums/Conjoint/[2000] Earprints [FLAC]/01. Earprint Nr1.flac" failed: Operation not permitted (1)

For every file and dir (I think because exfat doesn't support permissions etc, and the -a flag is the equivalent of -rlptgoD, which includes groups, permissions etc).

The rsync man page says, however, suggests that --size-only could be a useful flag here as:
"This modifies rsync's lqquick checkrq algorithm for finding files that need to be transferred, changing it from the default of transferring files with either a changed size or a changed last-modified time to just looking for files that have changed in size. This is useful when starting to use rsync after using another mirroring system which may not preserve timestamps exactly."

So I've changed my rsync command to:

rsync -vrltD --size-only --delete --exclude '.Recycle.Bin' /mnt/disk1/mymusic $MOUNTPOINT 2>&1 >> $LOGFILE

And it worked for a run (no changes to sync, however). I will find out when daylight savings next changes if this solution has worked (though I suspect it will). I'll report back if changing to 'size only' causes any weird sync issues.

Thanks for your help.

[ziggy99]

Running latest unassigned disks plugin on latest Unraid. When I mount a disk and browse the disk content from the Unraid "Main" tab, I can see the files on the disk. The issue is that if I copy new files to the disk (e.g. in a VM), these new files are not shown when browsing again. I have to unmount and mount again to see the new files in the browse function.

 

Is this as expected?

[dlandon]

40 minutes ago, ziggy99 said:

Is this as expected?

No.  What happens if you wait a while to browse for the new files?  It might be a refresh issue some where.

[theunraidhomeuser]

Hey, sorry, I posted on your github an issue that's probably not a code issue, just one for me. Hence here again:

 

Hi there, I'm trying to find documentation on this. I recently changed my setup, adding a lot of drives via USB (Syba 8 Bay Enclosures). These drives are not meant to be part of a pool, I just wnat to mount them. I believe previously, they may have been part of a mergerfs filesystem but normally that doesn't cause any issues as mergerfs is non-destructive to the partittion tables.

Do you have an idea how I can mount these drives? I can' create an UNRAID pool, they are more than 30 drives (and I didn't have them in an UNRAID pool before)....

Cheers!

[dlandon]

22 hours ago, theunraidhomeuser said:

Hey, sorry, I posted on your github an issue that's probably not a code issue, just one for me. Hence here again:

 

Hi there, I'm trying to find documentation on this. I recently changed my setup, adding a lot of drives via USB (Syba 8 Bay Enclosures). These drives are not meant to be part of a pool, I just wnat to mount them. I believe previously, they may have been part of a mergerfs filesystem but normally that doesn't cause any issues as mergerfs is non-destructive to the partittion tables.

Do you have an idea how I can mount these drives? I can' create an UNRAID pool, they are more than 30 drives (and I didn't have them in an UNRAID pool before)....

Cheers!

Can UD see them each with a unique identification?  Some drive bays present the disks all with the same ID.  UD can only mount them when they have unique ID's.