With the new USB creator tool that allows > 16GB flash drives, I tried the following to attempt to move Docker onto the USB flash drive.

 

I've moved I/O to a ram disk, so save your precious time bothering with replies telling me that performance will be bad. And no I don't care if I kill the USB drive early with this.

 

Here's what I did...

 

In the filesystem:

1. mkdir /boot/docker
2. mkdir /boot/docker/appdata

 

In the docker settings:

• Docker vdisk size: 4GB (the max for vfat)
• Docker storage location: /boot/docker/docker.img
• Default appdata storage location: /boot/docker/appdata/

 

I need some more verbose logs, but I suspect it's a permission issue (chown isn't letting me set  nobody:users as owners from the terminal)

 

Aug 25 20:53:14 backup ool www[1166]: /usr/local/emhttp/plugins/dynamix/scripts/emhttpd_update
Aug 25 20:53:14 backup emhttpd: req (45): cmdStatus=apply&csrf_token=CF7B7492738E3E76
Aug 25 20:53:14 backup emhttpd: Starting services...
Aug 25 20:53:14 backup emhttpd: shcmd (98715): /etc/rc.d/rc.docker start
Aug 25 20:53:14 backup root: no image mounted at /var/lib/docker
Aug 25 20:53:14 backup emhttpd: shcmd (98715): exit status: 1
Aug 25 20:53:14 backup emhttpd: shcmd (98717): umount /var/lib/docker
Aug 25 20:53:14 backup root: umount: /var/lib/docker: not mounted.
Aug 25 20:53:14 backup emhttpd: shcmd (98717): exit status: 32

 

Any ideas on how to overcome this to move forward?

You cannot set Linux style permissions on the /boot location contents as iit is formatted (by design) as FAT32 which does not support such permissions.   This is likely to stop /boot/docker/appdata being a viable location, although I must admit I have not tried this myself.

Thanks I got that much myself. Help me to work around this!

 

I know it's unorthodox but I'm trying to break some new ground here! I got a nice sized UnRAID USB flash drive and quite frankly more RAM than I should have, 128GB. I've got that RAM to manage much of the Docker container's I/O that I run, so I just need a means to put the docker.img and the /appdata config files on the UnRAID flash drive.

 

Right now, it's not working for me and it seems like some arbitrary blocker I can't see is in the way.

Only way I can see is copying stuff to RAM at boot, but wouldn't have thought that will help boot time much if you have a lot of containers,  And like @itimpi says you'll need to watch permissions etc.

 

you'd also need to copy stuff back to /boot afterwards, which seems a bit of a block to running stuff off the USB.

Edited August 26, 20178 yr by CHBMB

Until now the design concept was to minimize writing to the USB stick as much as possible.

 

Sounds like this approach is defeating that purpose, i.e. wouldn't that affect longevity of the USB device?

 

8 minutes ago, bonienl said:

Until now the design concept was to minimize writing to the USB stick as much as possible.

 

Sounds like this approach is defeating that purpose, i.e. wouldn't that affect longevity of the USB device?

 

Quote

I've moved I/O to a ram disk, so save your precious time bothering with replies telling me that performance will be bad. And no I don't care if I kill the USB drive early with this.

 

He doesn't mind.... 

2 minutes ago, CHBMB said:

 

He doesn't mind.... 

 

Haha, he should care, otherwise one would wind up purchasing a new unRAID license every month 

8 minutes ago, bonienl said:

 

Haha, he should care, otherwise one would wind up purchasing a new unRAID license every month 

 

Might be right! =D

 

I'm not afraid!

 

So how do I work around what I detailed that's blocking?

2 minutes ago, Lev said:

So how do I work around what I detailed that's blocking?

 

I believe it is what @itimpi and @CHBMB already mentioned. Docker containers rely on correct access rights, including group- and user names, which are not available on FAT32 formatted drives.

 

I would suggest that you first try to just move the docker.img file to the flash.  That should mean that permissions are not relevant as they are handled internally within the docker.img file, and if it works it will prove that the issue relates to the appdata folder.

 

If you have permission issues around the appdata folder then there will no be no simple answer.   The only way that MIGHT solve this issues is to have the USB drive partitioned with 2 partitions - the first one being FAT32 for use by unRAID and the other being a Linux format that can handle the permissions.   However whether this will upset the unRAID boot process I have no idea.

 

As was mentioned it seems unlikely that this will be of anything other than academic interest as it is almost certain that the USB drive will fail relatively rapidly due to the high number of writes.

Edited August 26, 20178 yr by itimpi

Have you thought  about a second partition on that USB drive?

To be fair, some of the more expensive USB3 sticks are basically a SSD, they have 'proper' flash and a decent controller, so it's probably a viable idea using a USB3 flash/cache combined.  I have a Sandisk 128GB stick in work, and it does 450MB/s read and about 300MB/s writes, and has typical SSD endurance. https://www.sandisk.co.uk/home/usb-flash/extremepro-usb

 

Most servers come with an internal USB3 port on the board, now too. (My Fujitsu TX1310M1 does, as do the Gen9/10 HPs).

 

So, would be a feasible idea to have combined boot and cache/appdata on a stick, save on a SATA connection.

i guess I have to ask if you want to put appdata on a flash drive, why does it have to be to on the boot flash drive?

 

if you want to do this it appears to me to make more sense to just use a 2nd flash drive:

• no worries about wearing out your boot drive (I know you have said you don't care about this).
• not restricted to the FAT file system required by the boot drive.

You should be able to create a loopback file system on the flash, much like docker.img, and keep your appdata in there. The loopback can be formatted with XFS or BTRFS. You could form a softlink from cache to the loopback file system.

 

You might even be able use docker.img. Just make it bigger. The location where docker.img is mounted is /var/lib/docker.

7 minutes ago, bjp999 said:

You might even be able use docker.img. Just make it bigger. The location where docker.img is mounted is /var/lib/docker.

I'd recommend a second image, just for the sake of keeping the current functionality of easily setting docker back up if docker.img gets corrupted or otherwise unserviceable. The current method of blowing away docker.img and rebuilding from templates has been fleshed out pretty well, if you use @Squid's CA it just a few clicks to get back running. If you start storing persistent data in that image, it complicates things.

 

I seem to remember a year or so ago there was talk of setting up appdata in an image all its own anyway, to keep permissions and such safe and separate from the other user shares. That never happened, I think largely because it was deemed unnecessary work, and would require a whole new way of allowing access to appdata config files instead of just accessing the appdata share.

 

It's been a while, my memory may be faulty.

@jonathanm

 

Yea, I am actually working on something like this for keeping appdata in its own loopback FS. Kind of put on back burner due to other priorities.

 

But I do think setting this up with the existing Docker might be fine for a quick POC. But agree that a separate loopback might be better so the Docker.img can be deleted and recreated without losing appdata in the process.

Lots of great info and suggestions here to try, thank you all.

 

After some thinking about each before attempting to try them, I had a few thoughts to share, maybe making me rethink my original desire to do this...

 

I think I was originally interested in doing this because it just seemed like a lot of wasted space on the unRAID USB drive, and if I/O could be minimized enough then perhaps it's enough to store Docker img or app data there. This is really about just having the simplicity of consolidation, however it's anything but simple as noted by all over your great comments.

 

Having a second partition seems like the way to go, it'll resolve the permissions and file size limitations now in place by FAT32. I'll probably just go with XFS in that case.

 

Assume I do that and its successful and it all works. When everything is working as intended no issues. However things always don't work as expected, and when a docker doesn't shutdown correctly, or a unclean shutdown happens, in my experience, open/locked files on a USB mount do not recover well. Assuming I formatted that second partition XFS, there will be a lot of running xfs_repair every-time this happens.

 

Anyway point of this post is I see the possibilities with all of your advice on how to make it work, and I may still yet give it a go. Thank you!

It has begun! See attachment.

 

@dlandon I'm well outside the lines here, exploring the danger zone. If there's some top secret way to get UD to turn off some off the safe-guards that prevent it from showing the UNRAID usb boot device, please send me a private message. If not, no worries, just thought I'd ask. I'm quite at home in the terminal.

The flash drive is mounted already at /boot

9 hours ago, Lev said:

It has begun! See attachment.

 

@dlandon I'm well outside the lines here, exploring the danger zone. If there's some top secret way to get UD to turn off some off the safe-guards that prevent it from showing the UNRAID usb boot device, please send me a private message. If not, no worries, just thought I'd ask. I'm quite at home in the terminal.

 

UD was never intended to work with the flash drive.  The flash drive is the "master link" for unRAID as it were, so any messing around here could be catastrophic.  I don't feel comfortable even suggesting ways to get around this.

 

How about using another USB (cheap one) for the flash drive, and inserting this one and use UD to work with it?

1 minute ago, CHBMB said:

The flash drive is mounted already at /boot

 

He wants to mount a second partition and use that.

16 minutes ago, dlandon said:

 

He wants to mount a second partition and use that.

 

Oh, I thought the plan was to put a loopback image on the fat32 partition?  My bad.....

23 minutes ago, CHBMB said:

 

Oh, I thought the plan was to put a loopback image on the fat32 partition?  My bad.....

 

I think you got caught up in the different ideas he's throwing around.  It's a moving target.

56 minutes ago, dlandon said:

 

UD was never intended to work with the flash drive.  The flash drive is the "master link" for unRAID as it were, so any messing around here could be catastrophic.  I don't feel comfortable even suggesting ways to get around this.

 

How about using another USB (cheap one) for the flash drive, and inserting this one and use UD to work with it?

 

No worries, appreciate the response!

 

I'm trying the suggestion proposed of using a second partition:

 

From my earlier screenshot:

128GB USB Flash Drive (/dev/sda)

Partition 1: 1GB FAT32 for UNRAID (sda1) /boot

Partition 2: 127GB XFS for Docker or whatever (sda2) /mnt/disks/usb

 

In the GUI I'm not able to see partition 2 after is been mounted. I suspect the Dynamix GUI has protection in place to safeguard to keep all of /dev/sda from accidents and crazies like me. 

 

This means to continue on this insane journey I'll be editing the docker.cfg file to force the path mount for Docker.img ... If the safeguards exist at a lower level than the GUI then that may be the end of the line. Stay tuned to find out.

 

 

 

 

 

 

Research complete. This was a great success! Completely achieved what I set out to research and learn so much more in the process. 

 

Big thank you to everyone who contributed in this thread, couldn't of did this without your input! I think I broke the limit on using @mentions here, restricted to 7 its seems and it breaks this reply post as well so I'll continue will more details in the next reply. Thank you: @itimpi  @bonienl @CHBMB @Frank1940 @HellDiverUK @remotevisitor  @SSD @bjp9999 @jonathanm @dlandon