Limiting docker app write speed to array


henris

Recommended Posts

I recently migrated my two old unRAID/Plex servers to a new 24-bay server. Everything else is working very nicely but whenever a docker app starts to transfer files from cache disk to array some of my not-so-well-built Plex clients like Samsung TV's experience buffering with UHD-Blu-ray material. My cache disk is a dual 500GB M.2 NVMe BTRFS pool. As part of the migration I upgraded my parity to 8TB WD Red. Data disks are ranging from a single 8TB WD RED to 4/3 TB WD Reds and Greens.

 

I've analysed the problem to be IO saturation on the write target disk. I'm using high water mark allocation so the likelihood of writes taking place on the same disk as the playback of latest movie is pretty high. I'm seeing 50+MBps write speed and looking at iotop and atop the targer disk seems saturated.

 

I searched and found that you can limit the io with docker. Eg: https://stackoverflow.com/questions/36145817/how-to-limit-io-speed-in-docker-and-share-file-with-system-in-the-same-time

 

I tried to give that as an extra parameter to the docker but it failed. I cannot figure out what the device path should be here. I tried shfs but that does not work.

--device-write-bps=shfs:10mb

What is the correct device path if I want to limit the docker app's write speed to any user share?

 

As a side note, I lost the whole docker when the start failed. I had to manually remove the parameter from the my-zzzz.xml and re-add the docker. Got the instruction from here: https://www.reddit.com/r/unRAID/comments/8bej8p/how_to_get_back_a_missing_docker_after_changing/

 

Edited by henris
Link to comment

The syntax part is a bit confusing. The official Docker documentation does not provide an example for these parameters: https://docs.docker.com/engine/reference/commandline/run/

 

Majority of the examples found use the "—-device-write-bps=/dev/mdX:10mb" syntax. Based on the error message given when I tried to start the container, the problem is not with the syntax but with the device definition itself.

 

I really do not want to experiment the device identifier too much since on the first time the whole container app disappeared. Hopefully someone would now what device identifier should be used here. I might experiment using a single disk device and test if this works at all. I know which disk is currently the high water target.

Edited by henris
Link to comment

I have yet to figure this one out using the docker bandwidth limiter. 

 

But I took a different approach; previously I used my cache only for dockers/vms. I had thought that using cache also for array shares would wear the SSDs out. But since most of the writes are actually originating from the dockers, the cache write will be a simple rename/move. So there is no extra wear on the cache drives since the files are already there under /mnt/cache/appdata. Using cache will also make the "move" from docker's folder to share immediate.

 

This is a basic feature that many use but I at least had not thought about it in this context; protecting the array from saturation during day time. I had only thought it as a way to increase the array write speed.

 

Short list of steps:

  • Stopped array
  • Activated cache disk in Settings -> System Settings -> Global Share Settings -> Cache Settings. Min free space 100GB. 
  • Configured a daily (or better nightly) mover in Settings -> User Preferences -> Scheduler -> Mover Settings
  • Configured relevant shares (movies, tv-shows, sports) to use cache disk (Use cache disk: Yes)
  • Configured my appdata to be only on cache disk (Use cache disk: Only)
  • Started array
Edited by henris
Link to comment
  • 1 year later...

I'm trying to find a way to avoid hitting 100% HDD usage when copying files from the downloads folder to their destination, and found this in the docker documentation, searched this forum and only found this thread with no real solution.

Tried

—-device-write-bps /dev/sdd:1mb

But didn't work. Anyone knows how to do this?

Link to comment
  • 3 years later...

It actually works.

 

I used the following command to get the exact device names:

 

df -h
Filesystem             Size  Used Avail Use% Mounted on
rootfs                  50G  323M   50G   1% /
tmpfs                   32M  1.6M   31M   5% /run
/dev/sda1               15G  2.5G   12G  17% /boot
overlay                 50G  323M   50G   1% /lib
overlay                 50G  323M   50G   1% /usr
devtmpfs               8.0M     0  8.0M   0% /dev
tmpfs                   32G     0   32G   0% /dev/shm
tmpfs                  128M  9.5M  119M   8% /var/log
/dev/mapper/md1p1      7.3T  3.3T  4.1T  45% /mnt/disk1
/dev/mapper/md2p1      7.3T  6.9T  419G  95% /mnt/disk2
/dev/mapper/md3p1      7.3T  3.9T  3.5T  53% /mnt/disk3
/dev/mapper/md4p1      7.3T  6.1T  1.3T  84% /mnt/disk4
/dev/mapper/md5p1      7.3T  6.6T  769G  90% /mnt/disk5
/dev/mapper/md6p1      7.3T  6.0T  1.4T  82% /mnt/disk6
/dev/mapper/md7p1      7.3T  3.9T  3.5T  53% /mnt/disk7
/dev/mapper/md8p1      7.3T  4.8T  2.5T  66% /mnt/disk8
/dev/mapper/md11p1     3.7T   26G  3.7T   1% /mnt/disk11
/dev/mapper/md12p1     3.7T   26G  3.7T   1% /mnt/disk12
/dev/mapper/md13p1     4.6T  3.2T  1.4T  70% /mnt/disk13
/dev/mapper/md14p1     4.6T  527G  4.1T  12% /mnt/disk14
/dev/mapper/nvme0n1p1  932G  197G  735G  22% /mnt/nvme
/dev/mapper/sdo1       932G  437G  495G  47% /mnt/sata_raid
shfs                    75T   45T   30T  61% /mnt/user0
shfs                    75T   45T   30T  61% /mnt/user
/dev/loop2              45G   30G   12G  72% /var/lib/docker
/dev/loop3             1.0G  4.7M  904M   1% /etc/libvirt

 

I tried limiting write speed to disk7 by adding the following extra parameter to the Krusader docker container:

 

--device-write-bps=/dev/mapper/md7p1:10mb

 

Seems to work as intended (see capture attached).

 

 

 

Capture.JPG

Edited by Opawesome
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.