Jump to content

kmwoley

Members
  • Content Count

    32
  • Joined

  • Last visited

Community Reputation

7 Neutral

About kmwoley

  • Rank
    Advanced Member

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. Thanks for your help. I kinda get it, but when looking at the disk usage before making these changes I would have also said that I had 31GB free. Am I interpreting that first `btrfs fi usage...` command wrong? How am I to know in the future when a balance operation is required before I "run out of space"? I ask because I'd like to write a cron job that warns me before I run out of space. Thanks again for the help.
  2. Thanks for the pointer. After reading that thread, here's what I did... I'd appreciate if you could check my understanding and help me to know how to avoid this in the future. 1) Checked the space reported by btrfs... root@lenny:~# btrfs fi usage /mnt/cache Overall: Device size: 352.13GiB Device allocated: 238.48GiB Device unallocated: 113.64GiB Device missing: 0.00B Used: 182.85GiB Free (estimated): 84.24GiB (min: 84.24GiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 376.52MiB (used: 0.00B) Data,RAID1: Size:118.21GiB, Used:90.79GiB /dev/sdb1 118.21GiB /dev/sdc1 118.21GiB Metadata,RAID1: Size:1.00GiB, Used:645.45MiB /dev/sdb1 1.00GiB /dev/sdc1 1.00GiB System,RAID1: Size:32.00MiB, Used:48.00KiB /dev/sdb1 32.00MiB /dev/sdc1 32.00MiB Unallocated: /dev/sdb1 113.64GiB /dev/sdc1 1.05MiB The way I read that, my RAID1 cache drives have 27.42GB free. Accept for that 1.05MB that's unallocated on /dev/sdc1 which I'm assuming is the problem... 2) I deleted some files off the drive(s) to make some space and ran the balance command as recommended: btrfs balance start -dusage=75 /mnt/cache That command completed with no errors. Afterwords, here's what free space is reported... root@lenny:/mnt/cache# btrfs fi usage /mnt/cache Overall: Device size: 352.13GiB Device allocated: 188.16GiB Device unallocated: 163.97GiB Device missing: 0.00B Used: 179.96GiB Free (estimated): 84.62GiB (min: 84.62GiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 330.16MiB (used: 0.00B) Data,RAID1: Size:92.05GiB, Used:89.41GiB /dev/sdb1 92.05GiB /dev/sdc1 92.05GiB Metadata,RAID1: Size:2.00GiB, Used:585.58MiB /dev/sdb1 2.00GiB /dev/sdc1 2.00GiB System,RAID1: Size:32.00MiB, Used:32.00KiB /dev/sdb1 32.00MiB /dev/sdc1 32.00MiB Unallocated: /dev/sdb1 138.80GiB /dev/sdc1 25.16GiB So, my question is... how much free space on the cache do I actually have now? How do I report on it and monitor it so I can set up some alerts when it's getting close to being a problem in the future? Given the catastrophic nature of running out of cache space (i.e. my entire infrastructure ground to a halt), I can't rely on this system if this is going to happen without warning in the future. Any guidance would be helpful. Thanks!
  3. Done. Attached to the original post. Sorry I left those off. Thanks!
  4. Hey Folks, I could really use some help. I've run into trouble that I noticed after having upgraded to 6.7 from 6.6.7, but it might not be related to the upgrade. I first noticed that Docker was filling up /var/log and my Docker containers were stopping on me. I turned off a number of my Docker containers and rebooted to see if I could isolate which Docker was log-spamming. That helped, but eventually my docker containers stopped again after a few hours, complaining about running out of disk space (on /mnt/user/appdata) where there's clearly enough disk space. In that process, I was exploring the system logs and found the errors below. I downgraded back to 6.6.7 from 6.7 to see if it'd fix it and it hasn't. The ACPI BIOS errors are present in 6.6.7. As are the BTRFS write errors... so maybe it's likely coincidence that these errors have come about shortly after the 6.7 upgrade. Now I'm now seeing errors from Fix Common Problems that "**** Unable to write to cache **** **** Unable to write to Docker Image ****"... which is no surprise if the cache disks are having issues. Any clue where to start on this issue? May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT3._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT2._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT1._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT5._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT0._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT1._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT2._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT0._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT3._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 07:51:55 lenny kernel: ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT5._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330) May 14 10:18:18 lenny kernel: loop: Write error at byte offset 852369408, length 4096. May 14 10:18:18 lenny kernel: print_req_error: I/O error, dev loop2, sector 1664768 May 14 10:18:18 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 20, rd 0, flush 0, corrupt 0, gen 0 May 14 11:07:15 lenny kernel: loop: Write error at byte offset 876773376, length 4096. May 14 11:07:15 lenny kernel: print_req_error: I/O error, dev loop2, sector 1712448 May 14 11:07:15 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 21, rd 0, flush 0, corrupt 0, gen 0 May 14 19:19:21 lenny kernel: loop: Write error at byte offset 1133211648, length 4096. May 14 19:19:21 lenny kernel: print_req_error: I/O error, dev loop2, sector 2213280 May 14 19:19:21 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 22, rd 0, flush 0, corrupt 0, gen 0 May 14 20:06:46 lenny kernel: loop: Write error at byte offset 12656640, length 4096. May 14 20:06:46 lenny kernel: print_req_error: I/O error, dev loop2, sector 24720 May 14 20:06:46 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 23, rd 0, flush 0, corrupt 0, gen 0 May 14 20:17:13 lenny kernel: dhcpcd[1674]: segfault at 88 ip 00000000004216d2 sp 00007ffd89a6dd70 error 4 in dhcpcd[407000+31000] May 14 20:17:50 lenny kernel: loop: Write error at byte offset 146325504, length 4096. May 14 20:17:50 lenny kernel: print_req_error: I/O error, dev loop2, sector 285792 May 14 20:17:50 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 24, rd 0, flush 0, corrupt 0, gen 0 May 14 20:20:20 lenny kernel: loop: Write error at byte offset 159694848, length 4096. May 14 20:20:20 lenny kernel: print_req_error: I/O error, dev loop2, sector 311904 May 14 20:20:20 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 25, rd 0, flush 0, corrupt 0, gen 0 May 14 20:22:21 lenny kernel: loop: Write error at byte offset 16392192, length 4096. May 14 20:22:21 lenny kernel: print_req_error: I/O error, dev loop2, sector 32016 May 14 20:22:21 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 26, rd 0, flush 0, corrupt 0, gen 0 May 14 20:25:00 lenny kernel: loop: Write error at byte offset 167190528, length 4096. May 14 20:25:00 lenny kernel: print_req_error: I/O error, dev loop2, sector 326528 May 14 20:25:00 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 27, rd 0, flush 0, corrupt 0, gen 0 May 14 20:26:51 lenny kernel: loop: Write error at byte offset 2207227904, length 4096. May 14 20:26:51 lenny kernel: print_req_error: I/O error, dev loop2, sector 4310784 May 14 20:26:51 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 28, rd 0, flush 0, corrupt 0, gen 0 May 14 20:26:51 lenny kernel: loop: Write error at byte offset 65536, length 4096. May 14 20:26:51 lenny kernel: print_req_error: I/O error, dev loop2, sector 128 May 14 20:26:51 lenny kernel: print_req_error: I/O error, dev loop2, sector 128 May 14 20:26:51 lenny kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 29, rd 0, flush 0, corrupt 0, gen 0 May 14 20:26:51 lenny kernel: BTRFS error (device loop2): error writing primary super block to device 1 May 14 20:26:51 lenny kernel: BTRFS: error (device loop2) in write_all_supers:3781: errno=-5 IO failure (1 errors while writing supers) May 14 20:26:51 lenny kernel: BTRFS: error (device loop2) in cleanup_transaction:1846: errno=-5 IO failure
  5. I cannot tell you why or how, but after downgrading from 6.6.7 -> 6.6.6 four days ago, and then upgrading again today from 6.6.6 -> 6.6.7 the issue I reported before (i.e. Custom Network Types not appearing in Docker Container configuration) appears to have gone away. 🤷‍♂️
  6. Confirmed. Rolling back to 6.6.6 solved this issue. Container configs can now select the Custom networks for Network Type.
  7. Hey folks - I ran into a problem with this upgrade. I have custom network's configured for Docker (MACVLAN). After the update, all of my containers started just fine. But one of them started to have some networking issues and I started investigating. I tried changing a setting in that container and when I saved the container, it disappeared! Looking deeper, I notice now that under the networking options for the container config any network which was using a MACVLAN network (e.g. br0.55) now shows Network Type of "None" and the custom networks for MACVLAN aren't an option in the container config. If I apply any changes to a container now, it functionally disappears. I'm going to try rolling back to 6.6.6 to see if it solves the issue... but thought you should know. Anyone else seeing this?
  8. You're right that masquerading is typically done for traffic exiting your local network, but it can also be used for things like the OpenVPN config. In the case of OpenVPN it's handing out IP addresses to the clients in a different range than the local network. In the bridged docker network configuration installation of OpenVPN, those remote client IP addresses were being masqueraded somewhere along the path before they left the host so they showed up as my host server's IP on the local network. Switching to host networking exposed those client IPs to the local network. At some point I'd like to understand why that was... but for now, I can configure my local network to work around the problem. At this point it's kinda academic for me to learn why that was; I'm curious.
  9. I honestly don't have a clue then how OpenVPN-AS (or any other VPN) running in Docker host networking mode is masquerading it's client traffic, then. It just doesn't make sense to me that there's no way to masquerade the OpenVPN client IP addresses on their way out the door. In the end, I gave up trying to get my host to masquerade the traffic and instead added static routes for my OpenVPN client address space to my networking stack outside of the unraid host. It's a fine solution, just annoying. If someone trips across this post in the future and finds a solution, I'd be interested in knowing what you figure out. Thanks.
  10. @bonienl thanks - that’s incredibly useful to know. Is that an unraid specific feature/limitation, or a general networking characteristic? Do you know how OpenVPN-AS (or other VPNs which run in host mode) and their hosts are configured to properly route the traffic? I could probably put a NAT entry on my router, but I don’t like the idea of solving this issue “off box” if there’s a cleaner way to do it.
  11. Thanks, @Squid - it's useful to see other folks config. Looking at your iptables, I'm even more confused. I don't see an entry which would handle the masquerade of your OpenVPN client IP (172.27.234.2). I'm at a loss as far as where to look next to understand why putting the container in 'bridge' networking works, but 'host' does not.
  12. Thanks for offering! Under the OpenVPN-AS Docker config settings, change networking from “bridge” to “host”. I would expect that your VPN clients can access the internet, local in whatever way you already had them configured.
  13. I don’t know. I’ve always used the one from kylemanna as it has a pretty big user base elsewhere. And I was already familiar with the bare OpenVPN over OpenVPN-AS If someone has the lsio one installed and configured under bridge networking, it’d be an easy test to switch the network type to host and see what happens.
  14. Hey folks, I'm beating my head against a wall here and could use some help reasoning through a networking problem. I've got an OpenVPN container (kylemanna/openvpn) that works just fine when it's in bridge networking mode (clients can reach internet, local network). However, when I put it into host networking the clients connected via OpenVPN cannot reach the internet or devices on the local network. I've narrowed it down to what I think is a masquerading problem. When OpenVPN is in bridge networking mode, tcpdump shows traffic originating from OpenVPN clients leaving the ethernet adapter on the server have been masqueraded correctly to the server's IP (10.4.10.8). In this example, I'm successfully connecting to a endpoint at 10.4.40.20 from a OpenVPN client: # tcpdump -i eth0 dst 10.4.40.20 09:21:37.044954 IP 10.4.10.8.64693 > 10.4.40.20.http: Flags [SEW], seq 1466305596, win 65535, options [mss 1361,nop,wscale 6,nop,nop,TS val 827422948 ecr 0,sackOK,eol], length 0 09:21:37.104329 IP 10.4.10.8.64693 > 10.4.40.20.http: Flags [.], ack 750528488, win 2065, options [nop,nop,TS val 827423017 ecr 232700051], length 0 09:21:37.247838 IP 10.4.10.8.64693 > 10.4.40.20.http: Flags [P.], seq 0:361, ack 1, win 2065, options [nop,nop,TS val 827423152 ecr 232700051], length 361: HTTP: POST /onvif/device_service HTTP/1.0 When OpenVPN is in host networking mode, tcpdump shows that same traffic which is coming from OpenVPN client as having the IP address assigned by OpenVPN (192.168.255.200): # tcpdump -i eth0 dst 10.4.40.20 23:50:27.172338 IP 192.168.255.200.64379 > 10.4.40.20.http: Flags [S], seq 119222741, win 65535, options [mss 1361,nop,wscale 6,nop,nop,TS val 814857219 ecr 0,sackOK,eol], length 0 For some reason which is unknown to me, that traffic isn't getting masqueraded correctly. So, 192.168.255.0/24 OpenVPN client IPs are leaking out to my local network which clearly has no idea how to route them back. Here's the relevant iptables from the server: # iptables -t nat -v -L POSTROUTING -n --line-number num pkts bytes target prot opt in out source destination 1 102 6434 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0 2 0 0 MASQUERADE all -- * eth0 192.168.255.0/24 0.0.0.0/0 3 0 0 MASQUERADE all -- * tun0 192.168.255.0/24 0.0.0.0/0 Line 2 is part of the default OpenVPN configuration. Line 3 I added in attempt to see if it'd help (it didn't make any difference). I've checked that, when OpenVPN is in the host networking config, the OpenVPN container itself has the expected access to the internet and local network (i.e. attaching to the container and pinging things works as expected). I've tried a handful of things, but not being an expert in networking, I've reached my limit of knowledge. Let me know if there's other config/settings that'd be useful in debugging this. Any hints would be very appreciated. Thanks.