Unbound has no access to anything outside the host?

scolastico

I’v recently updated how my VM’s reach the internet in a setup kinda like
internet <-> proxmox <-> vm’s

Every vm, and every other container runs absolutly fine on the new setup. Except mailcow or to be more exact unbound. Unbound fails its healthcheck, and if i do a docker exec and check a ping, it seems it cant reach anything outside the vm.

8.8.8.8, Google: FAIL
10.10.10.1, Proxmox (and Gateway): FAIL
10.10.10.2, The host vm where mailcow runs: SUCCESS
10.10.10.250, The host vm where mailcow runs: SUCCESS

Any ideas?

Oh and it cant be a firewall issue on proxmox itself as it has an “allow all” rule for icmp traffic. So pings should always work.

esackbauer

scolastico I’v recently updated how my VM’s reach the internet in a setup kinda like
internet <-> proxmox <-> vm’s

How was the setup before you “updated”?
What firewall are you using now, and does it allow outgoing traffic on ICMP and Port 53 TCP/UDP?
What OS is the VM? Any firewall on the VM itself active?

scolastico it cant be a firewall issue on proxmox itself as it has an “allow all” rule for icmp traffic.

The rule is active for both directions?

Can you ping 8.8.8.8 from the host VM?

scolastico

esackbauer How was the setup before you “updated”?

internet <-> proxmox <-> opnsense <-> vms

esackbauer What firewall are you using now, and does it allow outgoing traffic on ICMP and Port 53 TCP/UDP?

A script:

#!/bin/bash

METHOD=$1

# Flush existing rules
iptables -F
ip6tables -F

# Drop anything which is not allowed
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow all localhost traffic
iptables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -i lo -j ACCEPT

# Allow established and related traffic
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
ip6tables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow forwarding
sysctl -w net.ipv6.conf.default.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.default.proxy_ndp=1
sysctl -w net.ipv6.conf.all.proxy_ndp=1
sysctl -w net.ipv4.ip_forward=1

if [[ $METHOD == "up" ]]; then
  iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
  ip -6 route add foo:baa::/64 dev vmbr0
fi
if [[ $METHOD == "down" ]]; then
  iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
  ip -6 route del foo:baa::/64 dev vmbr0
fi

# Setup additional ip's
IP_LIST=(
  "xxx"
  "xxx"
  "xxx"
  "xxx"
  "xxx"
)

for ip in "${IP_LIST[@]}"; do
  if [[ $METHOD="up" ]]; then
    ip addr add $ip dev vmbr0
  fi
  if [[ $METHOD="down" ]]; then
    ip addr del $ip dev vmbr0
  fi
done

# Helper to get the vm ip
# vmIP <vm>
function vmIP {
  let res=$1-98
  echo 10.10.10.$res
}

# Forward a full ip to an vm
# forwardIP <ip> <vm>
function forwardIP {
  if [[ $METHOD == "up" ]]; then
    iptables -t nat -A PREROUTING -d $1 -j DNAT --to-destination $(vmIP $2)
    iptables -A FORWARD -d $(vmIP $2) -j ACCEPT
    iptables -t nat -A POSTROUTING -s $(vmIP $2) -j SNAT --to-source $1
  fi
  if [[ $METHOD == "down" ]]; then
    iptables -t nat -D PREROUTING -d $1 -j DNAT --to-destination $(vmIP $2)
    iptables -D FORWARD -d $(vmIP $2) -j ACCEPT
    iptables -t nat -D POSTROUTING -s $(vmIP $2) -j SNAT --to-source $1
  fi
}

forwardIP <ip.one> 100 <-- The two ips from the host (10.10.10.2 and 10.10.10.250)
forwardIP <ip.two> 10.10.10.250 <-- (this is an a little bit outdated version, in the newer version the forwardIP also accepts ips instead of vm ids)

# [...]

# Forward only a port (range) to an vm
# forwardPort <ip> <vm> <port> <?port range end>
function forwardPort {
  if [[ -z "$4" ]]; then
    PORT_RANGE=$3
  else
    PORT_RANGE=$3:$4
  fi
  if [[ $METHOD == "up" ]]; then
    iptables -t nat -A PREROUTING -p tcp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -A FORWARD -p tcp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
    iptables -t nat -A PREROUTING -p udp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -A FORWARD -p udp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
  fi
  if [[ $METHOD == "down" ]]; then
    iptables -t nat -D PREROUTING -p tcp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -D FORWARD -p tcp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
    iptables -t nat -D PREROUTING -p udp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -D FORWARD -p udp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
  fi
}

# [...]

# Setup load balancing for outgoing traffic
IP_LIST+=("<main.ip>")
counter=0
for ip in "${IP_LIST[@]}"; do
  if [[ $METHOD == "up" ]]; then
    iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -m statistic --mode nth --every ${#IP_LIST[@]} --packet $counter -j SNAT --to-source $ip
  fi
  if [[ $METHOD == "down" ]]; then
    iptables -t nat -D POSTROUTING -s 10.10.10.0/24 -m statistic --mode nth --every ${#IP_LIST[@]} --packet $counter -j SNAT --to-source $ip
  fi
  ((counter++))
done

# Allow internal network traffic to VMs
iptables -A INPUT -s 10.10.10.0/24 -j ACCEPT
ip6tables -A INPUT -s foo:baa::/64 -j ACCEPT

# Allow traffic from VMs to external network
iptables -A FORWARD -i vmbr1 -o vmbr0 -j ACCEPT
ip6tables -A FORWARD -i vmbr1 -o vmbr0 -j ACCEPT

# Allow ICMP traffic
sudo iptables -A INPUT -p icmp -j ACCEPT
sudo iptables -A OUTPUT -p icmp -j ACCEPT

# Allow forwarding of established and related connections
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# [...]

esackbauer What OS is the VM? Any firewall on the VM itself active?

The vm is Debian 12, currently no firewall active, as i first wanted to get the services back online, before handling that.

esackbauer The rule is active for both directions?

Yes

esackbauer Can you ping 8.8.8.8 from the host VM?

Yes, without any problems

esackbauer

Sorry, I am no iptables expert. But it seems you broke something or NAT rules are not properly working.
Any iptables configs should go to the DOCKER-USER chain as mentioned here:
https://docs.mailcow.email/getstarted/prerequisite-system/?h=firewall#firewall-ports
See the red box.

scolastico

esackbauer Yeah but it isnt like i enabled any big firewall or heavy rules for that server. The “firewall” script i use is more a collection of forwarding rules. The firewall and so on should be handled by the vms themself (which isnt even active currently) from now on. And everything else works perfectly with the new setup, just mailcow “doesnt have internet” for no clear reasons.

esackbauer

Only thing that comes to my mind is do some packet capturing and put logs to very verbose, so you can see at what point packets get lost.

scolastico

esackbauer Since I had planned to move mailcow to its own vm soon anyway, I have now done so. The previous system had grown historically and the plan is currently to move all services from there to new vms. After the move and a little bit of debugging because I still had 10.10.10.2 from the old vm in a config at one point, everything went wonderfully. So I think it was some kind of problem with the double ip setup. However, it’s strange because the duplicate ip has already existed for 2 years, and has only changed from 192.168.3.2 and 192.168.3.3 to 10.10.10.2 and 10.10.10.250. And the previous setup also worked with it without any problems.

fearz

scolastico I have the same issue, which config did you modify?