scolastico

esackbauer How was the setup before you “updated”?

internet <-> proxmox <-> opnsense <-> vms

esackbauer What firewall are you using now, and does it allow outgoing traffic on ICMP and Port 53 TCP/UDP?

A script:

#!/bin/bash

METHOD=$1

# Flush existing rules
iptables -F
ip6tables -F

# Drop anything which is not allowed
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow all localhost traffic
iptables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -i lo -j ACCEPT

# Allow established and related traffic
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
ip6tables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow forwarding
sysctl -w net.ipv6.conf.default.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.default.proxy_ndp=1
sysctl -w net.ipv6.conf.all.proxy_ndp=1
sysctl -w net.ipv4.ip_forward=1

if [[ $METHOD == "up" ]]; then
  iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
  ip -6 route add foo:baa::/64 dev vmbr0
fi
if [[ $METHOD == "down" ]]; then
  iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
  ip -6 route del foo:baa::/64 dev vmbr0
fi

# Setup additional ip's
IP_LIST=(
  "xxx"
  "xxx"
  "xxx"
  "xxx"
  "xxx"
)

for ip in "${IP_LIST[@]}"; do
  if [[ $METHOD="up" ]]; then
    ip addr add $ip dev vmbr0
  fi
  if [[ $METHOD="down" ]]; then
    ip addr del $ip dev vmbr0
  fi
done

# Helper to get the vm ip
# vmIP <vm>
function vmIP {
  let res=$1-98
  echo 10.10.10.$res
}

# Forward a full ip to an vm
# forwardIP <ip> <vm>
function forwardIP {
  if [[ $METHOD == "up" ]]; then
    iptables -t nat -A PREROUTING -d $1 -j DNAT --to-destination $(vmIP $2)
    iptables -A FORWARD -d $(vmIP $2) -j ACCEPT
    iptables -t nat -A POSTROUTING -s $(vmIP $2) -j SNAT --to-source $1
  fi
  if [[ $METHOD == "down" ]]; then
    iptables -t nat -D PREROUTING -d $1 -j DNAT --to-destination $(vmIP $2)
    iptables -D FORWARD -d $(vmIP $2) -j ACCEPT
    iptables -t nat -D POSTROUTING -s $(vmIP $2) -j SNAT --to-source $1
  fi
}

forwardIP <ip.one> 100 <-- The two ips from the host (10.10.10.2 and 10.10.10.250)
forwardIP <ip.two> 10.10.10.250 <-- (this is an a little bit outdated version, in the newer version the forwardIP also accepts ips instead of vm ids)

# [...]

# Forward only a port (range) to an vm
# forwardPort <ip> <vm> <port> <?port range end>
function forwardPort {
  if [[ -z "$4" ]]; then
    PORT_RANGE=$3
  else
    PORT_RANGE=$3:$4
  fi
  if [[ $METHOD == "up" ]]; then
    iptables -t nat -A PREROUTING -p tcp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -A FORWARD -p tcp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
    iptables -t nat -A PREROUTING -p udp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -A FORWARD -p udp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
  fi
  if [[ $METHOD == "down" ]]; then
    iptables -t nat -D PREROUTING -p tcp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -D FORWARD -p tcp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
    iptables -t nat -D PREROUTING -p udp --dport $PORT_RANGE -d $1 -j DNAT --to-destination $(vmIP $2):$PORT_RANGE
    iptables -D FORWARD -p udp --dport $PORT_RANGE -d $(vmIP $2) -j ACCEPT
  fi
}

# [...]

# Setup load balancing for outgoing traffic
IP_LIST+=("<main.ip>")
counter=0
for ip in "${IP_LIST[@]}"; do
  if [[ $METHOD == "up" ]]; then
    iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -m statistic --mode nth --every ${#IP_LIST[@]} --packet $counter -j SNAT --to-source $ip
  fi
  if [[ $METHOD == "down" ]]; then
    iptables -t nat -D POSTROUTING -s 10.10.10.0/24 -m statistic --mode nth --every ${#IP_LIST[@]} --packet $counter -j SNAT --to-source $ip
  fi
  ((counter++))
done

# Allow internal network traffic to VMs
iptables -A INPUT -s 10.10.10.0/24 -j ACCEPT
ip6tables -A INPUT -s foo:baa::/64 -j ACCEPT

# Allow traffic from VMs to external network
iptables -A FORWARD -i vmbr1 -o vmbr0 -j ACCEPT
ip6tables -A FORWARD -i vmbr1 -o vmbr0 -j ACCEPT

# Allow ICMP traffic
sudo iptables -A INPUT -p icmp -j ACCEPT
sudo iptables -A OUTPUT -p icmp -j ACCEPT

# Allow forwarding of established and related connections
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# [...]

esackbauer What OS is the VM? Any firewall on the VM itself active?

The vm is Debian 12, currently no firewall active, as i first wanted to get the services back online, before handling that.

esackbauer The rule is active for both directions?

Yes

esackbauer Can you ping 8.8.8.8 from the host VM?

Yes, without any problems

I’v recently updated how my VM’s reach the internet in a setup kinda like
internet <-> proxmox <-> vm’s

Every vm, and every other container runs absolutly fine on the new setup. Except mailcow or to be more exact unbound. Unbound fails its healthcheck, and if i do a docker exec and check a ping, it seems it cant reach anything outside the vm.

8.8.8.8, Google: FAIL
10.10.10.1, Proxmox (and Gateway): FAIL
10.10.10.2, The host vm where mailcow runs: SUCCESS
10.10.10.250, The host vm where mailcow runs: SUCCESS

Any ideas?

Oh and it cant be a firewall issue on proxmox itself as it has an “allow all” rule for icmp traffic. So pings should always work.