{"id":39,"date":"2023-09-17T21:21:32","date_gmt":"2023-09-17T18:21:32","guid":{"rendered":"https:\/\/blog.twis7.com\/?p=39"},"modified":"2023-10-08T19:00:10","modified_gmt":"2023-10-08T16:00:10","slug":"private-network-with-nat-for-vms-on-proxmox-with-single-public-ipv4","status":"publish","type":"post","link":"https:\/\/blog.twis7.com\/index.php\/2023\/09\/17\/private-network-with-nat-for-vms-on-proxmox-with-single-public-ipv4\/","title":{"rendered":"Private Network with NAT for VMs on Proxmox with Single Public IPv4"},"content":{"rendered":"

I have a dedicated server<\/a> with only 1 Public IPv4, without a router in front to manage the network. So I should use the power of iptables<\/a> and proxmox to create a private network for the VMs. Virtual machines will communicate with the world through NAT. Long story short:
Internet <-> vmbr0 (default bridge with static IP) <-> vmbr2(second bridge, that I will create) <-> Virtual Machines<\/em><\/p>\n

Proxmox Firewall, which is just iptables, will take care of network security.
Port forwarding? no worries, iptables will take care too.<\/p>\n\n\n

\"\"
network diagram<\/em><\/figcaption><\/figure>\n\n\n\n\n\n\n\n

Creating the bridge<\/h2>\n\n\n

Let’s get started<\/strong>! Our server should look like this (expand from menu Datacenter -> click on your node -> System -> Network<\/p>\n\n\n

\"\"<\/figure>\n\n\n

Click on create<\/em> and choose Linux Bridge<\/em>. It is crucial to leave Bridge ports empty<\/strong>, because we have only 1 Pubic IPv4 connected to the Network Device on the server.<\/p>\n\n\n

\"\"
What is Linux bridge, information from the internet:<\/em>
A Linux bridge is a kernel module that behaves like a network switch, forwarding packets between interfaces that are connected to it<\/strong>. It’s usually used for forwarding packets on routers, on gateways, or between VMs and network namespaces on a host -> read more<\/a>
A network bridge is a Link Layer device which forwards traffic between networks based on MAC addresses<\/em> and is therefore also referred to as a Layer 2 device -> read more<\/a>
In simple words, It is like the LAN on your home router.<\/em><\/strong><\/figcaption><\/figure>\n\n\n\n

Take care of the routing <\/h2>\n\n\n

so, our VMs from the private subnet 10.10.10.0\/24,\u00a0 can connect to internet.
First we need to enable IP forwarding<\/strong>. This is a way to make our Linux to act like a router.
SSH to the Proxmox Host or use Shell from web gui:<\/p>\n

edit \/etc\/sysctl.conf <\/em>with your favorite text editor and add this in the end of the file.<\/p>\n\n\n

<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
net<\/span>.<\/span>ipv4<\/span>.<\/span>ip_forward<\/span> <\/span>=<\/span> <\/span>1<\/span><\/span><\/code><\/pre><\/div>\n\n\n
After editing the file, you can run the following command to make the changes take effect right away.<\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
sysctl<\/span> <\/span>-<\/span>p<\/span><\/span><\/code><\/pre><\/div>\n\n\n
Configure the NAT <\/strong>with a method called MASQUERADE<\/strong>. In that way all virtual machines from the private network will connect to the internet from one single public IPv4.<\/p>\n
edit the file : \/etc\/network\/interfaces <\/em>adding this under VMBR2<\/strong><\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
post<\/span>-<\/span>up<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>A<\/span> <\/span>POSTROUTING<\/span> <\/span>-<\/span>s<\/span> <\/span>'<\/span>10.10.10.0\/24<\/span>'<\/span> <\/span>-<\/span>o<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>j<\/span> <\/span>MASQUERADE<\/span><\/span>\npost<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>D<\/span> <\/span>POSTROUTING<\/span> <\/span>-<\/span>s<\/span> <\/span>'<\/span>10.10.10.0\/24<\/span>'<\/span> <\/span>-<\/span>o<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>j<\/span> <\/span>MASQUERADE<\/span><\/span>\npost<\/span>-<\/span>up<\/span>   <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>raw<\/span> <\/span>-<\/span>I<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>fwbr<\/span>+<\/span> <\/span>-<\/span>j<\/span> <\/span>CT<\/span> <\/span>--<\/span>zone<\/span> <\/span>1<\/span><\/span>\npost<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>raw<\/span> <\/span>-<\/span>D<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>fwbr<\/span>+<\/span> <\/span>-<\/span>j<\/span> <\/span>CT<\/span> <\/span>--<\/span>zone<\/span> <\/span>1<\/span><\/span><\/code><\/pre><\/div>\n\n\n
The file \/etc\/network\/interfaces <\/em>should like something like this:<\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
auto<\/span> <\/span>vmbr0<\/span><\/span>\niface<\/span> <\/span>vmbr0<\/span> <\/span>inet<\/span> <\/span>static<\/span><\/span>\n\t<\/span>address<\/span> <\/span>202.202<\/span>.<\/span>202.202<\/span>\/<\/span>24<\/span><\/span>\n\t<\/span>gateway<\/span> <\/span>202.202<\/span>.<\/span>202.1<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>ports<\/span> <\/span>eno1<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>stp<\/span> <\/span>off<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>fd<\/span> <\/span>0<\/span><\/span>\n<\/span>\nauto<\/span> <\/span>vmbr2<\/span><\/span>\niface<\/span> <\/span>vmbr2<\/span> <\/span>inet<\/span> <\/span>static<\/span><\/span>\n\t<\/span>address<\/span> <\/span>10.10<\/span>.<\/span>10.1<\/span>\/<\/span>24<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>ports<\/span> <\/span>none<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>stp<\/span> <\/span>off<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>fd<\/span> <\/span>0<\/span><\/span>\n        <\/span>post<\/span>-<\/span>up<\/span>   <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>A<\/span> <\/span>POSTROUTING<\/span> <\/span>-<\/span>s<\/span> <\/span>'<\/span>10.10.10.0\/24<\/span>'<\/span> <\/span>-<\/span>o<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>j<\/span> <\/span>MASQUERADE<\/span><\/span>\n        <\/span>post<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>D<\/span> <\/span>POSTROUTING<\/span> <\/span>-<\/span>s<\/span> <\/span>'<\/span>10.10.10.0\/24<\/span>'<\/span> <\/span>-<\/span>o<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>j<\/span> <\/span>MASQUERADE<\/span><\/span>\n        <\/span>post<\/span>-<\/span>up<\/span>   <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>raw<\/span> <\/span>-<\/span>I<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>fwbr<\/span>+<\/span> <\/span>-<\/span>j<\/span> <\/span>CT<\/span> <\/span>--<\/span>zone<\/span> <\/span>1<\/span><\/span>\n        <\/span>post<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>raw<\/span> <\/span>-<\/span>D<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>fwbr<\/span>+<\/span> <\/span>-<\/span>j<\/span> <\/span>CT<\/span> <\/span>--<\/span>zone<\/span> <\/span>1<\/span><\/span><\/code><\/pre><\/div>\n\n\n
type ifreload -a <\/em>in the terminal to reload the interfaces or reboot the proxmox host.
If everything is fine, you will be able to start VMs with static IPs, that will have internet connection.
If it is not working try to add “post-up echo 1 > \/proc\/sys\/net\/ipv4\/ip_forward<\/em>” above the MASQUERADES rules.<\/p>\n<\/p>\n\n\n
Install and Configure the DHCP<\/h2>\n\n\n
We need DHCP for our VMs to take IP address automatically. For that we will use DNSMasq<\/a><\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
apt<\/span> <\/span>install<\/span> <\/span>dnsmasq<\/span> <\/span>-<\/span>y<\/span><\/span><\/code><\/pre><\/div>\n\n\n
edit the file \/etc\/dnsmasq.conf <\/em>and add the following in the end<\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
domain<\/span>=<\/span>proxmox<\/span>.<\/span>local<\/span><\/span>\ninterface<\/span>=<\/span>vmbr2<\/span><\/span>\ndhcp<\/span>-<\/span>range<\/span>=<\/span>10.10<\/span>.<\/span>10.10<\/span>,<\/span>10.10<\/span>.<\/span>10.220<\/span>,<\/span>12<\/span>h<\/span><\/span>\ndhcp<\/span>-<\/span>option<\/span>=<\/span>vmbr2<\/span>,<\/span>3<\/span>,<\/span>10.10<\/span>.<\/span>10.1<\/span><\/span>\nserver<\/span>=<\/span>1.1<\/span>.<\/span>1.1<\/span><\/span>\nserver<\/span>=<\/span>8.8<\/span>.<\/span>8.8<\/span><\/span>\ndhcp<\/span>-<\/span>leasefile<\/span>=\/var<\/span>\/<\/span>lib<\/span>\/misc\/dnsmasq.leases<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n\n\n
Enable and start the dnsmasq:<\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
systemctl<\/span> <\/span>enable<\/span> <\/span>dnsmasq<\/span> <\/span>&&<\/span> <\/span>systemctl<\/span> <\/span>start<\/span> <\/span>dnsmasq<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Activate Proxmox Firewall<\/h2>\n\n\n
Our proxmox host is with Public IP, which means that all ports are open to the world. Everybody from the internet can try to do nasty things with our hypervisor. To secure it, we will enable the built-in firewall.
Be very careful in this step, if something goes wrong you will be locked out without access to your host. If this happens, the only option will be to ask for IPMI activation to the host and remove the rules manually from the terminal.
Two ports are essential – 22 for ssh and 8006 for web-GUI. I will restrict them to be accessed only from my home static IP. My ISP never change my IP, because I am paying for it. If you do not have static IP, it is not a good idea to use dynamic IP. Maybe set a VPN, somewhere in the cloud and use it when access to proxmox is needed.<\/p>\n
Click on Datacenter -> Expand Firewall and click options
Firewall must be off \/\/ atleast for now<\/p>\n\n\n
\"\"<\/figure>\n\n\n
Click on Firewall -> Add | to make some firewall rules | 160.72.64.51 is my home IP (not really but..) and 8006 is the port for Proxmox web<\/p>\n\n\n
\"\"<\/figure>\n\n\n
Do the same but for SSH<\/p>\n\n\n
\"\"<\/figure>\n\n\n
It took me some time to understand why my DHCP is not working, ufc it was the f#**ing firewall, add this rules for VMBR2 <\/strong>interface<\/p>\n\n\n
\"\"<\/figure>\n\n\n
…and rule for DNS (DHCP needs it)<\/p>\n\n\n
\"\"<\/figure>\n\n\n
I will allow and the https traffic, because I will port forward port 443 to my webserver from private subnet. Every time you create a port forwarding, it is needed to create a rule in the firewall!<\/strong><\/p>\n\n\n
\"\"<\/figure>\n\n\n
Return to the menu Firewall -> Options and Enable the firewall. If everything is ok, you and only you, will have access to the proxmox administration. Nice tool<\/a> to test for open ports.<\/strong><\/p>\n\n\n
Port forwarding to VM<\/h2>\n\n\n
add at the end of the file \/etc\/network\/interfaces <\/em>where 10.10.10.200 is the webserver<\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
post<\/span>-<\/span>up<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>A<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>p<\/span> <\/span>tcp<\/span> <\/span>--<\/span>dport<\/span> <\/span>443<\/span> <\/span>-<\/span>j<\/span> <\/span>DNAT<\/span> <\/span>--<\/span>to<\/span>-<\/span>destination<\/span> <\/span>10.10<\/span>.<\/span>10.200<\/span>:<\/span>443<\/span><\/span>\npost<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>D<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>p<\/span> <\/span>tcp<\/span> <\/span>--<\/span>dport<\/span> <\/span>443<\/span> <\/span>-<\/span>j<\/span> <\/span>DNAT<\/span> <\/span>--<\/span>to<\/span>-<\/span>destination<\/span> <\/span>10.10<\/span>.<\/span>10.200<\/span>:<\/span>443<\/span><\/span><\/code><\/pre><\/div>\n\n\n
the file final \/etc\/network\/interfaces<\/em> should look like this<\/p>\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
auto<\/span> <\/span>lo<\/span><\/span>\niface<\/span> <\/span>lo<\/span> <\/span>inet<\/span> <\/span>loopback<\/span><\/span>\n<\/span>\niface<\/span> <\/span>eno1<\/span> <\/span>inet<\/span> <\/span>manual<\/span><\/span>\n<\/span>\niface<\/span> <\/span>eno2<\/span> <\/span>inet<\/span> <\/span>manual<\/span><\/span>\n<\/span>\nauto<\/span> <\/span>vmbr0<\/span><\/span>\niface<\/span> <\/span>vmbr0<\/span> <\/span>inet<\/span> <\/span>static<\/span><\/span>\n\t<\/span>address<\/span> <\/span>202.202<\/span>.<\/span>202.202<\/span>\/<\/span>24<\/span><\/span>\n\t<\/span>gateway<\/span> <\/span>202.202<\/span>.<\/span>202.1<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>ports<\/span> <\/span>eno1<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>stp<\/span> <\/span>off<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>fd<\/span> <\/span>0<\/span><\/span>\n<\/span>\nauto<\/span> <\/span>vmbr2<\/span><\/span>\niface<\/span> <\/span>vmbr2<\/span> <\/span>inet<\/span> <\/span>static<\/span><\/span>\n\t<\/span>address<\/span> <\/span>10.10<\/span>.<\/span>10.1<\/span>\/<\/span>24<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>ports<\/span> <\/span>none<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>stp<\/span> <\/span>off<\/span><\/span>\n\t<\/span>bridge<\/span>-<\/span>fd<\/span> <\/span>0<\/span><\/span>\n        <\/span>post<\/span>-<\/span>up<\/span>   <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>A<\/span> <\/span>POSTROUTING<\/span> <\/span>-<\/span>s<\/span> <\/span>'<\/span>10.10.10.0\/24<\/span>'<\/span> <\/span>-<\/span>o<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>j<\/span> <\/span>MASQUERADE<\/span><\/span>\n        <\/span>post<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>D<\/span> <\/span>POSTROUTING<\/span> <\/span>-<\/span>s<\/span> <\/span>'<\/span>10.10.10.0\/24<\/span>'<\/span> <\/span>-<\/span>o<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>j<\/span> <\/span>MASQUERADE<\/span><\/span>\n        <\/span>post<\/span>-<\/span>up<\/span>   <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>raw<\/span> <\/span>-<\/span>I<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>fwbr<\/span>+<\/span> <\/span>-<\/span>j<\/span> <\/span>CT<\/span> <\/span>--<\/span>zone<\/span> <\/span>1<\/span><\/span>\n        <\/span>post<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>raw<\/span> <\/span>-<\/span>D<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>fwbr<\/span>+<\/span> <\/span>-<\/span>j<\/span> <\/span>CT<\/span> <\/span>--<\/span>zone<\/span> <\/span>1<\/span><\/span>\n######## <\/span>open<\/span> <\/span>port<\/span> <\/span>443<\/span> <\/span>to<\/span> <\/span>the<\/span> <\/span>world<\/span><\/span>\n        <\/span>post<\/span>-<\/span>up<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>A<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>p<\/span> <\/span>tcp<\/span> <\/span>--<\/span>dport<\/span> <\/span>443<\/span> <\/span>-<\/span>j<\/span> <\/span>DNAT<\/span> <\/span>--<\/span>to<\/span>-<\/span>destination<\/span> <\/span>10.10<\/span>.<\/span>10.200<\/span>:<\/span>443<\/span><\/span>\n        <\/span>post<\/span>-<\/span>down<\/span> <\/span>iptables<\/span> <\/span>-<\/span>t<\/span> <\/span>nat<\/span> <\/span>-<\/span>D<\/span> <\/span>PREROUTING<\/span> <\/span>-<\/span>i<\/span> <\/span>vmbr0<\/span> <\/span>-<\/span>p<\/span> <\/span>tcp<\/span> <\/span>--<\/span>dport<\/span> <\/span>443<\/span> <\/span>-<\/span>j<\/span> <\/span>DNAT<\/span> <\/span>--<\/span>to<\/span>-<\/span>destination<\/span> <\/span>10.10<\/span>.<\/span>10.200<\/span>:<\/span>443<\/span><\/span><\/code><\/pre><\/div>\n\n\n
“ifreload -a” <\/em>to activate the changes. 
And that should be it. <\/p>\n
Enjoy!<\/p>","protected":false},"excerpt":{"rendered":"
I have a dedicated server with only 1 Public IPv4, without a router in front to manage the network. So I should use the power of iptables and proxmox to create a private network for the VMs. Virtual machines will communicate with the world through NAT. Long story short:Internet <-> vmbr0 (default bridge with static… Continue reading Private Network with NAT for VMs on Proxmox with Single Public IPv4<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,13],"tags":[12,15,14],"class_list":["post-39","post","type-post","status-publish","format-standard","hentry","category-debian","category-proxmox","tag-dedicated-server","tag-network-with-single-ipv4","tag-proxmox"],"_links":{"self":[{"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/posts\/39","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/comments?post=39"}],"version-history":[{"count":35,"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/posts\/39\/revisions"}],"predecessor-version":[{"id":94,"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/posts\/39\/revisions\/94"}],"wp:attachment":[{"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/media?parent=39"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/categories?post=39"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.twis7.com\/index.php\/wp-json\/wp\/v2\/tags?post=39"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
The part with “post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT –zone 1<\/em>” is very important, because we will activate the proxmox firewall later. From the proxmox documentation<\/a>:
“In some masquerade setups with firewall enabled, conntrack zones might be needed for outgoing connections. Otherwise the firewall could block outgoing connections since they will prefer the POSTROUTING<\/span><\/tt> of the VM bridge (and not MASQUERADE<\/span><\/tt>).”<\/strong><\/em><\/p>\n