First of all, to edit /etc/network/interfaces as follow.
auto lo
iface lo inet loopback
iface eno1 inet manual
#vmbr0 is real IP Address
auto vmbr0
iface vmbr0 inet static
address 103.99.88.11/24
gateway 103.99.88.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
#vmbr999 is private sub network
auto vmbr999
iface vmbr999 inet static
address 192.168.100.1/24
bridge-ports none
bridge-stp off
bridge-fd 0
#Enable IP Forward
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.100.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.100.0/24' -o vmbr0 -j MASQUERADE
#In some masquerade setups with firewall enabled, conntrack zones might be needed for outgoing connections. Otherwise the firewall could block outgoing connections since they will prefer the POSTROUTING of the VM bridge (and not MASQUERADE).
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
After made manual changes directly to the /etc/network/interfaces file, apply them by running ifreload -a
Configure in VM (Share IP with PVE host)
For VM share IP with PVE host, assign the Network to vmbr999, then set IPv4 address inside VM as 192.168.100.0/24, Gateway is 192.168.100.1 which is IP address on vmbr999.
Configure in VM (Dedicated Real IP)
For VM used a dedicated real IP address, assign the Network to vmbr0, then set IPv4 address inside VM with real ip and real gateway (for example 103.99.88.12/24 and 103.99.88.1). As vmbr0 is bridge port with eno1, so the VM will also count as connected to external directly. Please aware that, in the case, the firewall may need to manage on VM Configure directly.
Port Forwarding
For Port Forwarding if VM sharing IP with PVE host, iptables can help on it.
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -d 103.99.88.11 --dport 2222 -j DNAT --to 10.2.1.2:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -d 103.99.88.11 --dport 2222 -j DNAT --to 10.2.1.2:22
Where -i is pointing to Linux Bridge of PVE host linked with physical port eno1;
-d is PVE host real IP address
–dport is coming port
–to is IP of VM to receive the service, and port of the service in VM.
For VM with Dedicated Real IP, no need to do the port forwarding, as all ports should pointing to dedicated VM configured the real ip.
Refer – https://pve.proxmox.com/wiki/Network_Configuration