Snaky This is a quite long I didn’t post anything on the blog. Therefore, I would prepare a summary for a record keep to list what I did recently.
Server Changed
The most important changed which is one of my server changed. I have a very small size VPS in OVH Cloud which used over a year, and recently the low cost discount ended. Therefore I looked for a change. At the very beginning, I approached the service provider in HK which I rent a dedicated server to see if they can rend a small size VPS for me to take over all the stuff in VPS in OVH Cloud. However, they may busy on their business right the way, so I took almost a week but cannot completed my server change over. After that, I went to Hetzner which is a service provider in Germany. They have a cloud VPS similar to OVH Cloud. I prepared almost everything including Proxmox Cluster, pfSense, Docker and mysql, however, I ended up find their VPS cannot enable hardware virtualization which OVH VPS can do it. Thus, I looked for another service provider. Finally, I back to OVH, but this time I rent a dedicated server, not VPS. This is because they are running a promotion for dedicated server but it need a bind a 12-month contract to get a discount and free setup fee. Anyways, I ran around the world then back to OVH.
Docker Swarm and SQL Cluster
I did enhanced my docker environment, which I group most of my PVE LXC into a new Docker Swarm. Previously, I have a lot of LXC to running a small services, like private cloud, private vault, private tunnel etc. Most of them are able to run in docker environment. Therefore I built three set of Docker and link as Docker Swarm, and then transited them into Docker Swarm. Some of them a required DB, so I built another three set of MYSQL, then built as a cluster. So one docker node with one mysql are now being a pair, and I have three pairs now. Any docker node and/or db down, it must have other to take over the container and db query.
Proxmox Backup Server
In before, I have two VMs (one is Win 2022 and one is Debian) take care all my backup job in Proxmox, and both are sync to cloud storage for offsite backup. I recently built two new PBS, and then used PBS sync function to have one way sync from PBS to PBS. In addition, I used Proxmox Backup Client to have a file based backup to PBS. I also did a weekly sync the datastore to cloud as offsite.
Wasabi Cloud
This is a S3 Compatible cloud storage provider, and the cost is relatively lower than other cloud storage provider. This is my first time to use this kind of service, and I found this is very useful. I ended created four buckets, and have four different sub-user account and key to access related bucket. I also enabled the IP restriction, so this is only able access from my trusted source IP address.
ESXi and vCenter Server Appliance
This is most likely a my new project. As mentioned below regarding the Server Change. I acquired a new dedicated server in Hetzner, but this time I didn’t attach it int my Proxmox Cluster. It has been separated and I installed ESXi. This is a hard task to install ESXi into cloud dedicated server. I need to required a KVM console from Hetzner, then mount the ISO to server remotely. During the installation, my PC and remote KVM cannot lost the connection, otherwise the ISO will broken. I took around 3-4 hours to complete the installation. Second challenge is IP arrangement. Hetzner provided one IP address only, this is used as ESXi management port already. So I need to request additional IP. I tried to use the first IP to be pfSense WAN IP, but failed, as Hetzner bind the MAC with IP. I can up and run everything with two IP. After a week later, I successfully assigned the first IP to pfSense WAN port, as I hard code the pfSense WAN port MAC address as ESXi management port MAC address. I ended up can release the second IP address for cost saving.
Nested Proxmox and Proxmox SDN
So what I did on ESXi? I built three Proxmox inside the ESXi and nested virtualization, and link as a new Proxmox Cluster. I have a lot of testing in Nested PVE, the most valuable is SDN. I have a pfSense on nested PVE, then 2 vxlan, one is simple zone with SNAT enabled, one is vxlan zone. Simple zone is as pfSense WAN, vxlan zone is for all VM including pfSense LAN. Therefore, all VM can have some subnet (192.168.0.0/24) to talk which is also able to communicate cross PVE node. Simple zone with SNAT function, so can talk to PVE node as well as public internet, this is function on pfSense WAN port. This is now still under testing, but so far so good.
pfSense with HA Proxy
The other thing I am testing on pfSense is HA Proxy. This is a plug in which I can install from pfSense package page. In pfSense on ESXi layer, I set one backend connection pointed to three nested PVE port 3389, and I set one frontend connection as WAN port 3389. Then I set one firewall rule on WAN to allow 3389 coming. So I can use RDP from anywhere connect to my ESXi public IP, then it will transfer from 1st pfsense (ESXi layer) to 2nd pfsense (inside nested pve), then transfer to one of my Windows VM. This is very interest, because everything on nested PVE, including 2nd pfsense, windows vm, debian vm, lxc can seat any pve node, keepalive. Sound good, and I still testing it.
This is a long post, and a lot information to keep and share. If anyone interest to know detail on above, I have write another post to explain in details how to build and how it works. 🙂