recovering from a network change
This morning I moved the server that hosts my websites from a bookshelf to a standalone cabinet under the desk where I work from home. The reasons for doing so were: to cut noise from the server's fans; to shelter it better from dust; to take it off an overloaded power bar; and to get it away from the looming threat of a spill from the aquarium on top of the bookshelf should there be an earthquake.
Everything went fine with the hardware move, but things went sideways with the networking setup, which caused the server to stay offline from 12:30 on Saturday until I could address it this morning at 08:30. These were the symptoms:
- I could not reach the proxmox server that hosts the virtual machines.
- Those virtual machines could not see the Internet.
I eventually learned that what had happened during the move was that the host operating system (Debian Linux) created a second network interface (en2spo0 instead of en1sp0, which had disappeared). I believe that this happened because when I moved the server hardware, I took it off a direct connection to our ISP's router and attached it to a hub that was connected to our internal network's Wi-Fi router. This meant a change of network from 192.168.0.0/24 to 192.168.1.0/24. I thought this would be safe while I rewired the connection between the hub and the router, but it instead triggered some unexpected results.
The first thing that happened was the bridge network interface used on the proxmox operating system to provide network connectivity to the hosted VM's was invalidated. This happened because when the OS created a new network interface, the bridge interface was still pointing to the now absent original network interface. Again, I wasn't expecting the change from en1sp0 to ens2sp0, and that change did not get undone after I moved the hub from our Wi-Fi router to the ISP's router!
The next thing that happened -- and this is what took the most time to resolve -- was that the MAC address on each hosted VM also changed. The MAC address is used to uniquely identify the network interface. There was a time when there was a globally-unique MAC address for each physical network interface. While those days are gone, the MAC address is not meant to just change. I have no idea why this happened in this case, but it resulted in the firewall software on the ISP's router not recognizing the virtual machines on the network, and allowing them to use the IP address that I'd assigned to each. The firewall was essentially treating the VM's as if they were misconfigured or perhaps intruders.
On the whole, I'm glad to have learned a thing or two but I trust this setup less than I did prior: it seems too arbitrary and volatile for the purpose.