BGP over IPSec over WWAN; or how to connect your car to a datacenter

As a small deviation from my cradlepoint hacking, I took a break and worked on my mad networking skillz to bring up a much-needed tunnel from my vehicle to my colo datacenter. I have a number of network services that I don’t want to transit over the wider internet (a notoriously very insecure place), so an encrypted tunnel was mandatory. I also wanted to have dynamic routing so that I didn’t have to reconfigure the tunnel on each side every time I added a new network somewhere. Plus, I was hoping to have a mesh setup, so my car can find the best path through multiple sites.

My options for a tunnel were already very limited. IPIP is not encrypted, so that was out pretty much immediately. Cradlepoint doesn’t support Wireguard, nor does the border router at my datacenter, so that was out as well, as much as I wanted to use it. Cradlepoint supports OpenVPN, but I really hate that protocol for many reasons, plus the datacenter side doesn’t support it. So back to tried and true IPSec I went. I configured the tunnel as per best practice documentation, using strong encryption. I was able to bring up the tunnel between the DC and my car pretty quickly, but there was still one more asterisk - the dynamic routing.

I had a lot of options for routing protocols but I ended up on BGP, as I had actually never configured any sort of dynamic routing before and I wanted to apply BGP in a real world scenario, even if it was like using a hammer for everything that remotely resembles a nail. After converting the IPSec tunnel to a VTI on both ends, and setting an IP, I was able to bring up the BGP sessions. Some fiddling with prefix filtering later and I had a working setup.

So far I’m funneling syslog, location, and time/NTP across the tunnel from the router, but it also works as a dedicated tunnel back to the datacenter for true oh-crap situations. It’s really neat being able to basically make coverage plots off the Cradlepoint from the logs it sends back correlated with the GPS positions.

I found a deal recently on ebay for two IBR900s at a very good price. I bought them, and swapped my car’s modem with no issues. My partner expressed a desire for one as well, so I ended up tossing an IBR900 in my her car, and I did a pretty nice covert install. She has the same setup with BGP over IPSec over WWAN. The cell antennas were stick-on bars (see image below) that went on the top of the windshield inside, and the GPS antenna got tucked behind the A-pillar trim. The modem itself got mounted under the glove compartment above the passenger’s feet. I didn’t relocate the wifi antennas, as they were much less critical than the cell ones. I was amazed at the performance for being a covert install, I saw very good RSRP/SINR figures while I was out on a test drive. 20db SINR and -75dbm RSRP in the boonies isn’t bad at all. She’s been very happy with the setup as well, so I’m calling it a win.