My home network has evolved. I went from Xfinity’s all-in-one hardware (slow, monthly fee, ran hot), through Dynalink WRX-36 on stock firmware, to the same hardware running OpenWrt.
I don’t recall the exact pay-off period, but at the time I calculated that the one-time cost of buying the Dynalink hardware and cable modem would break even in about 2 years, which has now elapsed.
Moving to the Dynalink reduced ping times and variability. However, I put off trying OpenWrt for a long time. I was concerned about downtime for the rest of the household, as well as the potential to brick the hardware. I was motivated to finally try it by a desire to have less-trusted devices on the network, specifically a second-hand printer.
The flashing process took an hour or so, being careful with the steps. Configuring it took a few more hours; I had to build familiarity with the concepts, decide on the all-important funny SSID names, and reconnect all the IoT devices. I used AI (Claude) to build the plan, but didn’t give it direct access to the router.
Multiple Wireless SSIDs
OpenWrt allows you to configure as many SSIDs as your hardware allows. The WRX-36 has two radios, and I set up 3 networks across them:
Regular Network (lan)
Trusted devices go here. This is the only zone that is allowed intra zone forwarding, so it can access hosts on the other zones.
The stock firmware had blocked access to the cable modem’s admin page. Debugging WAN connectivity issues was annoying, because I had to physically connect a machine to the modem directly. This is not an issue with OpenWrt.
Guest Network (guest)
For visitors to use the Internet. Clients can’t communicate with each other, or access the router admin panel. Only the router’s UDP ports 53 and 67 (DNS and DHCP) are accessible to the guest network.
Limited Network (limited)
Clients can’t communicate with each other, or access the Internet. This network is for IoT devices, printers etc. that do not need direct Internet access. Exceptions can be carved out via Firewall rules for specific devices to access the Internet. It’s satisfying to check the logs and see these devices trying to connect to their Cloud backends in vain.
HomeAssistant & Tailscale
I ditched the horrible Kasa IoT app with its layout shifts and ad spam, and moved all IoT devices to a HomeAssistant setup on my Raspberry Pi.
I set up Tailscale on the router so I could access HomeAssistant away from home. Alternatively, I could have set up Tailscale on the Raspberry Pi.
DNS over HTTPS proxy
The router is configured to act as a DNS over HTTPS proxy, and DHCP registers the router as the default DNS resolver for all networks.
I connected an external USB HDD enclosure to the router, and used Claude to assist me in setting up a cryptsetup & LUKS encrypted volume. The key is in-memory, so every time the router power cycles I need to reconnect over SSH and run a small script to unlock the volume. The router and USB HDD are powered through a UPS, so this is not a concern. The volume is shared over Samba with the LAN and I get around 75MB/s read/write throughput. This is quite slow, but adequate for use as a backup disk. CPU use is very low because it has the native AES extensions.
I was pleasantly surprised to see new submenus show up in the OpenWrt Web UI to configure HDD Idle timeouts and Samba options.
In summary, I highly recommend OpenWrt and will be donating to the project.
This is how I got my external Apple Keyboard (pictured) to work with Xubuntu in a similar way to how it works with OSX on my Macbook at work. It’s not perfect, but it does avoid a lot of frustration from trying to use muscle memory OSX keyboard shortcuts on Linux. I am using a British layout keyboard so these changes might need to be adjusted depending on your locale.
Key re-mapping
I used XKB to change the following mappings of ‘real’ keys on the left to ‘perceived’ keys in the OS:
Swap Cmd <-> Left Ctrl
Left Ctrl -> Super/Menu/“Windows” key
Caps Lock -> Left Ctrl (I find this much more comfortable to use on a Mac keyboard, however here we have swapped Cmd and Ctrl anyway so it’s not so useful).
Left Alt+3 -> Hash (#)
Swap ± and ` (the default keymap appeared to have them the wrong way around)
I followed a useful guide to XKB to learn how to make the necessary modifications.
I made the following change to /usr/share/X11/xkb/symbols/gb:
NB: Changing files in /usr/share is not generally encouraged (your changes will affect other users on the system and can be overwritten by software upgrades) but I found this to be the most expedient solution at the time. Make a backup of /usr/share/X11/xkb/symbols/gb first by running:
cp /usr/share/X11/xkb/symbols/gb{,.bak}
I then edited /etc/default/keyboard to contain the following:
# Only XKBVARIANT and XKBOPTIONS needed to be changed
XKBMODEL="pc105"
XKBLAYOUT="gb"
XKBVARIANT="mac"
XKBOPTIONS="ctrl:swap_lwin_lctl,ctrl:nocaps"
BACKSPACE="guess"
Window Switching
Open the Xfce Settings manager -> Window Manager -> Keyboard:
Switch window for same application: ctrl + ` (reality is Cmd + `)
Cycle windows: Ctrl + tab (reality is Cmd + tab)
Cycle windows (reverse): Ctrl + shift + tab (reality is Cmd + shift + tab)
Spotlight
Open the “Keyboard -> Application Shortcuts” settings menu in Xfce. set xfce4-popup-whiskermenu to Ctrl + space (on your keyboard this will actually be Cmd + space)
Screenshots
I commonly take screenshots of an area of the screen with Cmd + Ctrl + Shift + 4 on OSX. You can achieve similar functionality by adding an Application Shortcut (like in the last step) in Xfce for xfce4-screenshooter -r -c as Ctrl + shift + super + 4
Changing fn key mode
I prefer to swap the fn key mode so that F keys do not activate their media functions unless the fn key is depressed.
This can be done by editing /etc/modprobe.d/hid_apple.conf to contain the following:
options hid_apple fnmode=2
Reboot for the change to take effect.
Changing mouse scroll speed
This is not strictly keyboard related, but I found that the default mouse scroll rate was much slower on Linux than on OSX. I changed it using the following instructions from the Unix Stackexchange.
Leftovers
Other things I’d like to do if I were to refine this setup:
Make F13 to F19 usable
Mimic the behaviour of the excellent SizeUp for OSX. I believe some of this is already possible in Xfce, however I found there were problematic conflicts with other applications using my chosen shortcuts of:
Cmd+Alt+left arrow -> window to left of screen
Cmd+Alt+right arrow -> window to right of screen
Cmd+Alt+m -> maximise
Get my common VSCode motion shortcuts working (should be possible in VSCode settings):
This is a quick script to resize all JPEGs in a folder recursively and output them to another folder. It will only resize new images on subsequent runs to save time. The settings in the example are to max width 1280px and file size 200KB.
This is a walkthrough of how I debugged and fixed intermittent HTTP 502 (Bad Gateway) errors on Google Kubernetes Engine (GKE).
Infrastructure Setup
GKE Kubernetes cluster
2 Nodes
1 Deployment, scaled to two pods. Each pod running a single Node.js-based HTTP server application
1 Service
1 GCE Ingress. It manages Google Cloud HTTP Load Balancers via the Kubernetes Ingress API. I was using a Network Endpoint Group (NEG) as a backend, which allows pods to be connected to from the load balancer directly.
The vanilla HTTP Load Balancer Architecture. In my setup, NEGs replace Instance Groups.
NEGs with Containers
Application Server Setup
Requests resulted in HTTP 502s seemingly at random. Running the load test suite was sufficient to reproduce the issue almost every time.
The HTTP Load Balancing docs have information about timeouts and retries. The Load Balancer keeps TCP connections idle for up to 10 minutes, therefore the application server’s timeout must be longer than this to avoid race conditions. My initial Node.js code to do this was as follows, but did not resolve the issue.
// https://nodejs.org/api/http.html#http_event_connection
server.on('connection', function(socket) {
// Set the socket timeout to 620 seconds
socket.setTimeout(620e3);
});
Checking for Known Issues
There was an open issue on the GCE Ingress GitHub with several ideas.
Some related to switching from externalTrafficPolicy: Cluster (which is the default for services) to externalTrafficPolicy: Local. By default, the GCE ingress will create an Instance Group targeting all nodes in the cluster. Any nodes not running a pod of the target Service need to forward traffic to another node which is. Using Network Endpoint Groups avoids this situation, as the pods are directly targeted.
There were also suggestions that nodes might be being terminated while receiving traffic (common if using pre-emptible machine types). That was not the issue in my case.
Checking the Logs
Stackdriver Logging creates logs for much of Google Cloud Platform by default, including HTTP Load Balancers:
This was puzzling since I had set the socket timeout in my application code. I opened a telnet session in the container to the server without sending any data and it was indeed closed after 620 seconds, indicating that the TCP timeout was set correctly.
The Server’s View
To see what was happening to these failed requests from the server’s view, I installed tshark (the CLI companion to Wireshark). I scaled down the deployment to a single pod, monitored the network traffic during a load-test run, and saved the output to a pcap file. kubectl cp makes it blessedly easy to download files from Kubernetes containers. I then opened the pcap file locally in the Wireshark GUI.
Looking for HTTP 502 errors in the trace would not be fruitful, because these errors were being sent by the load balancer, not the server. I tagged each request with a random X-My-Uuid header, and logged failed UUIDs during the load-test run.
Using a failed UUID as a display filter in Wireshark let me track down one of the failed requests. I then filtered the trace to only show packets from the same TCP connection.
The trace for the TCP connection containing the failed request. The second column is elapsed time in seconds. 130.211.0.143 is the load balancer. 10.56.5.6 is my server.
Two requests were served correctly in the space of 4 seconds. The failed request came 5 seconds later and resulted in a TCP RST from the server, closing the connection. This is the backend_connection_closed_before_data_sent_to_client seen in the Stackdriver logs.
Debugging the server and looking through the Node.js source for the HTTP server module yielded the following likely looking code in the ‘response finished’ callback:
server.keepAliveTimeout (default 5 seconds) was replacing the socket timeout I had set in the connection event listener whenever a request was received! Apparently this default keepAlive timeout was new in Node 8, there did not used to be a default timeout.
Setting the default timeout as follows resolved the issue:
Hosting Internet-accessible services (conventionally) requires a public IP address. In many cases, consumer Internet subscriptions are provided with a dynamic (rather than a static) IP address to alleviate IPv4 address exhaustion. The dynamic IP problem can be solved by using a dynamic DNS service such as No-IP which gives you a fixed hostname to use to find your server.
Most home Internet setups will incoporate a router with NAT (Network Address Translation). This allows multiple devices to share a single public IP. It also means that for inbound connections, port forwarding is needed to link external ports to specific devices on the private network. However, some ISPs, including my own (Hyperoptic in the UK) implement a Carrier Grade NAT (CGNAT). This means that multiple customers share a public IP address, and port forwarding is not possible. This is a major pain if you want to run public Internet services from home.
Fortunately there are workarounds - the easiest way to get around this is to use a pre-packaged reverse tunnelling solution such as Ngrok. There is a free version but there are limitations such as having to use a randomised hostname for your service, so I rolled my own system.
High Level Steps
Set up SSHD on your public server and allow TCP forwarding.
Set your home device up to connect persistently to the public server and allow remote tunnelling.
These steps are based on Ubuntu Server 16.10, so some steps may vary depending on your Linux distribution.
Setting up your public server
Edit /etc/ssh/sshd_config. Ensure that the line AllowTcpForwarding yes is present (if there is no mention of AllowTcpForwarding this is okay too as the default is allow). Also ensure that the line GatewayPorts clientspecified is present (otherwise the remote tunnel will only be accessible from localhost on the public server).
Create an SSH user and set up public key authentication. See a guide such as this one on DigitalOcean.
Ensure that if you have a firewall (including at service provider level, such as AWS Security Groups) the TCP port you want to access publicly is open.
Setting up your home device
As an example, we’ll run SSHD on the home device, so you can SSH straight into the home device via the public IP.
Install SSHD on your home device (many guides online).
Connect to the public server with SSH and setup the remote tunnel:
SSH does not handle unreliable connections very well by default, so you can use autossh which automatically restarts ssh if the connection to the external server fails.
(explainshell can’t do autossh arguments at the moment, but you can see the autossh man page).
Running autossh on startup
It’s useful to have autossh run on startup, so if your device restarts (as the Raspberry Pi can do often) the connection will be re-established. The steps here depend if you are using Sysvinit or Systemd. These steps work for Sysvinit which is what my Raspbian Wheezy installation is using.
(make permissions strict enough for ssh to accept them)
Add the public key to the user’s authorized_keys file on the public server:
no-pty,no-X11-forwarding,permitopen="255.255.255.255:9",command="/bin/false" <contents of id-autossh-rsa.pub>'
The sshd_config man page and sshd man page explain the options used. Essentially we only allow remote tunnels to be opened when using this key and disable running a useful shell. (Thanks to this article for the idea to disable the opening of local tunnels.)
Edit /etc/init.d/autossh and add the following, adjusting the TUNNEL_* and KEY_PATH variables to match your setup:
(Thanks to Clement-TS, whose init script this section derives from)
Conclusion
There are some limitations to the remote tunnelling approach:
Reverse tunnelling adds latency to your home services because all traffic needs to be routed through the public server. If you are running high-traffic services it may also cost you.
These solutions do not work for UDP traffic (I’m planning to do another article about UDP tunnelling over TCP).
However, it works pretty well with the robustness of autossh and is cost-efficient if you have a VPS or other server already running.
