Port Forwarding Behind a Carrier Grade NAT

6 May 2017

Hosting Internet-accessible services (conventionally) requires a public IP address. In many cases, consumer Internet subscriptions are provided with a dynamic (rather than a static) IP address to alleviate IPv4 address exhaustion. The dynamic IP problem can be solved by using a dynamic DNS service such as No-IP which gives you a fixed hostname to use to find your server.

Most home Internet setups will incoporate a router with NAT (Network Address Translation). This allows multiple devices to share a single public IP. It also means that for inbound connections, port forwarding is needed to link external ports to specific devices on the private network. However, some ISPs, including my own (Hyperoptic in the UK) implement a Carrier Grade NAT (CGNAT). This means that multiple customers share a public IP address, and port forwarding is not possible. This is a major pain if you want to run public Internet services from home.

Fortunately there are workarounds - the easiest way to get around this is to use a pre-packaged reverse tunnelling solution such as Ngrok. There is a free version but there are limitations such as having to use a randomised hostname for your service, so I rolled my own system.

High Level Steps

  • Set up SSHD on your public server and allow TCP forwarding.
  • Set your home device up to connect persistently to the public server and allow remote tunnelling.

These steps are based on Ubuntu Server 16.10, so some steps may vary depending on your Linux distribution.

Setting up your public server

Edit /etc/ssh/sshd_config. Ensure that the line AllowTcpForwarding yes is present (if there is no mention of AllowTcpForwarding this is okay too as the default is allow). Also ensure that the line GatewayPorts clientspecified is present (otherwise the remote tunnel will only be accessible from localhost on the public server).

Create an SSH user and set up public key authentication. See a guide such as this one on DigitalOcean.

Ensure that if you have a firewall (including at service provider level, such as AWS Security Groups) the TCP port you want to access publicly is open.

Setting up your home device

As an example, we’ll run SSHD on the home device, so you can SSH straight into the home device via the public IP.

Install SSHD on your home device (many guides online).

Connect to the public server with SSH and setup the remote tunnel:

ssh -nNTv -R 0.0.0.0:2048:localhost:22 server.example.com

(explanation)

You should now be able to run

ssh -p 2048 server.example.com

to SSH into your home server!

Making it resilient

SSH does not handle unreliable connections very well by default, so you can use autossh which automatically restarts ssh if the connection to the external server fails.

Install autossh on the home device:

sudo apt-get update && sudo apt-get install autossh

Run autossh to connect to the public server:

autossh -M 0 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -nNTv -R 0.0.0.0:2048:localhost:22 server.example.com

(explainshell can’t do autossh arguments at the moment, but you can see the autossh man page).

Running autossh on startup

It’s useful to have autossh run on startup, so if your device restarts (as the Raspberry Pi can do often) the connection will be re-established. The steps here depend if you are using Sysvinit or Systemd. These steps work for Sysvinit which is what my Raspbian Wheezy installation is using.

Create a passwordless SSH key on the home device:

ssh-keygen -t rsa -b 4096 -f id-autossh-rsa -q -N ""

(explanation)

chmod 700 id-autossh-rsa

(make permissions strict enough for ssh to accept them)

Add the public key to the user’s authorized_keys file on the public server:

no-pty,no-X11-forwarding,permitopen="255.255.255.255:9",command="/bin/false" <contents of id-autossh-rsa.pub>'

The sshd_config man page and sshd man page explain the options used. Essentially we only allow remote tunnels to be opened when using this key and disable running a useful shell. (Thanks to this article for the idea to disable the opening of local tunnels.)

Edit /etc/init.d/autossh and add the following, adjusting the TUNNEL_* and KEY_PATH variables to match your setup:

#! /bin/sh
# author: Andrew Moss
# date: 06/05/2017
# source: https://gist.github.com/Clement-TS/48ae8d23f6452cd1a3a071640c1bd07b
# source: https://gist.github.com/suma/8134207
# source: http://stackoverflow.com/questions/34094792/autossh-pid-is-not-equal-to-the-one-in-pidfile-when-using-start-stop-daemon

### BEGIN INIT INFO
# Provides:          autossh
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: autossh initscript
# Description:       establish a tunnelled connection for remote access
### END INIT INFO

. /etc/environment
. /lib/init/vars.sh
. /lib/lsb/init-functions

TUNNEL_HOST=server.example.com
TUNNEL_USER=andrew
TUNNEL_PORT=2048
KEY_PATH=/home/andrew/.ssh/id-autossh-rsa

NAME=autossh
DAEMON=/usr/lib/autossh/autossh
AUTOSSH_ARGS="-M 0 -f"
SSH_ARGS="-nNTv -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no \
         -i $KEY_PATH -R 0.0.0.0:$TUNNEL_PORT:localhost:22 $TUNNEL_USER@$TUNNEL_HOST"

DESC="autossh for reverse ssh"
SCRIPTNAME=/etc/init.d/$NAME
DAEMON_ARGS=" $AUTOSSH_ARGS $SSH_ARGS"

# Export PID for autossh
AUTOSSH_PIDFILE=/var/run/$NAME.pid
export AUTOSSH_PIDFILE

do_start() {
    start-stop-daemon --start --background --name $NAME --exec $DAEMON --test > /dev/null || return 1
    start-stop-daemon --start --background --name $NAME --exec $DAEMON -- $DAEMON_ARGS    || return 2
}

do_stop() {
    start-stop-daemon --stop --name $NAME --retry=TERM/5/KILL/9 --pidfile $AUTOSSH_PIDFILE
    rm -f "$AUTOSSH_PIDFILE"
    RETVAL="$?"
    [ "$RETVAL" = 2 ] && return 2
    start-stop-daemon --stop --oknodo --retry=0/5/KILL/9 --exec $DAEMON
    [ "$?" = 2 ] && return 2
    return "$RETVAL"
}

case "$1" in
  start)
    log_daemon_msg "Starting $DESC" "$NAME"
    do_start
    case "$?" in
        0|1) log_end_msg 0 ;;
        2) log_end_msg 1 ;;
    esac
    ;;
  stop)
    log_daemon_msg "Stopping $DESC" "$NAME"
    do_stop
    case "$?" in
        0|1) log_end_msg 0 ;;
        2) log_end_msg 1 ;;
    esac
    ;;
  status)
    status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|status|restart}" >&2
    exit 3
    ;;
esac

Now run the following to have this run on startup, and also start it now:

sudo chmod +x /etc/init.d/autossh
sudo update-rc.d -f autossh defaults 90 90 > /dev/null 2>&1
sudo service autossh start

(Thanks to Clement-TS, whose init script this section derives from)

Conclusion

There are some limitations to the remote tunnelling approach:

  • Reverse tunnelling adds latency to your home services because all traffic needs to be routed through the public server. If you are running high-traffic services it may also cost you.
  • These solutions do not work for UDP traffic (I’m planning to do another article about UDP tunnelling over TCP).

However, it works pretty well with the robustness of autossh and is cost-efficient if you have a VPS or other server already running.


Port Forwarding Behind a Carrier Grade NAT - Comments

A Brief Overview of Container Orchestration

4 March 2017

This article aims to provide an overview of some of the problems encountered when running containers in production.

Containers isolate applications by providing separate user-spaces (rather than entirely separate operating system instances, as in full virtualisation). This can yield benefits in security, repeatability1 and efficient resource utilisation.

I know about Docker, but what is an ‘orchestration system’, and why would I need one?

An orchestration system helps you run production services, in containers, as part of clusters. They can be thought of as the next layer up in the operational stack from manual container usage.

Service Registration and Health Checks

A service might consist of one or more container definitions2 which define the container images that are to be run as well as additional metadata such as CPU and memory limits and storage attachments.

Container Orchestration systems allow registering containers as part of a service, which acts as a logical unit for autoscaling and load balancing. Services are composed of a set of containers, with the goal being to maintain a desired number of containers running. The individual containers should be considered ephemeral (a good practice in general when running server applications3) as they can be terminated and replaced at any time. The adage containers should be cattle, not pets encapsulates this philosophy.

Exposing an interface for the orchestration system to check your containers’ health is crucial for many features to work effectively. A simple HTTP endpoint can be used to check if a container responds in a timely manner with a 200 OK, indicating it is able to service user requests.

Service Discovery may also be integrated to allow your applications to find each other easily in the cluster without additional tooling.

Scheduling

Placement strategies allow schedulers to decide which servers4 your containers will run on.

These can vary depending on the goals of your service. You may want to spread containers as diffusely as possible across the available server pool to minimise the impact of a crashed server. Or you might want to bin pack containers into as few servers as possible to reduce costs.

Deployments & Upgrades

Real applications need to be deployed more than once. Container orchestration systems often provide mechanisms for:

  • Automated blue/green5 redeployments of services, including verifying that the new containers are working before terminating all the old ones by integrating with health checks.
  • Automatic restarting of crashed containers (if a whole server has crashed, for example, Docker’s built-in restart is not sufficient)
  • Connection draining from old containers to avoid interruptions to user sessions.
  • Rapid rollbacks if needed.

Auto Scaling

One of the big advantages of cloud computing is the ability to elastically adjust capacity based on demand, bringing cost savings in troughs and meeting demand at peak times. For container clusters, this involves adding or removing containers as well as the underlying servers which provide the resources.

Automatic scaling actions may be defined based on:

  • CPU/Memory Usage - what resources are the containers actually using?
  • CPU/Memory Reservation - what do the container definitions say that the containers need?
  • Time schedules - if your demand is predictable you can preemptively ‘warm up’ more containers to increase service capacity.

Grouping of Containers

It is often useful to group a set of containers with different definitions together to work as a whole, for example, having a web server container and a log drain container running side-by-side. A Kubernetes pod (services are collections of pods) and an Amazon ECS task definition can both group multiple container definitions.

Notes on Software and Providers

I wrote this article as part of research into available options and am not intimately familiar with all of these products. If you spot anything I’ve written which seems incorrect, please let me know. I have used ECS most heavily out of the following.

Product Notes Billing Related
Kubernetes At the heart of many other offerings, seems like a solid bet for portability is probably the most popular tool in its class. Open Source Google Borg
Docker Swarm (now part of Docker engine as of 1.12) Open Source
Google Container Engine Hosted Kubernetes with additional integrations with Google Cloud Flat fee per cluster hour + compute Kubernetes
Amazon ECS Largely proprietary (open source ecs-agent) - heavily integrated with other AWS products (ALB, IAM, ASG) Compute Usage Hours (EC2) Host agent is open-source (ecs-agent)
Microsoft Azure Container Service Compute Usage Hours Docker Swarm, DC/OS, or Kubernetes
Apache Mesos Not specific to containers - pitched as a ‘distributed systems kernel’ for co-ordinating compute resources generically. Open Source
Marathon Container orchestration built on Mesos.
Mesosphere Makers of DC/OS (Data Center Operating System) which uses Mesos. Enterprise (support plans & deployment footprint based) Apache Mesos
Rancher Open source with multiple base options - seems to bear some similarity to a self-hosted Azure Container Service. Open Source & Premium Support Kubernetes, Swarm, Mesos

Conclusion

I’ve outlined some of the problems that this plethora of tools (many of which you may have heard of) are trying to solve. The feature sets are broadly similar across several of them, so I would simply advise reading the docs thoroughly and evaluating the risk of vendor lock-in when choosing how to invest your time.


  1. The full runtime environment of your application is defined in one place, rather than being an accumulation of scripting and manual changes to servers over time. [return]
  2. In Amazon ECS, these are called Task Definitions. [return]
  3. https://12factor.net/disposability [return]
  4. In Amazon ECS, these are called Container Instances. [return]
  5. https://martinfowler.com/bliki/BlueGreenDeployment.html [return]

A Brief Overview of Container Orchestration - Comments

Enable remote SSH access to Ubuntu 14.04 LTS Live

18 October 2015

Steps to enable remote SSH access to a computer running Ubuntu 14.04 Live. Useful for helping non-technical people remotely:

# Press windows key or click the top left, type 'Term'. Open 'Terminal'

sudo -i

apt-get update -y && apt-get -y install openssh-server
passwd root

# Type a password, press enter. Retype it, press enter

sed -i 's/PermitRootLogin .*/PermitRootLogin yes/g' /etc/ssh/sshd_config

service ssh restart

# Get their IP
curl ifconfig.co

# Setup port forwarding on their router to get access
# ssh [email protected]
# Enable public key auth only, create a new user and disable root login when you have gained access

Enable remote SSH access to Ubuntu 14.04 LTS Live - Comments

OpenVPN with DNS AdBlocking using Docker

18 October 2015

OpenVPN and DNS AdBlocking is a useful way to block ads on your smartphone without having to root it. This post describes how to setup such a service on your own server.

The idea is to set a DNS server in your OpenVPN DHCP options to push to clients. The DNS server runs in another Docker container and uses hosts files to block ads, trackers etc.

  1. See https://www.digitalocean.com/community/tutorials/how-to-set-up-an-openvpn-server-on-ubuntu-14-04 as an example of how to set up an OpenVPN Docker container on a Ubuntu VPS. At the ovpn_genconfig step, set -n 8.8.8.8 so there is only a single placeholder DNS server to overwrite later on. Otherwise your settings will fallback to Google’s secondary DNS.

  2. Setup the DNS container, this uses dnsmasq to block the bad hosts:

    git clone https://github.com/arthurkay/sagittarius-A && cd sagittarius-A && ./build.sh
    
  3. Run the dnsmasq container:

    docker rm saga-dns; docker run --restart=always --name=saga-dns --expose 53 --cap-add=NET_ADMIN arthurkay/sagittarius-a &
    

We expose port 53 explicitly as the file does not currently contain an EXPOSE directive.

  1. Run the OpenVPN container, linking to the saga-dns container:

    docker rm openvpn; docker run --restart=always --volumes-from ovpn-data --name openvpn --link saga-dns:saga-dns -p 1194:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn bash -c 'sed -i -E "s/(push dhcp-option DNS).*/\1 $SAGA_DNS_PORT_53_TCP_ADDR/" /etc/openvpn/openvpn.conf && ovpn_run' &
    

This updates the saga-dns container’s IP in the OpenVPN config before running OpenVPN.

(Hopefully) enjoy much faster browsing and less tracking on your mobile devices.


OpenVPN with DNS AdBlocking using Docker - Comments

Delete fdupes duplicates by directory

24 May 2015

A quick script to process fdupes output and allow interactive selection of files to delete. Differs from the built-in fdupes prompts in that you can select directories to condemn.


Delete fdupes duplicates by directory - Comments