Port Forwarding Behind a Carrier Grade NAT

Reading time: 5 minutes

Hosting Internet-accessible services (conventionally) requires a public IP address. In many cases, consumer Internet subscriptions are provided with a dynamic (rather than a static) IP address to alleviate IPv4 address exhaustion. The dynamic IP problem can be solved by using a dynamic DNS service such as No-IP which gives you a fixed hostname to use to find your server.

Most home Internet setups will incoporate a router with NAT (Network Address Translation). This allows multiple devices to share a single public IP. It also means that for inbound connections, port forwarding is needed to link external ports to specific devices on the private network. However, some ISPs, including my own (Hyperoptic in the UK) implement a Carrier Grade NAT (CGNAT). This means that multiple customers share a public IP address, and port forwarding is not possible. This is a major pain if you want to run public Internet services from home.

Fortunately there are workarounds - the easiest way to get around this is to use a pre-packaged reverse tunnelling solution such as Ngrok. There is a free version but there are limitations such as having to use a randomised hostname for your service, so I rolled my own system.

High Level Steps

  • Set up SSHD on your public server and allow TCP forwarding.
  • Set your home device up to connect persistently to the public server and allow remote tunnelling.

These steps are based on Ubuntu Server 16.10, so some steps may vary depending on your Linux distribution.

Setting up your public server

Edit /etc/ssh/sshd_config. Ensure that the line AllowTcpForwarding yes is present (if there is no mention of AllowTcpForwarding this is okay too as the default is allow). Also ensure that the line GatewayPorts clientspecified is present (otherwise the remote tunnel will only be accessible from localhost on the public server).

Create an SSH user and set up public key authentication. See a guide such as this one on DigitalOcean.

Ensure that if you have a firewall (including at service provider level, such as AWS Security Groups) the TCP port you want to access publicly is open.

Setting up your home device

As an example, we’ll run SSHD on the home device, so you can SSH straight into the home device via the public IP.

Install SSHD on your home device (many guides online).

Connect to the public server with SSH and setup the remote tunnel:

ssh -nNTv -R 0.0.0.0:2048:localhost:22 server.example.com

(explanation)

You should now be able to run

ssh -p 2048 server.example.com

to SSH into your home server!

Making it resilient

SSH does not handle unreliable connections very well by default, so you can use autossh which automatically restarts ssh if the connection to the external server fails.

Install autossh on the home device:

sudo apt-get update && sudo apt-get install autossh

Run autossh to connect to the public server:

autossh -M 0 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -nNTv -R 0.0.0.0:2048:localhost:22 server.example.com

(explainshell can’t do autossh arguments at the moment, but you can see the autossh man page).

Running autossh on startup

It’s useful to have autossh run on startup, so if your device restarts (as the Raspberry Pi can do often) the connection will be re-established. The steps here depend if you are using Sysvinit or Systemd. These steps work for Sysvinit which is what my Raspbian Wheezy installation is using.

Create a passwordless SSH key on the home device:

ssh-keygen -t rsa -b 4096 -f id-autossh-rsa -q -N ""

(explanation)

chmod 700 id-autossh-rsa

(make permissions strict enough for ssh to accept them)

Add the public key to the user’s authorized_keys file on the public server:

no-pty,no-X11-forwarding,permitopen="255.255.255.255:9",command="/bin/false" <contents of id-autossh-rsa.pub>'

The sshd_config man page and sshd man page explain the options used. Essentially we only allow remote tunnels to be opened when using this key and disable running a useful shell. (Thanks to this article for the idea to disable the opening of local tunnels.)

Edit /etc/init.d/autossh and add the following, adjusting the TUNNEL_* and KEY_PATH variables to match your setup:

#! /bin/sh
# author: Andrew Moss
# date: 06/05/2017
# source: https://gist.github.com/Clement-TS/48ae8d23f6452cd1a3a071640c1bd07b
# source: https://gist.github.com/suma/8134207
# source: http://stackoverflow.com/questions/34094792/autossh-pid-is-not-equal-to-the-one-in-pidfile-when-using-start-stop-daemon

### BEGIN INIT INFO
# Provides:          autossh
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: autossh initscript
# Description:       establish a tunnelled connection for remote access
### END INIT INFO

. /etc/environment
. /lib/init/vars.sh
. /lib/lsb/init-functions

TUNNEL_HOST=server.example.com
TUNNEL_USER=andrew
TUNNEL_PORT=2048
KEY_PATH=/home/andrew/.ssh/id-autossh-rsa

NAME=autossh
DAEMON=/usr/lib/autossh/autossh
AUTOSSH_ARGS="-M 0 -f"
SSH_ARGS="-nNTv -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no \
         -i $KEY_PATH -R 0.0.0.0:$TUNNEL_PORT:localhost:22 $TUNNEL_USER@$TUNNEL_HOST"

DESC="autossh for reverse ssh"
SCRIPTNAME=/etc/init.d/$NAME
DAEMON_ARGS=" $AUTOSSH_ARGS $SSH_ARGS"

# Export PID for autossh
AUTOSSH_PIDFILE=/var/run/$NAME.pid
export AUTOSSH_PIDFILE

do_start() {
    start-stop-daemon --start --background --name $NAME --exec $DAEMON --test > /dev/null || return 1
    start-stop-daemon --start --background --name $NAME --exec $DAEMON -- $DAEMON_ARGS    || return 2
}

do_stop() {
    start-stop-daemon --stop --name $NAME --retry=TERM/5/KILL/9 --pidfile $AUTOSSH_PIDFILE
    rm -f "$AUTOSSH_PIDFILE"
    RETVAL="$?"
    [ "$RETVAL" = 2 ] && return 2
    start-stop-daemon --stop --oknodo --retry=0/5/KILL/9 --exec $DAEMON
    [ "$?" = 2 ] && return 2
    return "$RETVAL"
}

case "$1" in
  start)
    log_daemon_msg "Starting $DESC" "$NAME"
    do_start
    case "$?" in
        0|1) log_end_msg 0 ;;
        2) log_end_msg 1 ;;
    esac
    ;;
  stop)
    log_daemon_msg "Stopping $DESC" "$NAME"
    do_stop
    case "$?" in
        0|1) log_end_msg 0 ;;
        2) log_end_msg 1 ;;
    esac
    ;;
  status)
    status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|status|restart}" >&2
    exit 3
    ;;
esac

Now run the following to have this run on startup, and also start it now:

sudo chmod +x /etc/init.d/autossh
sudo update-rc.d -f autossh defaults 90 90 > /dev/null 2>&1
sudo service autossh start

(Thanks to Clement-TS, whose init script this section derives from)

Conclusion

There are some limitations to the remote tunnelling approach:

  • Reverse tunnelling adds latency to your home services because all traffic needs to be routed through the public server. If you are running high-traffic services it may also cost you.
  • These solutions do not work for UDP traffic (I’m planning to do another article about UDP tunnelling over TCP).

However, it works pretty well with the robustness of autossh and is cost-efficient if you have a VPS or other server already running.


comments powered by Disqus