[Ovmsdev] Reverse connections (module -> developer)

Stephen Casner casner at acm.org
Tue Feb 4 07:45:52 HKT 2020


At the job from which I retired we made heavy use of SSH port
forwarding and reverse ssh clients to allow support access to our
products installed at customer sites.  To achieve an acceptable level
of security we dedicated one Linux server at our HQ to be the target
of "phone home" SSH connections from the customer units.  This server
exposed only the sshd service and allowed only key-based logins, not
passwords.  For customers who agreed to allow this access inside their
networks, the unit would maintain a continuous SSH connection to the
server, automatically reconnecting after an outage.

A Perl script named phonehome on the customer unit established and
maintained the ssh connection using an unlocked private key.  The
login shell for the target account on the server was another Perl
script, phonehomed, so if a customer unit were compromised then an
attacker who reached the server would not be able to do anything.
Phonehome sent along some identity information to the server so we
could tell which customer it was, and phonehomed would just send back
"success" to indicate successful connection up to the application
level and then sleep.

The phonehome client script would randomly select a port number and
use that to establish port forwarding from that port number at the
server end to port 22 on the customer unit.  With many customer units
calling in, there would occasionallly be a collision for the random
port number, in which case phonehomed would send "collision" and close
the connection so the phonehome script would try again.

Support staff would be allowed access to the server.  A script named
et on the server listed all the open connections from customer units
with the randomly chosen port numbers so we'd know how to connect to
the desired customer using ssh to localhost at that port.  No SSH
private keys were allowed to be stored on the server; instead, staff
were required to use ssh-agent forwarding to connect from their own
computers through the server to the customer units.

Now, to relate all this back to the problem at hand, perhaps we could
have a server somewhere with a similar setup.  This would avoid the
need for multiple developers to open holes in their personal NAT
and/or firewall to allow incoming connections from remote OVMS units.
Rather than having all the OVMS units try to maintain call-back
connections to that server all the time, the existing ovms v2 protocol
shell provides sufficient access to initiate a call back when one is

A prerequisite for developing such a service would be for me to get
off my duff and update the SSH implementation in OVMS to use the
current WolfSSH release.  They took back the extensions I made to
their code to support SCP, but they made some changes that I would
need to adapt our code to fit.  They added SFTP support, too, but I
have not studied it to see if it can fit into task structure.

                                                        -- Steve

On Mon, 3 Feb 2020, Mark Webb-Johnson wrote:

> Seeking opinions (in particular Stephen, our ssh guru).
> As I (and others) are doing more work remotely (OVMS modules in cars, and trying to connect in remotely to diagnose, test, or develop), I am seeing the same issue come up often. If the car is on your own home WiFi, it is simple and easy. But if the car is elsewhere, on someone else’s WiFi, then connecting _to_ the car becomes hard. Firewalls, port forwarding, and all that mess. Security also becomes an issue (once ports are opened and forwarded).
> The core requirements seem to be:
> A way to get a proper shell. The ovms v2 protocol shell is ok, but quite painful to work with (especially as logs are not shown and there is no command expansion).
> A way to get CAN dumps out of the car. Can save to sd card, but can’t copy in/out.
> Both of these are simple TCP connections.
> The second issue (CAN dumps) is resolvable if the developer has a public address. Open the firewall, ’nc -l’ a port, and ‘can log start tcpclient …’ on the module. But a little kludgy, and doesn’t support SavvyCAN (which can only connect _to_ the module, not receive a connection _from_ the module).
> I can see three possible approaches to this:
> Extension of OVMS v2 protocol
> We could extend the OVMS v2 protocol to support a PIPE mode. Have it allow connections and data to be transferred over the v2 protocol link using simple messages. Then, the pipe could be used to tunnel SSH and CAN log connections. The same could be done with v3 over MQTT (the transport is different, but the console / port forwarding bits inside the ovms module are the same).
> The advantage here is that the OVMS v2 server becomes a man-in-the-middle broker. Both the car module and the developer connect to the OVMS v2 server to exchange data. No incoming connections, so everything can be dynamic.
> The disadvantage is security is fairly weak on that protocol. Very dated RC4 encryption and poor authentication.
> Leverage SSH
> SSH has a port forwarding facility built in, that wolfssh seems to support. It is also possible to do a reverse ssh client.
> The advantage here is security (very strong encryption and authentication).
> The disadvantage is complexity and the fact that the developer end needs to poke a hole in their firewall (and possibly do NAT port forwarding, etc) to allow the incoming connections from the car.
> STUNNEL / WireGuard / Mosh / VPN / etc
> Perhaps someone else knows of something suitable? The issues here are that the LWIP stack is not the easiest to work with.
> My initial preference is #1 or #2, but open to discussion…
> Regards, Mark.
> P.S. To save you time, the reverse SSH (using normal Unix like systems) works like this:
> Say developer at IP address A wants to connect to module at IP address B.
> On the module: $ ssh -R 19999:localhost:22 <user-A>@<A>
> (And go through authentication for B to ssh to A)
> Then, on the developer’s workstation: $ ssh -p 19999 <user-B>@localhost

More information about the OvmsDev mailing list