[Ovmsdev] Reverse connections (module -> developer)

Mon Feb 10 11:20:21 HKT 2020

My day job side note:

My experience is similar to Steve’s. My day job is looking after thousands of remote linux devices distributed around the world. These devices are implementing security policies, so that is our primary concern. When we started doing this 20 years ago, we connected out from our Security Operation Centres to the remote devices, but every deployment was painful having to deal with upstream firewalls and network address translation issues. 10 years ago we switched to the devices phoning home, and Security Operation Centre staff then tunnelling back to the device through that connection. That approach solved all those issues.

Our technology for this is proprietary. In essence it is a SSL connection using both client and server side certificates trusting only a specific private certificate authority. Both ends authenticate each other using those certificates. Once that tunnel is established, a protocol can open/close channels (in either direction) to connect to specific services (either tcp or uds). Rules at each end govern who can connect and/or open channels to what. Essentially a proxy.

We do use ssh for shell connections, and scp for file copy, but normally only forward. We can tunnel ssh (as it is just another tcp/ip service) through channels in the single SSL connection.

We don’t use VPNs for this, for a few reason (probably specific to our environment):

We don’t want to mess up the customer’s security policy with our own firewall rules.
We don’t want to interfere with the customer’s own VPN connections (or others on the same device), by mixing in our own.
We don’t trust anyone, and don’t want to potentially open up the entire network.
We don’t require access to the LAN, only to the device itself.
We want to control which Security Operation Centre engineers have access to which devices.

OVMS requirements:

I’m not saying that a proxy over SSL approach is better than Steve’s outlined reverse SSH approach, just that at my day job we had specific requirements and we did it that way to meet those requirements. Even with 20/20 hindsight, I wouldn’t change it.

I am curious, @Steve, if you implemented any controls for restriction of engineer access? For example, in this OVMS requirement, say a developer configured his module to SSH tunnel back to the central control server, then he can SSH to that control server to tunnel into his module. But what is stopping other developers also connecting to his module? Presumably everybody is just connecting to 127.0.0.1 on the central controls server?

I am also concerned about performance. For console access, that shouldn’t matter - but for can bus dumps we need every bps of speed we can get.

Irrespective of what we do for this, we need proper SSL support in the OVMS module firmware, so perhaps that is the first step? Implement a mechanism for definition of trusted CAs (with some provided in firmware, and others configurable in the code), and then start to offer that. Modification of the OVMS v2 server code to support an SSL connection option would be trivial, as would the Apps. MQTT to mosquito over SSL similarly so. I’ll outline my thoughts on this in a separate eMail.

Regards, Mark.

> On 6 Feb 2020, at 2:38 AM, Greg D. <gregd2350 at gmail.com> wrote:
> 
> Hi Michael,
> 
> Yes and no, I think...  There's still the firewall issue, especially with the cellular network.  I'm also not in favor of opening up my firewall to incoming connections, even with authentication.  It just invites problems, as everyone has different equipment, configuration, and expertise.  I believe the reverse SSH solves that.
> 
> What did I miss?
> 
> Greg
> 
> 
> Michael Balzer wrote:
>> This illustrates why my preference would be a standard VPN client. VPNs can be used by normal users.
>> 
>> Regards,
>> Michael
>> 
>> 
>> Am 04.02.20 um 00:45 schrieb Stephen Casner:
>>> Mark,
>>> 
>>> At the job from which I retired we made heavy use of SSH port
>>> forwarding and reverse ssh clients to allow support access to our
>>> products installed at customer sites.  To achieve an acceptable level
>>> of security we dedicated one Linux server at our HQ to be the target
>>> of "phone home" SSH connections from the customer units.  This server
>>> exposed only the sshd service and allowed only key-based logins, not
>>> passwords.  For customers who agreed to allow this access inside their
>>> networks, the unit would maintain a continuous SSH connection to the
>>> server, automatically reconnecting after an outage.
>>> 
>>> A Perl script named phonehome on the customer unit established and
>>> maintained the ssh connection using an unlocked private key.  The
>>> login shell for the target account on the server was another Perl
>>> script, phonehomed, so if a customer unit were compromised then an
>>> attacker who reached the server would not be able to do anything.
>>> Phonehome sent along some identity information to the server so we
>>> could tell which customer it was, and phonehomed would just send back
>>> "success" to indicate successful connection up to the application
>>> level and then sleep.
>>> 
>>> The phonehome client script would randomly select a port number and
>>> use that to establish port forwarding from that port number at the
>>> server end to port 22 on the customer unit.  With many customer units
>>> calling in, there would occasionallly be a collision for the random
>>> port number, in which case phonehomed would send "collision" and close
>>> the connection so the phonehome script would try again.
>>> 
>>> Support staff would be allowed access to the server.  A script named
>>> et on the server listed all the open connections from customer units
>>> with the randomly chosen port numbers so we'd know how to connect to
>>> the desired customer using ssh to localhost at that port.  No SSH
>>> private keys were allowed to be stored on the server; instead, staff
>>> were required to use ssh-agent forwarding to connect from their own
>>> computers through the server to the customer units.
>>> 
>>> Now, to relate all this back to the problem at hand, perhaps we could
>>> have a server somewhere with a similar setup.  This would avoid the
>>> need for multiple developers to open holes in their personal NAT
>>> and/or firewall to allow incoming connections from remote OVMS units.
>>> Rather than having all the OVMS units try to maintain call-back
>>> connections to that server all the time, the existing ovms v2 protocol
>>> shell provides sufficient access to initiate a call back when one is
>>> needed.
>>> 
>>> A prerequisite for developing such a service would be for me to get
>>> off my duff and update the SSH implementation in OVMS to use the
>>> current WolfSSH release.  They took back the extensions I made to
>>> their code to support SCP, but they made some changes that I would
>>> need to adapt our code to fit.  They added SFTP support, too, but I
>>> have not studied it to see if it can fit into task structure.
>>> 
>>>                                                         -- Steve
>>> 
>>> On Mon, 3 Feb 2020, Mark Webb-Johnson wrote:
>>> 
>>>> Seeking opinions (in particular Stephen, our ssh guru).
>>>> 
>>>> As I (and others) are doing more work remotely (OVMS modules in cars, and trying to connect in remotely to diagnose, test, or develop), I am seeing the same issue come up often. If the car is on your own home WiFi, it is simple and easy. But if the car is elsewhere, on someone else’s WiFi, then connecting _to_ the car becomes hard. Firewalls, port forwarding, and all that mess. Security also becomes an issue (once ports are opened and forwarded).
>>>> 
>>>> The core requirements seem to be:
>>>> 
>>>> A way to get a proper shell. The ovms v2 protocol shell is ok, but quite painful to work with (especially as logs are not shown and there is no command expansion).
>>>> 
>>>> A way to get CAN dumps out of the car. Can save to sd card, but can’t copy in/out.
>>>> 
>>>> Both of these are simple TCP connections.
>>>> 
>>>> The second issue (CAN dumps) is resolvable if the developer has a public address. Open the firewall, ’nc -l’ a port, and ‘can log start tcpclient …’ on the module. But a little kludgy, and doesn’t support SavvyCAN (which can only connect _to_ the module, not receive a connection _from_ the module).
>>>> 
>>>> I can see three possible approaches to this:
>>>> 
>>>> Extension of OVMS v2 protocol
>>>> 
>>>> We could extend the OVMS v2 protocol to support a PIPE mode. Have it allow connections and data to be transferred over the v2 protocol link using simple messages. Then, the pipe could be used to tunnel SSH and CAN log connections. The same could be done with v3 over MQTT (the transport is different, but the console / port forwarding bits inside the ovms module are the same).
>>>> 
>>>> The advantage here is that the OVMS v2 server becomes a man-in-the-middle broker. Both the car module and the developer connect to the OVMS v2 server to exchange data. No incoming connections, so everything can be dynamic.
>>>> 
>>>> The disadvantage is security is fairly weak on that protocol. Very dated RC4 encryption and poor authentication.
>>>> 
>>>> Leverage SSH
>>>> 
>>>> SSH has a port forwarding facility built in, that wolfssh seems to support. It is also possible to do a reverse ssh client.
>>>> 
>>>> The advantage here is security (very strong encryption and authentication).
>>>> 
>>>> The disadvantage is complexity and the fact that the developer end needs to poke a hole in their firewall (and possibly do NAT port forwarding, etc) to allow the incoming connections from the car.
>>>> 
>>>> STUNNEL / WireGuard / Mosh / VPN / etc
>>>> 
>>>> Perhaps someone else knows of something suitable? The issues here are that the LWIP stack is not the easiest to work with.
>>>> 
>>>> My initial preference is #1 or #2, but open to discussion…
>>>> 
>>>> Regards, Mark.
>>>> 
>>>> P.S. To save you time, the reverse SSH (using normal Unix like systems) works like this:
>>>> 
>>>> Say developer at IP address A wants to connect to module at IP address B.
>>>> 
>>>> On the module: $ ssh -R 19999:localhost:22 <user-A>@<A>
>>>> (And go through authentication for B to ssh to A)
>>>> 
>>>> Then, on the developer’s workstation: $ ssh -p 19999 <user-B>@localhost
>>>> 
>>>> 
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>> 
>> 
>> -- 
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>> 
>> 
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
> 
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20200210/f0ef9139/attachment.htm>