<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">My day job side note:</span></div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></span></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">My experience is similar to Steve’s. My day job is looking after thousands of remote linux devices distributed around the world. These devices are implementing security policies, so that is our primary concern. When we started doing this 20 years ago, we connected out from our Security Operation Centres to the remote devices, but every deployment was painful having to deal with upstream firewalls and network address translation issues. 10 years ago we switched to the devices phoning home, and Security Operation Centre staff then tunnelling back to the device through that connection. That approach solved all those issues.</span><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></font></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">Our technology for this is proprietary. In essence it is a SSL connection using both client and server side certificates trusting only a specific private certificate authority. Both ends authenticate each other using those certificates. Once that tunnel is established, a protocol can open/close channels (in either direction) to connect to specific services (either tcp or uds). Rules at each end govern who can connect and/or open channels to what. Essentially a proxy.</span></font></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></font></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">We do use ssh for shell connections, and scp for file copy, but normally only forward. We can tunnel ssh (as it is just another tcp/ip service) through channels in the single SSL connection.</span></font></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></font></div><div class=""><font color="#000000" class="">We don’t use VPNs for this, for a few reason (probably specific to our environment):</font></div><div class=""><font color="#000000" class=""><br class=""></font></div></blockquote><div class=""><ol class="MailOutline"><ol class=""><li class=""><font color="#000000" class="">We don’t want to mess up the customer’s security policy with our own firewall rules.</font></li><li class=""><font color="#000000" class="">We don’t want to interfere with the customer’s own VPN connections (or others on the same device), by mixing in our own.</font></li><li class=""><font color="#000000" class="">We don’t trust anyone, and don’t want to potentially open up the entire network.</font></li><li class=""><font color="#000000" class="">We don’t require access to the LAN, only to the device itself.</font></li><li class=""><font color="#000000" class="">We want to control which Security Operation Centre engineers have access to which devices.</font></li></ol></ol><div><br class=""></div><div>OVMS requirements:</div><div><br class=""></div><div>I’m not saying that a proxy over SSL approach is better than Steve’s outlined reverse SSH approach, just that at my day job we had specific requirements and we did it that way to meet those requirements. Even with 20/20 hindsight, I wouldn’t change it.</div><div><br class=""></div><div>I am curious, @Steve, if you implemented any controls for restriction of engineer access? For example, in this OVMS requirement, say a developer configured his module to SSH tunnel back to the central control server, then he can SSH to that control server to tunnel into his module. But what is stopping other developers also connecting to his module? Presumably everybody is just connecting to 127.0.0.1 on the central controls server?</div><div><br class=""></div><div>I am also concerned about performance. For console access, that shouldn’t matter - but for can bus dumps we need every bps of speed we can get.</div><div><br class=""></div><div>Irrespective of what we do for this, we need proper SSL support in the OVMS module firmware, so perhaps that is the first step? Implement a mechanism for definition of trusted CAs (with some provided in firmware, and others configurable in the code), and then start to offer that. Modification of the OVMS v2 server code to support an SSL connection option would be trivial, as would the Apps. MQTT to mosquito over SSL similarly so. I’ll outline my thoughts on this in a separate eMail.</div><div><br class=""></div><div>Regards, Mark.</div><div><br class=""></div><div><blockquote type="cite" class=""><div class="">On 6 Feb 2020, at 2:38 AM, Greg D. <<a href="mailto:gregd2350@gmail.com" class="">gregd2350@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
Hi Michael,<br class="">
<br class="">
Yes and no, I think... There's still the firewall issue, especially
with the cellular network. I'm also not in favor of opening up my
firewall to incoming connections, even with authentication. It just
invites problems, as everyone has different equipment,
configuration, and expertise. I believe the reverse SSH solves
that.<br class="">
<br class="">
What did I miss?<br class="">
<br class="">
Greg<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Michael Balzer wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:dc79f879-c828-982d-667e-4281569cce28@expeedo.de" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div class="moz-cite-prefix">This illustrates why my preference
would be a standard VPN client. VPNs can be used by normal
users.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 04.02.20 um 00:45 schrieb Stephen Casner:<br class="">
</div>
<blockquote type="cite" cite="mid:alpine.OSX.2.21.9999.2002031403160.29737@auge.attlocal.net" class="">
<pre class="moz-quote-pre" wrap="">Mark,
At the job from which I retired we made heavy use of SSH port
forwarding and reverse ssh clients to allow support access to our
products installed at customer sites. To achieve an acceptable level
of security we dedicated one Linux server at our HQ to be the target
of "phone home" SSH connections from the customer units. This server
exposed only the sshd service and allowed only key-based logins, not
passwords. For customers who agreed to allow this access inside their
networks, the unit would maintain a continuous SSH connection to the
server, automatically reconnecting after an outage.
A Perl script named phonehome on the customer unit established and
maintained the ssh connection using an unlocked private key. The
login shell for the target account on the server was another Perl
script, phonehomed, so if a customer unit were compromised then an
attacker who reached the server would not be able to do anything.
Phonehome sent along some identity information to the server so we
could tell which customer it was, and phonehomed would just send back
"success" to indicate successful connection up to the application
level and then sleep.
The phonehome client script would randomly select a port number and
use that to establish port forwarding from that port number at the
server end to port 22 on the customer unit. With many customer units
calling in, there would occasionallly be a collision for the random
port number, in which case phonehomed would send "collision" and close
the connection so the phonehome script would try again.
Support staff would be allowed access to the server. A script named
et on the server listed all the open connections from customer units
with the randomly chosen port numbers so we'd know how to connect to
the desired customer using ssh to localhost at that port. No SSH
private keys were allowed to be stored on the server; instead, staff
were required to use ssh-agent forwarding to connect from their own
computers through the server to the customer units.
Now, to relate all this back to the problem at hand, perhaps we could
have a server somewhere with a similar setup. This would avoid the
need for multiple developers to open holes in their personal NAT
and/or firewall to allow incoming connections from remote OVMS units.
Rather than having all the OVMS units try to maintain call-back
connections to that server all the time, the existing ovms v2 protocol
shell provides sufficient access to initiate a call back when one is
needed.
A prerequisite for developing such a service would be for me to get
off my duff and update the SSH implementation in OVMS to use the
current WolfSSH release. They took back the extensions I made to
their code to support SCP, but they made some changes that I would
need to adapt our code to fit. They added SFTP support, too, but I
have not studied it to see if it can fit into task structure.
-- Steve
On Mon, 3 Feb 2020, Mark Webb-Johnson wrote:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">Seeking opinions (in particular Stephen, our ssh guru).
As I (and others) are doing more work remotely (OVMS modules in cars, and trying to connect in remotely to diagnose, test, or develop), I am seeing the same issue come up often. If the car is on your own home WiFi, it is simple and easy. But if the car is elsewhere, on someone else’s WiFi, then connecting _to_ the car becomes hard. Firewalls, port forwarding, and all that mess. Security also becomes an issue (once ports are opened and forwarded).
The core requirements seem to be:
A way to get a proper shell. The ovms v2 protocol shell is ok, but quite painful to work with (especially as logs are not shown and there is no command expansion).
A way to get CAN dumps out of the car. Can save to sd card, but can’t copy in/out.
Both of these are simple TCP connections.
The second issue (CAN dumps) is resolvable if the developer has a public address. Open the firewall, ’nc -l’ a port, and ‘can log start tcpclient …’ on the module. But a little kludgy, and doesn’t support SavvyCAN (which can only connect _to_ the module, not receive a connection _from_ the module).
I can see three possible approaches to this:
Extension of OVMS v2 protocol
We could extend the OVMS v2 protocol to support a PIPE mode. Have it allow connections and data to be transferred over the v2 protocol link using simple messages. Then, the pipe could be used to tunnel SSH and CAN log connections. The same could be done with v3 over MQTT (the transport is different, but the console / port forwarding bits inside the ovms module are the same).
The advantage here is that the OVMS v2 server becomes a man-in-the-middle broker. Both the car module and the developer connect to the OVMS v2 server to exchange data. No incoming connections, so everything can be dynamic.
The disadvantage is security is fairly weak on that protocol. Very dated RC4 encryption and poor authentication.
Leverage SSH
SSH has a port forwarding facility built in, that wolfssh seems to support. It is also possible to do a reverse ssh client.
The advantage here is security (very strong encryption and authentication).
The disadvantage is complexity and the fact that the developer end needs to poke a hole in their firewall (and possibly do NAT port forwarding, etc) to allow the incoming connections from the car.
STUNNEL / WireGuard / Mosh / VPN / etc
Perhaps someone else knows of something suitable? The issues here are that the LWIP stack is not the easiest to work with.
My initial preference is #1 or #2, but open to discussion…
Regards, Mark.
P.S. To save you time, the reverse SSH (using normal Unix like systems) works like this:
Say developer at IP address A wants to connect to module at IP address B.
On the module: $ ssh -R 19999:localhost:22 <user-A>@<A>
(And go through authentication for B to ssh to A)
Then, on the developer’s workstation: $ ssh -p 19999 <user-B>@localhost</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
</blockquote>
<br class="">
<br class="">
<pre class="moz-signature" cols="144">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre wrap="" class="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></div></blockquote></div><br class=""></div></body></html>