[Ovmsdev] MQTT and Ovms Server v3

[Jakob Löw]

Hey,

Regarding naming:
I suppose the following topic names:
metrics:       <prefix>/metric/#
events:        <prefix>/event/#
notifications: <prefix>/notify/#
config:        <prefix>/config/#
logs:          <prefix>/log/<tag>
active:        <prefix>/client/<clientid>/active
requests:      <prefix>/client/<clientid>/request/#
commands:      <prefix>/client/<clientid>/command/<command id>
cmd responses: <prefix>/client/<clientid>/response/<command id>

Filtering:
In order to decrease used data a blacklist of metrics which are not
send (like m.freeram or m.monotonic) would be cool.

heartbeats:
I've seen you implemented that clients have to send a message every two
minutes. A topic like the above .../<clientid>/active which is set to 1
when a client connects and to 0 by its last will would have the same
effect but not require heartbeats to be sent every two minutes and thus
use less of our precious cellular data.

Requesting values:
In order to decrease data usage even more allowing clients to request
the value of a metric using <prefix>/client/<clientid>/request/metric
and the metric name in the value or using
<prefix>/client/<clientid>/request/config and the config name in the
value. Allowing wildcards like m.net.* or just * allows to request
multiple or all metrics with a single request. Then e.g. the app would
request all metrics it wants to display.

Encryption:
TLS in general seems to be a missing thing with OVMS. Downloading
firmware updates over http and not doing any verification is kind of
meh. The problem is (as noted in my last email) with the CA certs. My
suggestion is to ship with one default CA (e.g. Let's Encrypt) and
allow the user to replace it with a different one.
I would love to implement this, but I am currently in exam phase and
won't have much time to spare until next week.

Authentification:
If you can live with moving to mosquitto[0] (MQTT broker by eclipse)
they have a very good auth plugin[1]. All you have to do is to write an
sql statement which receives the hashed passwords from the OVMS
database by username (see [2]). If you are on debian its just an apt
install mosquitto mosquitto-auth-plugin away.
It is very important to set up ACL (access control lists) which make
sure only user X can subscribe/publish to ovms/X and noone else.
Luckily this is also handeled by mosquitto_auth_plug

[0] https://mosquitto.org/
[1] https://github.com/jpmens/mosquitto-auth-plug
[2] https://github.com/jpmens/mosquitto-auth-plug#postgresql

On Thu, 2018-07-05 at 09:26 +0800, Mark Webb-Johnson wrote:
> I am far from an expert in MQTT. Not even a novice. So, the work
> below is ‘best efforts’. Any help / comments / suggestions would be
> most appreciated. In particular, a big thanks to Jakob for his
> contributions to this so far.
> 
> With yesterday’s merge, and commits, we have a very very basic OVMS
> server v3 implementation. It sends the metrics, and doesn’t crash
> (much). The overall design is as follows:
> 
> We use the mongoose mqtt library. We don’t do anything special, and
> everything is following the mqtt 3.1 standard.
> 
> MQTT has the concept of topics. Our default prefix for everything is:
> 
> ovms/<mqtt-username>/<vehicleid>/
> 
> (that can be customised with config server.v3 topic.prefix)
> 
> Metrics are sent as topics. The metric name is added to the topic
> prefix + “m/“ suffix, and “.” characters converted to “/“ to match
> the MQTT conventions. The value is simply the metric value (as a
> string). With our current arrangement, this adds m/m/, m/s/, and m/v/
> sub-trees to the the MQTT topic hierarchy. Clients can subscribe to
> <prefix>/m/# to receive all metrics. The ‘retained’ flag is set on
> these metrics, at QOS level 0 (so subscribing clients will get a copy
> of the last known values for these metrics, even with a disconnected
> vehicle).
> 
> The metric s.v3.connected is maintained by the ServerV3 code. When a
> successful MQTT connection is made, and login successful, that is set
> true (yes). A MQTT last-will-testament is set so that if the OVMS
> module disconnects the server will automatically update that to false
> (no). The ‘retained’ flag is set on this metric, at QOS level 0 (so
> subscribing clients will get a copy of the last known state). Clients
> can use this to see if the vehicle is connected to the server.
> 
> Connecting clients are expected to write a “1” value to the
> <prefix>/c/<clientid> topic, and to repeat that write once a minute.
> They are also expected to use a last-will-testament on that same
> topic with value “0”. QOS 1 should be used, and these topics should
> not be retained. The server V3 code subscribes to this <prefix>/c/#
> topic, so it gets informed of all connected clients. It can then
> update the s.v3.peers metric appropriately. Clients are expected to
> monitor the <prefix>/s/v3/connected topic, so that if it becomes
> ‘yes’ (true) the client should send <prefix>/c/<clientid> “1”
> immediately. This mechanism allows a newly connected vehicle to
> immediately know if one or more clients is connected.
> 
> The Server v3 sets a timeout for <prefix>/c/<clientid> connections of
> 2 minutes. If that mqtt topic is not sent again within that time, it
> is expired (and that client is treated as disconnected).
> 
> Similar to the v2 code, the server v3 transmits modified metrics once
> a minute if there are one or more clients connected, or once every
> ten minutes if there are no clients connected.
> 
> All the above has been implemented. To reach parity with v2, and
> exceed it’s functionality in places, we have a few more things to do:
> 
> Notifications
> 
> On the vehicle side, I am proposing to use the <prefix>/notify/<type>
> namespace for these, using QOS 2 messages without retention. Clients
> can listen to those, if necessary. We can also have a central daemon
> running that listens to ovms/+/+/n/# topic pattern to receive these
> notifications and handle appropriately. Using QOS 2 we can confirm
> the reception of the notification / historical data, and mark it as
> delivered, appropriately. However, that would only confirm delivery
> to the MQTT server, not to the central daemon; if the daemon was not
> running, the notification would be lost.
> 
> Textual Commands
> 
> I am proposing to use the <prefix>/cmd/<clientid>/c/ and
> <prefix>/cmd/<clientid>/r/ namespaces for this, using QOS 2 messages
> without retention. The value in the /c/ would be the command, and the
> response would be written to matching the /r/ topic. The commands and
> responses could be prefixed by an identifier (like in imap protocol)
> so responses can be matched to commands by the clients). The client
> side can simply subscribe to itself, and the vehicle side subscribes
> to <prefix>/cmd/#. In this way, commands cannot be duplicated, and
> clients don’t see responses to commands they didn’t initiate (which
> was an issue with v2).
> 
> Numeric Commands
> 
> I am not sure if we need to implement the numeric commands, as used
> in the v2 protocol. It seems to me that we can use textual commands.
> 
> Config Access
> 
> I am not sure if we need this, beyond the command processor. If we
> do, we could expose a /config/ namespace.
> 
> Events
> 
> It would be nice to expose events (except for the ticker.* ones, of
> course). This could be done by a <prefix>/events topic, using QOS 2
> and no retention.
> 
> Logs
> 
> It would be nice to expose logs. This could be done by a
> <prefix>/logs topic, using QOS 1 and no retention.
> 
> Security
> 
> We need to add SSL support. I am trying to get an authentication
> plugin for mosquitto / vernemq written so that we can authenticate
> straight from the OVMS database that is already running on the
> servers, and give each user their own ovms/<userid>/# namespace. That
> way, the configuration for v3 on the vehicle/apps/clients is simple -
> just put in the server username and password (no separate vehicle
> passwords necessary).
> 
> I think that is it. The above would form the basis of the
> specification for this. As this is the basis for future work and
> direction in OVMS, it is important that we get it right, so all
> comments / suggestions most welcome.
> 
> Regards, Mark.
> 
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20180705/41b98b2c/attachment-0001.sig>