<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Jakob,<div class=""><br class=""></div><div class=""><blockquote type="cite" class="">I suppose the following topic names:<br class="">metrics: <prefix>/metric/#<br class="">events: <prefix>/event/#<br class="">notifications: <prefix>/notify/#<br class="">config: <prefix>/config/#<br class="">logs: <prefix>/log/<tag><br class="">active: <prefix>/client/<clientid>/active<br class="">requests: <prefix>/client/<clientid>/request/#<br class="">commands: <prefix>/client/<clientid>/command/<command id><br class="">cmd responses: <prefix>/client/<clientid>/response/<command id></blockquote><br class=""></div><div class="">All ok. I am fine with this, and looks clean. I will make the changes today, as I want to get this into some cars asap so we can get a feel for how it behaves.</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class="">Filtering:<br class="">In order to decrease used data a blacklist of metrics which are not<br class="">send (like m.freeram or m.monotonic) would be cool.<br class=""></blockquote><div class=""><br class=""></div>Yep. Makes sense.</div><div class=""><br class=""><blockquote type="cite" class="">heartbeats:<br class="">I've seen you implemented that clients have to send a message every two<br class="">minutes. A topic like the above .../<clientid>/active which is set to 1<br class="">when a client connects and to 0 by its last will would have the same<br class="">effect but not require heartbeats to be sent every two minutes and thus<br class="">use less of our precious cellular data.<br class=""></blockquote><div class=""><br class=""></div>The problem is if the client does not set the last will correctly, or server/library doesn’t handle it. We’d get stuck thinking a client is connected and use 10 times the data forever.</div><div class=""><br class=""></div><div class="">There is also the related issue of a locked-up client (not disconnected). The heartbeat lets us know that the client is both connected and working.</div><div class=""><br class=""></div><div class="">Compared to the amount of data coming from the module to client, one single message once a minute client->module, and only when the client App is actually running and connected, is minimal.</div><div class=""><br class=""></div><div class="">Any other way to handle the misbehaving client issue?</div><div class=""><br class=""><blockquote type="cite" class="">Requesting values:<br class="">In order to decrease data usage even more allowing clients to request<br class="">the value of a metric using <prefix>/client/<clientid>/request/metric<br class="">and the metric name in the value or using<br class=""><prefix>/client/<clientid>/request/config and the config name in the<br class="">value. Allowing wildcards like <a href="http://m.net" class="">m.net</a>.* or just * allows to request<br class="">multiple or all metrics with a single request. Then e.g. the app would<br class="">request all metrics it wants to display.<br class=""></blockquote><div class=""><br class=""></div>ok. Let’s deal with that later. Rather than the traditional broadcast mechanism, there is also the opportunity to have a request-broadcast system where the client tells the module what metrics it is interested in. I am just not sure how that fits into our framework.</div><div class=""><br class=""><blockquote type="cite" class="">Encryption:<br class="">TLS in general seems to be a missing thing with OVMS. Downloading<br class="">firmware updates over http and not doing any verification is kind of<br class="">meh. The problem is (as noted in my last email) with the CA certs. My<br class="">suggestion is to ship with one default CA (e.g. Let's Encrypt) and<br class="">allow the user to replace it with a different one.<br class="">I would love to implement this, but I am currently in exam phase and<br class="">won't have much time to spare until next week.<br class=""></blockquote><div class=""><br class=""></div><div class="">I think we can have a list of trusted CAs. Some can be provided as standard in the firmware (for example, <a href="http://api.openvehicles.com" class="">api.openvehicles.com</a> currently uses '/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root’ so we should have that in there). Others can be provided in the config (or /store file system). We build up our list, to be used by TLS library, by combining the two sources.</div><br class=""><blockquote type="cite" class="">Authentification:<br class="">If you can live with moving to mosquitto[0] (MQTT broker by eclipse)<br class="">they have a very good auth plugin[1]. All you have to do is to write an<br class="">sql statement which receives the hashed passwords from the OVMS<br class="">database by username (see [2]). If you are on debian its just an apt<br class="">install mosquitto mosquitto-auth-plugin away.<br class="">It is very important to set up ACL (access control lists) which make<br class="">sure only user X can subscribe/publish to ovms/X and noone else.<br class="">Luckily this is also handeled by mosquitto_auth_plug<br class=""><br class="">[0] <a href="https://mosquitto.org/" class="">https://mosquitto.org/</a><br class="">[1] <a href="https://github.com/jpmens/mosquitto-auth-plug" class="">https://github.com/jpmens/mosquitto-auth-plug</a><br class="">[2] <a href="https://github.com/jpmens/mosquitto-auth-plug#postgresql" class="">https://github.com/jpmens/mosquitto-auth-plug#postgresql</a></blockquote><div class=""><br class=""></div>I am currently using mosquitto, but honestly think it severely lacking with regard to plugins (and authentication specifically). While jpmens has written a neat library, there is still no simple way to just check a user+password for authentication. The closest is the HTTP API, but that seems kludgy. I also don’t understand why this is not packaged with mosquitto by default - on centos we have to do a nasty custom build-from-source-code.</div><div class=""><br class=""></div><div class="">The problem with jpmens, and why it won’t work for us, is:</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><span style="caret-color: rgb(36, 41, 46); color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; background-color: rgb(255, 255, 255);" class="">The SQL query for looking up a user's password hash is mandatory. The query MUST return a single row only (any other number of rows is considered to be "user not found"), and it MUST return a single column only with the PBKDF2 password hash.</span></div></blockquote><div class=""><br class=""></div><div class="">Drupal passwords, while stored in a mysql database, are not in PBKDF2 format. The jpmens approach is fine if you want to have a custom database table just for this application, but if you are trying to authenticate against some other application using a different password hash algorithm.</div><div class=""><br class=""></div><div class="">By comparison, vernemq includes all these authentication plugins as standard code as part of the main distribution. It also includes a simple system to write hook scripts (including authentication if you want) in Lua (or the base erlang if you are desperate).</div><div class=""><br class=""></div><div class="">I’ve been struggling so hard with mosquitto for this, that I’m about ready to give up and just extend the drupal openvehicles extension to allow a username+password to be checked via API. But it is way too much work and way too kludgy for something that really should be simple and commonplace.</div><div class=""><br class=""></div><div class="">Regards, Mark.</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 5 Jul 2018, at 5:41 PM, Jakob Löw <<a href="mailto:ovms@m4gnus.de" class="">ovms@m4gnus.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hey,<br class=""><br class="">Regarding naming:<br class="">I suppose the following topic names:<br class="">metrics: <prefix>/metric/#<br class="">events: <prefix>/event/#<br class="">notifications: <prefix>/notify/#<br class="">config: <prefix>/config/#<br class="">logs: <prefix>/log/<tag><br class="">active: <prefix>/client/<clientid>/active<br class="">requests: <prefix>/client/<clientid>/request/#<br class="">commands: <prefix>/client/<clientid>/command/<command id><br class="">cmd responses: <prefix>/client/<clientid>/response/<command id><br class=""><br class="">Filtering:<br class="">In order to decrease used data a blacklist of metrics which are not<br class="">send (like m.freeram or m.monotonic) would be cool.<br class=""><br class="">heartbeats:<br class="">I've seen you implemented that clients have to send a message every two<br class="">minutes. A topic like the above .../<clientid>/active which is set to 1<br class="">when a client connects and to 0 by its last will would have the same<br class="">effect but not require heartbeats to be sent every two minutes and thus<br class="">use less of our precious cellular data.<br class=""><br class="">Requesting values:<br class="">In order to decrease data usage even more allowing clients to request<br class="">the value of a metric using <prefix>/client/<clientid>/request/metric<br class="">and the metric name in the value or using<br class=""><prefix>/client/<clientid>/request/config and the config name in the<br class="">value. Allowing wildcards like <a href="http://m.net" class="">m.net</a>.* or just * allows to request<br class="">multiple or all metrics with a single request. Then e.g. the app would<br class="">request all metrics it wants to display.<br class=""><br class="">Encryption:<br class="">TLS in general seems to be a missing thing with OVMS. Downloading<br class="">firmware updates over http and not doing any verification is kind of<br class="">meh. The problem is (as noted in my last email) with the CA certs. My<br class="">suggestion is to ship with one default CA (e.g. Let's Encrypt) and<br class="">allow the user to replace it with a different one.<br class="">I would love to implement this, but I am currently in exam phase and<br class="">won't have much time to spare until next week.<br class=""><br class="">Authentification:<br class="">If you can live with moving to mosquitto[0] (MQTT broker by eclipse)<br class="">they have a very good auth plugin[1]. All you have to do is to write an<br class="">sql statement which receives the hashed passwords from the OVMS<br class="">database by username (see [2]). If you are on debian its just an apt<br class="">install mosquitto mosquitto-auth-plugin away.<br class="">It is very important to set up ACL (access control lists) which make<br class="">sure only user X can subscribe/publish to ovms/X and noone else.<br class="">Luckily this is also handeled by mosquitto_auth_plug<br class=""><br class="">[0] <a href="https://mosquitto.org/" class="">https://mosquitto.org/</a><br class="">[1] <a href="https://github.com/jpmens/mosquitto-auth-plug" class="">https://github.com/jpmens/mosquitto-auth-plug</a><br class="">[2] <a href="https://github.com/jpmens/mosquitto-auth-plug#postgresql" class="">https://github.com/jpmens/mosquitto-auth-plug#postgresql</a><br class=""><br class="">On Thu, 2018-07-05 at 09:26 +0800, Mark Webb-Johnson wrote:<br class=""><blockquote type="cite" class="">I am far from an expert in MQTT. Not even a novice. So, the work<br class="">below is ‘best efforts’. Any help / comments / suggestions would be<br class="">most appreciated. In particular, a big thanks to Jakob for his<br class="">contributions to this so far.<br class=""><br class="">With yesterday’s merge, and commits, we have a very very basic OVMS<br class="">server v3 implementation. It sends the metrics, and doesn’t crash<br class="">(much). The overall design is as follows:<br class=""><br class="">We use the mongoose mqtt library. We don’t do anything special, and<br class="">everything is following the mqtt 3.1 standard.<br class=""><br class="">MQTT has the concept of topics. Our default prefix for everything is:<br class=""><br class="">ovms/<mqtt-username>/<vehicleid>/<br class=""><br class="">(that can be customised with config server.v3 topic.prefix)<br class=""><br class="">Metrics are sent as topics. The metric name is added to the topic<br class="">prefix + “m/“ suffix, and “.” characters converted to “/“ to match<br class="">the MQTT conventions. The value is simply the metric value (as a<br class="">string). With our current arrangement, this adds m/m/, m/s/, and m/v/<br class="">sub-trees to the the MQTT topic hierarchy. Clients can subscribe to<br class=""><prefix>/m/# to receive all metrics. The ‘retained’ flag is set on<br class="">these metrics, at QOS level 0 (so subscribing clients will get a copy<br class="">of the last known values for these metrics, even with a disconnected<br class="">vehicle).<br class=""><br class="">The metric s.v3.connected is maintained by the ServerV3 code. When a<br class="">successful MQTT connection is made, and login successful, that is set<br class="">true (yes). A MQTT last-will-testament is set so that if the OVMS<br class="">module disconnects the server will automatically update that to false<br class="">(no). The ‘retained’ flag is set on this metric, at QOS level 0 (so<br class="">subscribing clients will get a copy of the last known state). Clients<br class="">can use this to see if the vehicle is connected to the server.<br class=""><br class="">Connecting clients are expected to write a “1” value to the<br class=""><prefix>/c/<clientid> topic, and to repeat that write once a minute.<br class="">They are also expected to use a last-will-testament on that same<br class="">topic with value “0”. QOS 1 should be used, and these topics should<br class="">not be retained. The server V3 code subscribes to this <prefix>/c/#<br class="">topic, so it gets informed of all connected clients. It can then<br class="">update the s.v3.peers metric appropriately. Clients are expected to<br class="">monitor the <prefix>/s/v3/connected topic, so that if it becomes<br class="">‘yes’ (true) the client should send <prefix>/c/<clientid> “1”<br class="">immediately. This mechanism allows a newly connected vehicle to<br class="">immediately know if one or more clients is connected.<br class=""><br class="">The Server v3 sets a timeout for <prefix>/c/<clientid> connections of<br class="">2 minutes. If that mqtt topic is not sent again within that time, it<br class="">is expired (and that client is treated as disconnected).<br class=""><br class="">Similar to the v2 code, the server v3 transmits modified metrics once<br class="">a minute if there are one or more clients connected, or once every<br class="">ten minutes if there are no clients connected.<br class=""><br class="">All the above has been implemented. To reach parity with v2, and<br class="">exceed it’s functionality in places, we have a few more things to do:<br class=""><br class="">Notifications<br class=""><br class="">On the vehicle side, I am proposing to use the <prefix>/notify/<type><br class="">namespace for these, using QOS 2 messages without retention. Clients<br class="">can listen to those, if necessary. We can also have a central daemon<br class="">running that listens to ovms/+/+/n/# topic pattern to receive these<br class="">notifications and handle appropriately. Using QOS 2 we can confirm<br class="">the reception of the notification / historical data, and mark it as<br class="">delivered, appropriately. However, that would only confirm delivery<br class="">to the MQTT server, not to the central daemon; if the daemon was not<br class="">running, the notification would be lost.<br class=""><br class="">Textual Commands<br class=""><br class="">I am proposing to use the <prefix>/cmd/<clientid>/c/ and<br class=""><prefix>/cmd/<clientid>/r/ namespaces for this, using QOS 2 messages<br class="">without retention. The value in the /c/ would be the command, and the<br class="">response would be written to matching the /r/ topic. The commands and<br class="">responses could be prefixed by an identifier (like in imap protocol)<br class="">so responses can be matched to commands by the clients). The client<br class="">side can simply subscribe to itself, and the vehicle side subscribes<br class="">to <prefix>/cmd/#. In this way, commands cannot be duplicated, and<br class="">clients don’t see responses to commands they didn’t initiate (which<br class="">was an issue with v2).<br class=""><br class="">Numeric Commands<br class=""><br class="">I am not sure if we need to implement the numeric commands, as used<br class="">in the v2 protocol. It seems to me that we can use textual commands.<br class=""><br class="">Config Access<br class=""><br class="">I am not sure if we need this, beyond the command processor. If we<br class="">do, we could expose a /config/ namespace.<br class=""><br class="">Events<br class=""><br class="">It would be nice to expose events (except for the ticker.* ones, of<br class="">course). This could be done by a <prefix>/events topic, using QOS 2<br class="">and no retention.<br class=""><br class="">Logs<br class=""><br class="">It would be nice to expose logs. This could be done by a<br class=""><prefix>/logs topic, using QOS 1 and no retention.<br class=""><br class="">Security<br class=""><br class="">We need to add SSL support. I am trying to get an authentication<br class="">plugin for mosquitto / vernemq written so that we can authenticate<br class="">straight from the OVMS database that is already running on the<br class="">servers, and give each user their own ovms/<userid>/# namespace. That<br class="">way, the configuration for v3 on the vehicle/apps/clients is simple -<br class="">just put in the server username and password (no separate vehicle<br class="">passwords necessary).<br class=""><br class="">I think that is it. The above would form the basis of the<br class="">specification for this. As this is the basis for future work and<br class="">direction in OVMS, it is important that we get it right, so all<br class="">comments / suggestions most welcome.<br class=""><br class="">Regards, Mark.<br class=""><br class="">_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</blockquote>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></div></div></blockquote></div><br class=""></div></body></html>