[Ovmsdev] General trip & grid log proposal

Mon Jan 18 10:38:04 HKT 2021

Regarding MQTT, I definitely think it is the correct direction for the future, and hope it can replace v2 protocol in the long run. The advantages are many, and the only disadvantage seems to be data size on the wire.

To address that, I have been waiting for v5 MQTT to become more widespread. As well as some other desirable features, that adds topic aliases. This means we can replace the topic name with a 4 byte integer. Along with the protocol overhead that is still slightly more than the single byte comma field separator of v2 protocol, but has the advantage that a single changed metric can be transmitted without having to transmit the entire v2 protocol message.

Implementation is relatively straightforward. The first time a particular message is sent, the full topic name and value are used, along with a 4 byte integer topic alias. Subsequent updates (on the same connection session) can leave the full topic name blank and just provide the 4 byte integer topic alias (with the expansion done at the server side). There is also a bunch of stuff in v5 regarding session resumption, to further reduce the overhead.

There is now a development version of mosquito that supports v5, and hivemq has become sort of open source. Our current networking library doesn’t support it, however. Perhaps now is the time to re-visit this (in particular if others are willing to work on it)?

Regards, Mark.

> On 16 Jan 2021, at 3:50 PM, Michael Balzer <dexter at expeedo.de> wrote:
> 
> Signed PGP part
> Steve,
> 
> forgot to mention: the V3 server already supports "data" notifications. These are published under the topic scheme "<prefix>/notify/data/<subtype>/<id>/<timestamp>".
> 
> So you can already send compact custom records. V2 "history" records are sent that way, see OvmsVehicleRenaultTwizy::SendTripLog() for an example.
> 
> Regards,
> Michael
> 
> 
> Am 15.01.21 um 13:55 schrieb Michael Balzer:
>> Steve,
>> 
>> Am 15.01.21 um 10:04 schrieb Steve Davies:
>>> This seems more manageable than the continuous streaming, a sensible compromise.
>> 
>> As I said, the streaming mode wasn't meant to stream all metrics. The V3 server lacks a means to control metrics update rates. Feel free to add one, also take into account what has been discussed on this before, i.e. https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/344 <https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/344>
>> 
>>> But should I be working with MQTT?  Right now Mark and Michael you are by far the biggest contributors so I'd definitely take a steer from you about this.  I don't think it helps the project to have two different approaches "competing" in the same code base.
>> 
>> It's not a competition, it's supporting two different approaches. We even support a third approach in form of our WebSocket protocol, which can be used for direct local connections to the module.
>> 
>> V2 / MP is a custom protocol and more of a legacy support. But it has undeniable advantages over the current MQTT implementation.
>> 
>>> You say:
>>> 
>>>   > The V3 server needs further development from actual users like you. I don't use MQTT because of the protocol overhead. So go ahead, refine it.
>>> 
>>> So the blunt question is whether this MQTT / V3 approach is dying on the vine or if it is worth investing time into.
>>> 
>>> Looks like Mark wrote most of it 3 years ago.
>>> 
>>> For me I'm used to MQTT, I like it and use it in my house control systems, energy monitoring, OpenEVSE etc.  It's exactly designed for the IOT world.  NodeRed, Homeassistant and other tools support it well.
>>> 
>>> But OVMS is already invested in the V2 protocol and you are still developing it.
>>> 
>>> As implemented, the V3/MQTT approach will definitely have more overhead - and indeed sending every metric in a separate message will use a lot more data.
>>> 
>>> But nothing says that we can't have a "compact" format similar to the data messages that V2 sends.    MQTT adds just a few bytes per published message for the MQTT protocol so that would end up being nearly as efficient unless I've missed something?
>>> 
>>> It does mean that consumers of the MQTT messages need to work harder to parse them - they would need to know the order of the values.  We couldn't use a self-describing format like JSON since that has plenty of overhead of its own.
>> 
>> I think you're describing a valid approach to reduce the MQTT overhead, basically something like sending the V2 MP records over MQTT instead of single metrics.
>> 
>> I just don't know if that's what people expect from MQTT. It removes all the simplicity from receiving metric values via MQTT. Will standard MQTT clients & tools be able to parse such records? And even if they do, it would still introduce a hurdle for users trying to get into telemetry processing. Part of the appeal of MQTT is being able to use a broad selection of generic standard clients.
>> 
>>> Another approach is - instead of having the V3/MQTT "server" on OVMS, to rather have a two way gateway that runs server-side that connects to the V2 server and proxies back and forth from MQTT.  I would see this as an "app" side gateway - so metric records coming from the car get published to mqtt (v2->v3) and commands are translated back the other way (v3->v2).  Then the extra protocol overhead for the existing MQTT format doesn't matter since we aren't going to be sending over an "IOT" SIM over 3G.
>>> 
>>> What are your thoughts?
>> 
>> That would introduce the need for a second server / gateway instance just to attach to MP via MQTT. Possible, but then, MP isn't that hard to connect directly to. There are many example clients. The gateway would still have a benefit if MQTT clients can parse MP records though.
>> 
>> How about eliminating the need for that gateway by adding optional standard & configurable custom topic record structures to the V3 server? In the most simple form, a record structure is simply a list of metric names to join under a definable topic. A little bit of number formatting would be useful, e.g. using printf style format strings.
>> 
>> Example: the MP "L" record structure could be described by some string like this:
>> 
>> {v.p.latitude},{v.p.longitude},{v.p.direction},{v.p.altitude},{v.p.gpslock},{v.p.speed},{v.p.trip},{v.e.drivemode%x},{v.b.power%.3f},{v.b.energy.used%.3f},…
>> 
>> JSON format could be supported by some keyword to request JSON value encoding. You could even introduce a way to get the value from a Javascript function, like Greg does for the obd2ecu system.
>> 
>> Combine that with a decent metric filter & update rate configuration (see issue link above – btw, we already have topic pattern support from Mongoose…), and you get the best of both approaches, without any added transport complexity or points of failure.
>> 
>> Standard users will be able to subscribe to single metric values, and power users will be able to easily create custom records.
>> 
>> Just my 2 cents as a yet-to-become MQTT user.
>> 
>>> Thanks,
>>> Steve
>> 
>> Regards,
>> Michael
>> 
>> --
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>> 
>> 
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
> 
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210118/6c30dcb8/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210118/6c30dcb8/attachment.sig>