[Ovmsdev] General trip & grid log proposal

[Michael Balzer]

Steve,

Am 15.01.21 um 10:04 schrieb Steve Davies:
> This seems more manageable than the continuous streaming, a sensible 
> compromise.

As I said, the streaming mode wasn't meant to stream all metrics. The V3 
server lacks a means to control metrics update rates. Feel free to add 
one, also take into account what has been discussed on this before, i.e. 
https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/344

> But should I be working with MQTT?  Right now Mark and Michael you are 
> by far the biggest contributors so I'd definitely take a steer from 
> you about this.  I don't think it helps the project to have two 
> different approaches "competing" in the same code base.

It's not a competition, it's supporting two different approaches. We 
even support a third approach in form of our WebSocket protocol, which 
can be used for direct local connections to the module.

V2 / MP is a custom protocol and more of a legacy support. But it has 
undeniable advantages over the current MQTT implementation.

> You say:
>
>   > The V3 server needs further development from actual users like 
> you. I don't use MQTT because of the protocol overhead. So go ahead, 
> refine it.
>
> So the blunt question is whether this MQTT / V3 approach is dying on 
> the vine or if it is worth investing time into.
>
> Looks like Mark wrote most of it 3 years ago.
>
> For me I'm used to MQTT, I like it and use it in my house control 
> systems, energy monitoring, OpenEVSE etc.  It's exactly designed for 
> the IOT world.  NodeRed, Homeassistant and other tools support it well.
>
> But OVMS is already invested in the V2 protocol and you are still 
> developing it.
>
> As implemented, the V3/MQTT approach will definitely have more 
> overhead - and indeed sending every metric in a separate message will 
> use a lot more data.
>
> But nothing says that we can't have a "compact" format similar to the 
> data messages that V2 sends.    MQTT adds just a few bytes per 
> published message for the MQTT protocol so that would end up being 
> nearly as efficient unless I've missed something?
>
> It does mean that consumers of the MQTT messages need to work harder 
> to parse them - they would need to know the order of the values.  We 
> couldn't use a self-describing format like JSON since that has plenty 
> of overhead of its own.

I think you're describing a valid approach to reduce the MQTT overhead, 
basically something like sending the V2 MP records over MQTT instead of 
single metrics.

I just don't know if that's what people expect from MQTT. It removes all 
the simplicity from receiving metric values via MQTT. Will standard MQTT 
clients & tools be able to parse such records? And even if they do, it 
would still introduce a hurdle for users trying to get into telemetry 
processing. Part of the appeal of MQTT is being able to use a broad 
selection of generic standard clients.

> Another approach is - instead of having the V3/MQTT "server" on OVMS, 
> to rather have a two way gateway that runs server-side that connects 
> to the V2 server and proxies back and forth from MQTT.  I would see 
> this as an "app" side gateway - so metric records coming from the car 
> get published to mqtt (v2->v3) and commands are translated back the 
> other way (v3->v2).  Then the extra protocol overhead for the existing 
> MQTT format doesn't matter since we aren't going to be sending over an 
> "IOT" SIM over 3G.
>
> What are your thoughts?

That would introduce the need for a second server / gateway instance 
just to attach to MP via MQTT. Possible, but then, MP isn't that hard to 
connect directly to. There are many example clients. The gateway would 
still have a benefit if MQTT clients can parse MP records though.

How about eliminating the need for that gateway by adding optional 
standard & configurable custom topic record structures to the V3 server? 
In the most simple form, a record structure is simply a list of metric 
names to join under a definable topic. A little bit of number formatting 
would be useful, e.g. using printf style format strings.

Example: the MP "L" record structure could be described by some string 
like this:

{v.p.latitude},{v.p.longitude},{v.p.direction},{v.p.altitude},{v.p.gpslock},{v.p.speed},{v.p.trip},{v.e.drivemode%x},{v.b.power%.3f},{v.b.energy.used%.3f},…

JSON format could be supported by some keyword to request JSON value 
encoding. You could even introduce a way to get the value from a 
Javascript function, like Greg does for the obd2ecu system.

Combine that with a decent metric filter & update rate configuration 
(see issue link above – btw, we already have topic pattern support from 
Mongoose…), and you get the best of both approaches, without any added 
transport complexity or points of failure.

Standard users will be able to subscribe to single metric values, and 
power users will be able to easily create custom records.

Just my 2 cents as a yet-to-become MQTT user.

> Thanks,
> Steve

Regards,
Michael

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210115/ab2babdb/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210115/ab2babdb/attachment-0001.sig>