[Ovmsdev] General trip & grid log proposal

[Michael Balzer]

Steve,

forgot to mention: the V3 server already supports "data" notifications. 
These are published under the topic scheme 
"<prefix>/notify/data/<subtype>/<id>/<timestamp>".

So you can already send compact custom records. V2 "history" records are 
sent that way, see OvmsVehicleRenaultTwizy::SendTripLog() for an example.

Regards,
Michael


Am 15.01.21 um 13:55 schrieb Michael Balzer:
> Steve,
>
> Am 15.01.21 um 10:04 schrieb Steve Davies:
>> This seems more manageable than the continuous streaming, a sensible 
>> compromise.
>
> As I said, the streaming mode wasn't meant to stream all metrics. The 
> V3 server lacks a means to control metrics update rates. Feel free to 
> add one, also take into account what has been discussed on this 
> before, i.e. 
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/344
>
>> But should I be working with MQTT?  Right now Mark and Michael you 
>> are by far the biggest contributors so I'd definitely take a steer 
>> from you about this.  I don't think it helps the project to have two 
>> different approaches "competing" in the same code base.
>
> It's not a competition, it's supporting two different approaches. We 
> even support a third approach in form of our WebSocket protocol, which 
> can be used for direct local connections to the module.
>
> V2 / MP is a custom protocol and more of a legacy support. But it has 
> undeniable advantages over the current MQTT implementation.
>
>> You say:
>>
>>   > The V3 server needs further development from actual users like 
>> you. I don't use MQTT because of the protocol overhead. So go ahead, 
>> refine it.
>>
>> So the blunt question is whether this MQTT / V3 approach is dying on 
>> the vine or if it is worth investing time into.
>>
>> Looks like Mark wrote most of it 3 years ago.
>>
>> For me I'm used to MQTT, I like it and use it in my house control 
>> systems, energy monitoring, OpenEVSE etc. It's exactly designed for 
>> the IOT world.  NodeRed, Homeassistant and other tools support it well.
>>
>> But OVMS is already invested in the V2 protocol and you are still 
>> developing it.
>>
>> As implemented, the V3/MQTT approach will definitely have more 
>> overhead - and indeed sending every metric in a separate message will 
>> use a lot more data.
>>
>> But nothing says that we can't have a "compact" format similar to the 
>> data messages that V2 sends.    MQTT adds just a few bytes per 
>> published message for the MQTT protocol so that would end up being 
>> nearly as efficient unless I've missed something?
>>
>> It does mean that consumers of the MQTT messages need to work harder 
>> to parse them - they would need to know the order of the values.  We 
>> couldn't use a self-describing format like JSON since that has plenty 
>> of overhead of its own.
>
> I think you're describing a valid approach to reduce the MQTT 
> overhead, basically something like sending the V2 MP records over MQTT 
> instead of single metrics.
>
> I just don't know if that's what people expect from MQTT. It removes 
> all the simplicity from receiving metric values via MQTT. Will 
> standard MQTT clients & tools be able to parse such records? And even 
> if they do, it would still introduce a hurdle for users trying to get 
> into telemetry processing. Part of the appeal of MQTT is being able to 
> use a broad selection of generic standard clients.
>
>> Another approach is - instead of having the V3/MQTT "server" on OVMS, 
>> to rather have a two way gateway that runs server-side that connects 
>> to the V2 server and proxies back and forth from MQTT.  I would see 
>> this as an "app" side gateway - so metric records coming from the car 
>> get published to mqtt (v2->v3) and commands are translated back the 
>> other way (v3->v2).  Then the extra protocol overhead for the 
>> existing MQTT format doesn't matter since we aren't going to be 
>> sending over an "IOT" SIM over 3G.
>>
>> What are your thoughts?
>
> That would introduce the need for a second server / gateway instance 
> just to attach to MP via MQTT. Possible, but then, MP isn't that hard 
> to connect directly to. There are many example clients. The gateway 
> would still have a benefit if MQTT clients can parse MP records though.
>
> How about eliminating the need for that gateway by adding optional 
> standard & configurable custom topic record structures to the V3 
> server? In the most simple form, a record structure is simply a list 
> of metric names to join under a definable topic. A little bit of 
> number formatting would be useful, e.g. using printf style format strings.
>
> Example: the MP "L" record structure could be described by some string 
> like this:
>
> {v.p.latitude},{v.p.longitude},{v.p.direction},{v.p.altitude},{v.p.gpslock},{v.p.speed},{v.p.trip},{v.e.drivemode%x},{v.b.power%.3f},{v.b.energy.used%.3f},…
>
> JSON format could be supported by some keyword to request JSON value 
> encoding. You could even introduce a way to get the value from a 
> Javascript function, like Greg does for the obd2ecu system.
>
> Combine that with a decent metric filter & update rate configuration 
> (see issue link above – btw, we already have topic pattern support 
> from Mongoose…), and you get the best of both approaches, without any 
> added transport complexity or points of failure.
>
> Standard users will be able to subscribe to single metric values, and 
> power users will be able to easily create custom records.
>
> Just my 2 cents as a yet-to-become MQTT user.
>
>> Thanks,
>> Steve
>
> Regards,
> Michael
>
> -- 
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210116/d2812128/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210116/d2812128/attachment.sig>