Steve, the trip log will have one entry per trip, i.e. at the end of the trip, as a regular trip log. The grid log is meant to get one entry per change of either the charge or the generator state. Both are historical records, i.e. kept on the server until deletion or expiry. If you're looking for a GPS log, we've already got that in form of the "L" messages. These are kept for 24 hours, and the log frequency adapts to user choice and App connection state ("streaming mode", feature #8 / config vehicle stream). Regards, Michael Am 14.01.21 um 07:29 schrieb Steve Davies:

On Tue, 12 Jan 2021 at 20:50, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

followup to my previous mail, these are the record structures for the general trip & grid logs I would like to add.

Michael - as a newbie I'm probably asking rather elementary questions:  is the idea to post one "LOG-Trip" at the end of a trip, or to post a string of them as long as the car is "on"?

Thanks, Steve

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Steve, not your fault, missing V3 documentation. These are features ported from V2, which was configured by a fixed array of "features" & "parameters". You can find info on these in the V2 documentation. Setting features & parameters is still supported by the Apps, and is more convenient for users than learning a config command. For complex features, adding an editor to the Apps is recommended, but simple configuration values can be integrated easily this way. V3 has a mapper for these, you can extend for custom vehicle features: class OvmsVehicle …     virtual bool SetFeature(int key, const char* value);     virtual const std::string GetFeature(int key); We only support a few V2 standard features by default in V3, the streaming mode is one of them. V2 only uses the streaming mode for the location message, and additionally requires the vehicle to be on and some client to be connected. It also supports not only enabling the streaming mode, but also configuring the interval (seconds) by the feature / config. The V3 / MQTT server is still experimental, the implementation may change and be changed by you. It currently apparently only supports enabling/disabling the streaming mode and, you're right, if enabled applies this to all metrics without any interval. I suppose that was an initial, very simple implementation of streaming mode. I think it isn't very useful, as you may have metrics that change a hundred times per second, and the V3 server will try to transmit every single change with streaming enabled. It looks like the interval support is prepared in the Ticker1() method though, the same way as in the V2 server. The V3 server needs further development from actual users like you. I don't use MQTT because of the protocol overhead. So go ahead, refine it. The way MQTT works is outlined in the list archive and by an example script: - Topic structure: http://lists.openvehicles.com/pipermail/ovmsdev/2018-July/005354.html - Command shell example: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/blob/master... …and all this really needs documentation… Regards, Michael Am 14.01.21 um 16:53 schrieb Steve Davies:

On Thu, 14 Jan 2021 at 09:23, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Both are historical records, i.e. kept on the server until deletion or expiry. If you're looking for a GPS log, we've already got that in form of the "L" messages. These are kept for 24 hours, and the log frequency adapts to user choice and App connection state ("streaming mode", feature #8 / config vehicle stream).

Thanks!

Is there documentation that I failed to find?  IE "feature #8" - seems like from a list somewhere?

I'm working at the moment with the MQTT server.

In the mqtt server code I found "m_streaming" and it looks like I need "config set vehicle stream 1"

If I understand correctly this will result in metrics being published to mqtt as soon as they change.

Looking at the code I'm also trying to understand what exactly a "client" is.  Here's how it looks to me:  a client sends a message with topic <prefix>/client/<clientid>/active with the payload being non-0 to "connect" and 0 or empty to disconnect.   Must send that at least once every 120 seconds to stay connected.

With a client "connected" the MQTT server will send metrics at the updatetime.connected rate rather than the idle rate (nowithstanding that with "stream 1" they are sent immediately).

Clients can also send commands too: <prefix>/client/<clientid>/command

Am I on the right track?

Thanks, Steve

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Steve, Am 15.01.21 um 10:04 schrieb Steve Davies:

This seems more manageable than the continuous streaming, a sensible compromise.

As I said, the streaming mode wasn't meant to stream all metrics. The V3 server lacks a means to control metrics update rates. Feel free to add one, also take into account what has been discussed on this before, i.e. https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/344

But should I be working with MQTT?  Right now Mark and Michael you are by far the biggest contributors so I'd definitely take a steer from you about this.  I don't think it helps the project to have two different approaches "competing" in the same code base.

It's not a competition, it's supporting two different approaches. We even support a third approach in form of our WebSocket protocol, which can be used for direct local connections to the module. V2 / MP is a custom protocol and more of a legacy support. But it has undeniable advantages over the current MQTT implementation.

You say:

  > The V3 server needs further development from actual users like you. I don't use MQTT because of the protocol overhead. So go ahead, refine it.

So the blunt question is whether this MQTT / V3 approach is dying on the vine or if it is worth investing time into.

Looks like Mark wrote most of it 3 years ago.

For me I'm used to MQTT, I like it and use it in my house control systems, energy monitoring, OpenEVSE etc.  It's exactly designed for the IOT world.  NodeRed, Homeassistant and other tools support it well.

But OVMS is already invested in the V2 protocol and you are still developing it.

As implemented, the V3/MQTT approach will definitely have more overhead - and indeed sending every metric in a separate message will use a lot more data.

But nothing says that we can't have a "compact" format similar to the data messages that V2 sends.    MQTT adds just a few bytes per published message for the MQTT protocol so that would end up being nearly as efficient unless I've missed something?

It does mean that consumers of the MQTT messages need to work harder to parse them - they would need to know the order of the values.  We couldn't use a self-describing format like JSON since that has plenty of overhead of its own.

I think you're describing a valid approach to reduce the MQTT overhead, basically something like sending the V2 MP records over MQTT instead of single metrics. I just don't know if that's what people expect from MQTT. It removes all the simplicity from receiving metric values via MQTT. Will standard MQTT clients & tools be able to parse such records? And even if they do, it would still introduce a hurdle for users trying to get into telemetry processing. Part of the appeal of MQTT is being able to use a broad selection of generic standard clients.

Another approach is - instead of having the V3/MQTT "server" on OVMS, to rather have a two way gateway that runs server-side that connects to the V2 server and proxies back and forth from MQTT.  I would see this as an "app" side gateway - so metric records coming from the car get published to mqtt (v2->v3) and commands are translated back the other way (v3->v2).  Then the extra protocol overhead for the existing MQTT format doesn't matter since we aren't going to be sending over an "IOT" SIM over 3G.

What are your thoughts?

That would introduce the need for a second server / gateway instance just to attach to MP via MQTT. Possible, but then, MP isn't that hard to connect directly to. There are many example clients. The gateway would still have a benefit if MQTT clients can parse MP records though. How about eliminating the need for that gateway by adding optional standard & configurable custom topic record structures to the V3 server? In the most simple form, a record structure is simply a list of metric names to join under a definable topic. A little bit of number formatting would be useful, e.g. using printf style format strings. Example: the MP "L" record structure could be described by some string like this: {v.p.latitude},{v.p.longitude},{v.p.direction},{v.p.altitude},{v.p.gpslock},{v.p.speed},{v.p.trip},{v.e.drivemode%x},{v.b.power%.3f},{v.b.energy.used%.3f},… JSON format could be supported by some keyword to request JSON value encoding. You could even introduce a way to get the value from a Javascript function, like Greg does for the obd2ecu system. Combine that with a decent metric filter & update rate configuration (see issue link above – btw, we already have topic pattern support from Mongoose…), and you get the best of both approaches, without any added transport complexity or points of failure. Standard users will be able to subscribe to single metric values, and power users will be able to easily create custom records. Just my 2 cents as a yet-to-become MQTT user.

Thanks, Steve

Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Regarding MQTT, I definitely think it is the correct direction for the future, and hope it can replace v2 protocol in the long run. The advantages are many, and the only disadvantage seems to be data size on the wire. To address that, I have been waiting for v5 MQTT to become more widespread. As well as some other desirable features, that adds topic aliases. This means we can replace the topic name with a 4 byte integer. Along with the protocol overhead that is still slightly more than the single byte comma field separator of v2 protocol, but has the advantage that a single changed metric can be transmitted without having to transmit the entire v2 protocol message. Implementation is relatively straightforward. The first time a particular message is sent, the full topic name and value are used, along with a 4 byte integer topic alias. Subsequent updates (on the same connection session) can leave the full topic name blank and just provide the 4 byte integer topic alias (with the expansion done at the server side). There is also a bunch of stuff in v5 regarding session resumption, to further reduce the overhead. There is now a development version of mosquito that supports v5, and hivemq has become sort of open source. Our current networking library doesn’t support it, however. Perhaps now is the time to re-visit this (in particular if others are willing to work on it)? Regards, Mark.

On 16 Jan 2021, at 3:50 PM, Michael Balzer <dexter@expeedo.de> wrote:

Signed PGP part Steve,

forgot to mention: the V3 server already supports "data" notifications. These are published under the topic scheme "<prefix>/notify/data/<subtype>/<id>/<timestamp>".

So you can already send compact custom records. V2 "history" records are sent that way, see OvmsVehicleRenaultTwizy::SendTripLog() for an example.

Regards, Michael

Am 15.01.21 um 13:55 schrieb Michael Balzer:

...

Steve,

Am 15.01.21 um 10:04 schrieb Steve Davies:

...

This seems more manageable than the continuous streaming, a sensible compromise.

As I said, the streaming mode wasn't meant to stream all metrics. The V3 server lacks a means to control metrics update rates. Feel free to add one, also take into account what has been discussed on this before, i.e. https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/344 <https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/344>

...

But should I be working with MQTT? Right now Mark and Michael you are by far the biggest contributors so I'd definitely take a steer from you about this. I don't think it helps the project to have two different approaches "competing" in the same code base.

It's not a competition, it's supporting two different approaches. We even support a third approach in form of our WebSocket protocol, which can be used for direct local connections to the module.

V2 / MP is a custom protocol and more of a legacy support. But it has undeniable advantages over the current MQTT implementation.

...

You say:

...

The V3 server needs further development from actual users like you. I don't use MQTT because of the protocol overhead. So go ahead, refine it.

So the blunt question is whether this MQTT / V3 approach is dying on the vine or if it is worth investing time into.

Looks like Mark wrote most of it 3 years ago.

For me I'm used to MQTT, I like it and use it in my house control systems, energy monitoring, OpenEVSE etc. It's exactly designed for the IOT world. NodeRed, Homeassistant and other tools support it well.

But OVMS is already invested in the V2 protocol and you are still developing it.

As implemented, the V3/MQTT approach will definitely have more overhead - and indeed sending every metric in a separate message will use a lot more data.

But nothing says that we can't have a "compact" format similar to the data messages that V2 sends. MQTT adds just a few bytes per published message for the MQTT protocol so that would end up being nearly as efficient unless I've missed something?

It does mean that consumers of the MQTT messages need to work harder to parse them - they would need to know the order of the values. We couldn't use a self-describing format like JSON since that has plenty of overhead of its own.

I think you're describing a valid approach to reduce the MQTT overhead, basically something like sending the V2 MP records over MQTT instead of single metrics.

I just don't know if that's what people expect from MQTT. It removes all the simplicity from receiving metric values via MQTT. Will standard MQTT clients & tools be able to parse such records? And even if they do, it would still introduce a hurdle for users trying to get into telemetry processing. Part of the appeal of MQTT is being able to use a broad selection of generic standard clients.

...

Another approach is - instead of having the V3/MQTT "server" on OVMS, to rather have a two way gateway that runs server-side that connects to the V2 server and proxies back and forth from MQTT. I would see this as an "app" side gateway - so metric records coming from the car get published to mqtt (v2->v3) and commands are translated back the other way (v3->v2). Then the extra protocol overhead for the existing MQTT format doesn't matter since we aren't going to be sending over an "IOT" SIM over 3G.

What are your thoughts?

That would introduce the need for a second server / gateway instance just to attach to MP via MQTT. Possible, but then, MP isn't that hard to connect directly to. There are many example clients. The gateway would still have a benefit if MQTT clients can parse MP records though.

How about eliminating the need for that gateway by adding optional standard & configurable custom topic record structures to the V3 server? In the most simple form, a record structure is simply a list of metric names to join under a definable topic. A little bit of number formatting would be useful, e.g. using printf style format strings.

Example: the MP "L" record structure could be described by some string like this:

{v.p.latitude},{v.p.longitude},{v.p.direction},{v.p.altitude},{v.p.gpslock},{v.p.speed},{v.p.trip},{v.e.drivemode%x},{v.b.power%.3f},{v.b.energy.used%.3f},…

JSON format could be supported by some keyword to request JSON value encoding. You could even introduce a way to get the value from a Javascript function, like Greg does for the obd2ecu system.

Combine that with a decent metric filter & update rate configuration (see issue link above – btw, we already have topic pattern support from Mongoose…), and you get the best of both approaches, without any added transport complexity or points of failure.

Standard users will be able to subscribe to single metric values, and power users will be able to easily create custom records.

Just my 2 cents as a yet-to-become MQTT user.

...

Thanks, Steve

Regards, Michael

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Dear Michael, thanks for your great work! This was something I've been very much looking forward to :) Will this be integrated in the android app? Best regards, sharkcow Am 24.01.21 um 17:08 schrieb Michael Balzer:

Everyone,

the trip & grid logs are now implemented as proposed.

I've made a small change to the trip log to include min/max values of the tyre temperatures, pressures and health levels.

Documentation:

* https://docs.openvehicles.com/en/latest/userguide/notifications.html#grid-hi... * https://docs.openvehicles.com/en/latest/userguide/notifications.html#trip-hi...

Note: both logs are disabled by default.

Regards, Michael

Am 12.01.21 um 19:49 schrieb Michael Balzer:

...

Everyone,

followup to my previous mail, these are the record structures for the general trip & grid logs I would like to add.

I'm going to implement this with configurable server hold times, with 0 = no server storage at all.

As this has more potential for data privacy impact as the standard 24 hour storage of telemetry records, I wouldn't enable the logs by default, but leave this decision to the user.

Please check & provide feedback.

Regards, Michael

====================================================================================================

History type "*-LOG-Trip" Notification type "data", subtype "log.trip"

    OvmsMetricBool*   ms_v_pos_gpslock;     OvmsMetricFloat*  ms_v_pos_latitude;     OvmsMetricFloat*  ms_v_pos_longitude;     OvmsMetricFloat*  ms_v_pos_altitude;     OvmsMetricString* ms_v_pos_location;              // Name of current location if defined

    OvmsMetricFloat*  ms_v_pos_odometer;

    OvmsMetricFloat*  ms_v_pos_trip;                  // Trip distance     OvmsMetricInt*    ms_v_env_drivetime;             // Trip duration     OvmsMetricInt*    ms_v_env_drivemode;             // Active drive profile number [1]

    OvmsMetricFloat*  ms_v_bat_soc;                   // State of charge [%]     OvmsMetricFloat*  ms_v_bat_range_est;             // Estimated range [km]     OvmsMetricFloat*  ms_v_bat_range_ideal;           // Ideal range [km]     OvmsMetricFloat*  ms_v_bat_range_full;            // Ideal range at 100% SOC & current conditions [km]

    OvmsMetricFloat*  ms_v_bat_energy_used;           // Main battery energy used on trip [kWh]     OvmsMetricFloat*  ms_v_bat_energy_recd;           // Main battery energy recovered on trip [kWh]     OvmsMetricFloat*  ms_v_bat_coulomb_used;     OvmsMetricFloat*  ms_v_bat_coulomb_recd;

    OvmsMetricFloat*  ms_v_bat_soh;                   // State of health [%]     OvmsMetricString* ms_v_bat_health;                // General textual description of battery health     OvmsMetricFloat*  ms_v_bat_cac;                   // Calculated capacity [Ah]

    OvmsMetricFloat*  ms_v_bat_energy_used_total;     // Battery energy used total (life time) [kWh]     OvmsMetricFloat*  ms_v_bat_energy_recd_total;     // Battery energy recovered total (life time) [kWh]     OvmsMetricFloat*  ms_v_bat_coulomb_used_total;    // Battery coulomb used total (life time) [Ah]     OvmsMetricFloat*  ms_v_bat_coulomb_recd_total;    // Battery coulomb recovered total (life time) [Ah]

    OvmsMetricFloat*  ms_v_env_temp;                  // Ambient temperature [°C]     OvmsMetricFloat*  ms_v_env_cabintemp;             // Cabin temperature [°C]     OvmsMetricFloat*  ms_v_bat_temp;                  // Battery temperature [°C]     OvmsMetricFloat*  ms_v_inv_temp;                  // Inverter temperature [°C]     OvmsMetricFloat*  ms_v_mot_temp;                  // Motor temperature [°C]     OvmsMetricFloat*  ms_v_charge_12v_temp;           // Temperature of DC/DC-converter [°C]

    OvmsMetricFloat*  MAX(ms_v_tpms_temp);            // Maximum tyre temperature     OvmsMetricFloat*  MIN(ms_v_tpms_pressure);        // Minimum tyre pressure     OvmsMetricFloat*  MIN(ms_v_tpms_health);          // Minimum tyre health

====================================================================================================

History type "*-LOG-Grid" Notification type "data", subtype "log.grid"

    OvmsMetricBool*   ms_v_pos_gpslock;     OvmsMetricFloat*  ms_v_pos_latitude;     OvmsMetricFloat*  ms_v_pos_longitude;     OvmsMetricFloat*  ms_v_pos_altitude;     OvmsMetricString* ms_v_pos_location;            // Name of current location if defined

    OvmsMetricFloat*  ms_v_pos_odometer;

    OvmsMetricString* ms_v_charge_type;               // Grid connection type     OvmsMetricString* ms_v_charge_state;              // charging, topoff, done, prepare, …     OvmsMetricString* ms_v_charge_substate;           // scheduledstop, scheduledstart, …     OvmsMetricString* ms_v_charge_mode;               // standard, range, performance, storage     OvmsMetricFloat*  ms_v_charge_climit;             // Maximum charger output current [A]     OvmsMetricFloat*  ms_v_charge_limit_range;        // Sufficient range limit for current charge [km]     OvmsMetricFloat*  ms_v_charge_limit_soc;          // Sufficient SOC limit for current charge [%]

    OvmsMetricString* ms_v_gen_type;                  // Grid connection type     OvmsMetricString* ms_v_gen_state;                 // TBD     OvmsMetricString* ms_v_gen_substate;              // TBD     OvmsMetricString* ms_v_gen_mode;                  // TBD     OvmsMetricFloat*  ms_v_gen_climit;                // Maximum battery output current [A]     OvmsMetricFloat*  ms_v_gen_limit_range;           // Min range [km]     OvmsMetricFloat*  ms_v_gen_limit_soc;             // Min SOC [%]

    OvmsMetricInt*    ms_v_charge_time;               // Duration of running charge session [sec]     OvmsMetricFloat*  ms_v_charge_kwh;                // Energy charged into battery for running session [kWh]     OvmsMetricFloat*  ms_v_charge_kwh_grid;           // Energy drawn from grid during running session [kWh]     OvmsMetricFloat*  ms_v_charge_kwh_grid_total;     // Energy drawn from grid total (life time) [kWh]

    OvmsMetricInt*    ms_v_gen_time;                  // Duration of running generator session [sec]     OvmsMetricFloat*  ms_v_gen_kwh;                   // Energy taken from battery for running session [kWh]     OvmsMetricFloat*  ms_v_gen_kwh_grid;              // Energy sent to grid during running session [kWh]     OvmsMetricFloat*  ms_v_gen_kwh_grid_total;        // Energy sent to grid total (life time) [kWh]

    OvmsMetricFloat*  ms_v_bat_soc;                   // State of charge [%]     OvmsMetricFloat*  ms_v_bat_range_est;             // Estimated range [km]     OvmsMetricFloat*  ms_v_bat_range_ideal;           // Ideal range [km]     OvmsMetricFloat*  ms_v_bat_range_full;            // Ideal range at 100% SOC & current conditions [km]

    OvmsMetricFloat*  ms_v_bat_voltage;               // Main battery momentary voltage [V]     OvmsMetricFloat*  ms_v_bat_temp;                  // Battery temperature [°C]

    OvmsMetricFloat*  ms_v_charge_temp;               // Charger temperature [°C]     OvmsMetricFloat*  ms_v_charge_12v_temp;           // Temperature of DC/DC-converter [°C]     OvmsMetricFloat*  ms_v_env_temp;                  // Ambient temperature [°C]     OvmsMetricFloat*  ms_v_env_cabintemp;             // Cabin temperature [°C]

    OvmsMetricFloat*  ms_v_bat_soh;                   // State of health [%]     OvmsMetricString* ms_v_bat_health;                // General textual description of battery health     OvmsMetricFloat*  ms_v_bat_cac;                   // Calculated capacity [Ah]

    OvmsMetricFloat*  ms_v_bat_energy_used_total;     // Battery energy used total (life time) [kWh]     OvmsMetricFloat*  ms_v_bat_energy_recd_total;     // Battery energy recovered total (life time) [kWh]     OvmsMetricFloat*  ms_v_bat_coulomb_used_total;    // Battery coulomb used total (life time) [Ah]     OvmsMetricFloat*  ms_v_bat_coulomb_recd_total;    // Battery coulomb recovered total (life time) [Ah]

_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev