[Ovmsdev] Notifications concept draft

[Michael Balzer]

Hi Mark,

apologies as well, I'm quite low on spare time currently :-/

I agree it's better to implement the message broker directly, as adding it later on would need changes to all listeners.

ACK on your thoughts, except a) the fixed alert texts and b) the message producing scheme.

a) Providing default messages for standard alert types is good, but replacing them with custom message text should be possible. I.e. the charge alerts need to
be specific for the Twizy. (When using commands for message generation this would automatically be handled by the vehicle specific "stat" command implementation)

b) The point I don't see in the scheme of vehicles sending messages to the broker instead of "get my message" events/calls: the car does not know about the
verbosity levels needed by the broker's subscribers. That's why I think it's better to let the broker fetch the messages than to have the producer send them to
the broker.

Example: the car raises a custom alert. The car implementation wants to add detail info to the basic alert if possible.

An SMS channel has subscribed to the broker with verbosity=minimal (140), and for example a shell and an ssh session (thanks, Steven!) each have subscribed with
verbosity=normal (1024). So the broker knows it will need messages of both 140 and 1024 char verbosity.

The car now sends the "get my alert" event (or call to the broker). The broker in turn requests the alert message from the car twice, one variant tailored for
verbosity=140 and another for verbosity=1024. It then queues & delivers the message variants to the listeners according to their verbosity levels and finally
discards them.

As the verbosity is already part of the ovms_command callback API, all command implementations should take care of the message length limitation anyway, so can
easily be used for alert and notification generation as well.


Regards,
Michael


Am 10.11.2017 um 07:07 schrieb Mark Webb-Johnson:
> Apologies for the delay in getting back to you on this.
>
> We’re up to the stage now where we have to determine how best to implement this. The rest of the ovms_server_v2 stuff is pretty much done.
>
> So, let’s break the requirements down:
>
>  1. Metrics -> v2 messages
>  2. Push notifications (car -> server -> apps)
>  3. Error notifications (car -> server -> apps)
>  4. Historical data submissions (car -> server, and presumably later server -> apps)
>
>
> Let’s go through them one by one.
>
> _*1. Metrics -> v2 messages*_
>
> I think this works pretty much ok at the moment. The metrics have modified flags, and ovms_server_v2 (and others) can get notified whenever a metric is
> modified. We can either issue the update immediately, or delay it for later transmission.
>
> The way we do it at the moment in ovms_server_v2 is to delay until the next poll. I think that should be ok. 99% of the time, if a bunch of metrics are
> updated, they’ll all make it into the same update. If not, it shouldn’t do too much harm (just client app screen will get two updates, rather than one). For
> ovms_server_v3, most likely these are going to go out immediately, via MQTT, as they are changed.
>
> _*2. Push notifications (car -> server -> apps)*_
>
> In v2, these are fixed strings sent by net_msg_alert() - but originally raised by net_req_notification(). It is a simple bit - set it and at some time later
> the alert will be sent then the bit cleared. It seems that the twizy vehicle also manually sends push alerts as well.
>
> For v3, I think we need something similar. The ability to queue a notification message (with a timeout would be helpful) and have it delivered as a push
> notification. I suggest we have standardised message types, plus the ability to send any custom string. It would be good to de-duplicate based on the message
> type, and rate limit.
>
> _*3. Error notifications (car -> server -> apps)*_
>
> In v2, these are (code,data) sent by net_msg_erroralert() - but originally raised by net_req_notification_error(). There is some logic (that doesn’t work very
> well) to try to stop duplicate (repetitive) alerts, but essentially it is implemented by setting a code, and at some time later the error notification will be
> sent then the code cleared.
>
> For v3, I think we need something similar. I think this can be merged in with push notifications, as it is pretty similar. It is also interesting how we can
> deal with OBDII DTCs, as they seem to have a similar requirement.
>
> _*4. Historical data submissions (car -> server, and presumably later server -> apps)*_
>
> In v2, the logging.{h,c} module provides functions to put a log message into the queue, and then is polled by the net framework to send outstanding log
> messages. It seems that the Twizzy vehicle also manually send historical data submissions as well.
>
> For v3, I think we need to make a generic module to handle this. Something that can queue a message, and have it reliably delivered at some later point in time.
>
> _*v3 approach*_
>
> I think that we can do something along the lines of what you are suggesting, but that all three should fit into a standardised structure and queueing mechanism.
>
> Looking through the way things were done in v2, and what we now know we need, I suggest that overloading the events system is not necessary. On the producer
> side, we can simply have commands to raise these alerts/error/historical-data. On the custom consumer side, we expose it to scripts very simply.
>
> However, reviewing the v2 code, we do need something more sophisticated. Some examples:
>
>   * We really need to de-duplicate and/or rate-limit notifications. Particularly Error notifications. Sometimes the car goes crazy and outputs multiple alerts
>     (not always the same code), and those get delivered as a stream of annoying push notifications. We need something like syslog in unix - where rather than
>     giving you 1,000 lines saying the same thing, it says something like “1,000 copies of X were raised”.
>
>   * It would be good to have timeouts, and possibly time stamps. Sometimes you have no connectivity, then five hours later when it comes back you get an alert
>     that charging was interrupted. It would be good to have a timeout on a message (don’t bother to send it if it can’t be sent in the next 5 minutes), and
>     timestamps (so if message is not sent in realtime, when it is actually sent, it has a timestamp added to show when).
>
>   * We need more reliable delivery. Historical data has ACKs for this. I think for PUSH, we can do something similar, using v2 protocol PINGs. The car can
>     send the push notification, followed by a v2 ping. If it gets an ack, it can assume the notification message was received.
>
>
> The ESP32 platform has plenty of power for this. We just need to be careful with RAM usage. Perhaps we can overflow to flash, if necessary? Anyway, that can
> be hidden inside the specific implementation.
>
> There are two possible implementations of this, once a producer send in a message.
>
>  1. Each subscriber gets a copy of that message, and is responsible for all the above maintenance as well as queueing for delivery.
>  2. The message is maintained and queued centrally. Subscribers then receive notifications of changes.
>
>
> Implementation #2 is my preference. Having a centralised middle queue between the publishers and subscribers means we only need to queue once. Rather than
> each subscriber having to handle its own queueing. Given the RAM constraints, I really think this is the best approach. We’ve already done something similar
> for metrics, and that is very frugal on RAM for the modification monitors.
>
> Regarding the method of gathering the data to send, I think we can be flexible. Support fixed IDs, static text, and callback commands (just a type+data
> storage structure). When the subscriber calls the method to retrieve the actual data, the centralised approach retrieves the data and returns it to that
> specific subscriber. Some examples:
>
>   * An alert that charging has been interrupted. We just want to queue it using a pre-defined fixed type (associated with a fixed message). We don’t need
>     command callback, or anything more complex.
>   * An error alert. Similar to the previous one, we just want to queue the alert code, data.
>   * A piece of historical data. In most cases, the data is a textual string that is just queued when it is raised (for example, a drive log).
>   * A status update. In this case, the command callback makes the most sense.
>
>
> Does that meet your requirements? I don’t mind taking on the implementation of the central queue code. The ovms_server_v2 should be simple.
>
> Regards, Mark.
>
>> On 5 Nov 2017, at 10:23 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>
>> Sorry, I missed to explain my previous thoughts.
>>
>> First of all, this is not about atomic updates of single metrics, those can be done completely independant of the vehicle implementation through the metrics
>> system. This is about text notifications/alerts and pushing structured data/logging objects (historical messages).
>>
>> Most of my push notifications, alerts and data updates are vehicle specific, and most do already have a "pull" command interface as well. For example the
>> output of "power report" will also be sent as the trip efficiency notification after a drive, and "batt status" is used for the battery alert. Just like the
>> standard "stat" command is also used by the "charge" notifications (just with slightly different content on the Twizy).
>>
>> I think that's a pattern, as most textual status command outputs will be usable for notifications and alerts as well. So generalizing this was the main idea.
>> I don't know if using the commands for scripting and logging does offer an advantage, just added that as another potential benefit.
>>
>> Automatic notifications on metrics change would require all metrics updates to be organized so that the "trigger" metric is updated after all other metrics
>> required by a message. So the vehicle developer always needs to be aware of this implicit logic. Also a vehicle module may find it necessary to send a
>> notification independant of the value change for the "trigger" metric.
>>
>> Independant of the signal source (event/metric), transmissions of any type cannot be done synchronously in any listener, both because it would be executed in
>> the sender context and because any transmission channel needs to cope with temporary connection losses, so needs to queue the data anyway. So the "send"
>> event listeners should always just fetch and queue the message during event handling. I.e.
>>
>>   * ovms_server_v2 has a BufferedShell object as the transmit queue anyway
>>   * RT vehicle module sends a “xrt log RT-BAT-C” send.data event
>>   * ovms_server_v2 calls the “xrt log RT-BAT-C” command
>>   * The command executes and populates BufferedShell with textual data.
>>   * ovms_server_v2 flags internally for pending transmission & returns from signal handling
>>
>> If many such channels turn out to exist in parallel, a message broker may step in place for the queueing. The LogBuffers class already has a
>> producer/consumer pattern, so the broker could manage the buffers for all channels.
>>
>> The major problem I see with sending the data with the signal is a conflict with the verbosity concept: at the time of the signal creation, the sender is not
>> aware of the verbosity required for the individual listeners. So the sender would need to fill a message structure containing variants for all verbosity
>> levels for any notification, which may be unnecessary overhead for most cases.
>>
>> That is, unless we drop the verbosity concept for unsolicited messages. It may be irrelevant for notifications and alerts, it sure is for data? An SMS
>> channel for example can use just as much characters as fit, as notifications and alerts will normally contain the most important info at the beginning.
>>
>> The minor problem I see is again the additional memory needed -- as the channels will need to buffer the message anyway, the sender can write directly to the
>> buffer. That's minor, the memory would just be needed for the signal transport.
>>
>>
>> Regards,
>> Michael
>>
>>
>> Am 05.11.2017 um 13:48 schrieb Mark Webb-Johnson:
>>> For send.info <http://send.info/>, I think it would be much better if the ovms_server_v* code could work out for itself what needs to be sent. Have a look
>>> at MetricModified() that I’ve just committed, as a starting point. (Sorry, I’d started work on that last night, but hadn’t committed yet. Done now.)
>>> Suggestion is to put the logic on ovms_server_v* MetricModified() rather than the individual vehicle modules. Things like if the car is turned on we should
>>> notify the apps, are universal.
>>>
>>> If there are cases where this is vehicle specific, then the send.info <http://send.info/> mechanism you suggest is ok; but I still think it better we don’t
>>> do this. Remember ovms_server_v3 is going to work differently (individual metrics, rather than groups). So, for vehicle specific cases (in particular for
>>> metrics not in metrics_standard.h) this approach is fine, but for standard metrics I suggest we use MetricModified() in ovms_server_v2.
>>>
>>> I like the idea of using events for send.alert and send.data. But not sure of the purpose of a command feedback for this. From what I can see, this means
>>> (for example):
>>>
>>>   * RT vehicle module sends a “xrt log RT-BAT-C” send.data event.
>>>   * ovms_server_v2 receives that event, allocates a BufferedShell object, calls the “xrt log RT-BAT-C” command.
>>>   * The command executes and populates BufferedShell with textual data.
>>>   * The command returns the data to ovms_server_v2.
>>>   * ovms_server_v2 retrieves the textual data from BufferedShell, sends it on it’s way, then frees the BufferedShell object.
>>>   * The event signal returns back to RT vehicle module.
>>>
>>> All the above is happening on the stack of the RT vehicle module (remember that commands are expensive, memory wise).
>>>
>>> My question is what does this give us, vs:
>>>
>>>   * RT vehicle module builds the textual data to send. Then signals event send.data, passing that const char* as the parameter.
>>>   * ovms_server_v2 receives the send.data event, send the const char* data on it’s way, then returns.
>>>   * The RT vehicle module gets back control, and completes.
>>>
>>>
>>> The only advantage for using commands, that I see, is being able to test this easily as a developer. But given the overhead, I can’t see the benefit
>>> outweighing the overhead.
>>>
>>> We could, of course, have commands for things like PUSH notifications, that could be used by scripts. Or alternatively, the script could just raise the
>>> event itself.
>>>
>>> Why not just send the data with the signal? What is the advantage of the command callback arrangement?
>>>
>>> Regards, Mark
>>>
>>>> On 5 Nov 2017, at 5:39 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>
>>>> Some thoughts about how to implement vehicle notifications and data logging. Please check & comment.
>>>>
>>>>
>>>> Concept:
>>>>  - handle all unsolicited transmissions from vehicles by commands
>>>>  - vehicle module signals event if a transmission is due
>>>>  - … with event data = command line to generate output
>>>>  - listeners retrieve the transmission content by executing the command
>>>>  - queueing for async delivery is done by the listeners (i.e. through BufferedShell)
>>>>
>>>>  → any shell command can be used for transmissions
>>>>  → all transmissions can also be generated on the shell / by script
>>>>  → all transmission output can be sent to any channel
>>>>  → data logging on SD can be done transparently by the system
>>>>
>>>> Interactive consoles can output unsolicited transmissions by event type
>>>> according to the current log level.
>>>>
>>>> Events:
>>>>  "send.info <http://send.info/>"      → send text notification
>>>>  "send.alert"     → send text alert
>>>>  "send.data"      → send data record (CSV, content part of v2 MP)
>>>>
>>>> Scheme:
>>>>  MyEvents.SignalEvent("send.<type>", "command [args]");
>>>>
>>>> Examples:
>>>>  MyEvents.SignalEvent("send.info <http://send.info/>", "stat");
>>>>
>>>>  MyEvents.SignalEvent("send.alert", "xrt batt status");
>>>>
>>>>  MyEvents.SignalEvent("send.data", "xrt log RT-BAT-C");
>>>>  → command output:
>>>>    H,RT-BAT-C,4,86400,1,1,4025,3675,4045,5,14,14,19,1
>>>>
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>> -- 
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>>>
>>>
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.teslaclub.hk
>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>>
>> -- 
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.teslaclub.hk
> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20171118/4e9b874d/attachment.htm>