<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
If you don't care about verbosity, you can use the NotifyString()
method.<br>
<br>
A string notification will only be queued once and automatically
only be sent to the readers capable of handling its length.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 02.12.2017 um 20:08 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:944d71ed-b0a8-47db-8700-3d8c57ab0e80@expeedo.de">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Just pushed my changes, please have a look.<br>
<br>
Yes, this requires all needed levels of verbosity need to be
generated at the time of the event. I see no other way to support
both varying levels of verbosity and consistency of transmitted
notifications.<br>
<br>
But as the broker now knows which levels are needed by the
readers, it will only generate these.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 02.12.2017 um 18:39 schrieb
Stephen Casner:<br>
</div>
<blockquote type="cite"
cite="mid:alpine.OSX.2.21.1712020939120.22554@auge.attlocal.net">
<pre wrap="">Wouldn't that require the commands to produce all the levels of
verbosity in parallel?
-- Steve
On Sat, 2 Dec 2017, Michael Balzer wrote:
</pre>
<blockquote type="cite">
<pre wrap="">The callback needs to run at notification creation / queueing.
My idea was to let the readers register with their verbosity, so the framework can fetch all verbosity levels needed.
I'll have a look at adding that.
Regards,
Michael
Am 01.12.2017 um 05:52 schrieb Mark Webb-Johnson:
</pre>
<blockquote type="cite">
<pre wrap="">It is trivial for us to run the command at the time the OvmsNotifyEntryCommand object is created, but that would not allow the verbosity to be controlled. Just not sure if that is the requirement.
Regards, Mark.
</pre>
<blockquote type="cite">
<pre wrap="">On 1 Dec 2017, at 12:50 PM, Stephen Casner <a class="moz-txt-link-rfc2396E" href="mailto:casner@acm.org" moz-do-not-send="true"><casner@acm.org></a> wrote:
Restating a comment I made a while ago: if the command is run later,
the results might not be the same.
-- Steve
On Fri, 1 Dec 2017, Mark Webb-Johnson wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Michael,
All implemented now (including command callbacks).
@michael: I wasn't sure when to issue the command callback. At the moment it is done when the first reader requests the value (including the verbosity). But, thinking about it that may not be correct. Say the notification is raised, but reader is not ready to handle it so queued. Then ten seconds later, the reader handles it, calls GetValue(), and that causes the command to run (to gather the output). Is that correct? The command is run at the time it is dispatched, or the time it is queued? It seems that using verbosity, we need to delay the command to dispatch time.
Regards, Mark.
</pre>
<blockquote type="cite">
<pre wrap="">On 30 Nov 2017, at 2:19 PM, Mark Webb-Johnson <a class="moz-txt-link-rfc2396E" href="mailto:mark@webb-johnson.net" moz-do-not-send="true"><mark@webb-johnson.net></a> wrote:
I've completed the implementation of this (both in ovms_notify and ovms_server_v2), at least for 'text' style notification records.
It went pretty cleanly. Here are some examples:
OVMS > notify raise text info hello
Raise text notification for info as hello
I (38043) notify: Raise text info: hello
I (38773) ovms-server-v2: Send MP-0 PAhello
I (38773) notify: Cleanup type info id 1
OVMS > notify status
Notification system has 1 readers registered
Notify types:
alert: 0 entries
data: 0 entries
info: 0 entries
OVMS > notify raise text data "*-X-TEST,0,3600,mark-test-data-2"
Raise text notification for data as *-X-TEST,0,3600,mark-test-data-2
I (185702) notify: Raise text data: *-X-TEST,0,3600,mark-test-data-2
I (186572) ovms-server-v2: Send MP-0 h5,1,*-X-TEST,0,3600,mark-test-data-2
I (186792) ovms-server-v2: Incoming Msg: MP-0 h5
I (186792) notify: Cleanup type data id 5
OVMS > notify raise text alert TR,1090,0
Raise text notification for alert as TR,1090,0
I (41033) notify: Raise text alert: TR,1090,0
I (41033) ovms-server-v2: Send MP-0 PETR,1090,0
I (41033) notify: Cleanup type alert id 1
It is pretty cool to be able to 'notify raise text info hello' and have "hello" appear as a push notification on my phone.
The queueing and acknowledgement system seems to work. If I start the ovms v2 server connection, turn off wifi, raise a bunch of notifications, we can see them queued. Then, start the wifi, the server connections, and queued notifications get delivered nicely, then housekeep cleaned up. Looks good.
I haven't implemented command callback yet; I'll talk to Steve to see the best way to do that.
Regards, Mark.
</pre>
<blockquote type="cite">
<pre wrap="">On 29 Nov 2017, at 4:24 PM, Mark Webb-Johnson <<a class="moz-txt-link-abbreviated" href="mailto:mark@webb-johnson.net" moz-do-not-send="true">mark@webb-johnson.net</a> <a class="moz-txt-link-rfc2396E" href="mailto:mark@webb-johnson.net" moz-do-not-send="true"><mailto:mark@webb-johnson.net></a>> wrote:
Michael,
I've committed a framework for this. It is not complete, but probably a good state for open discussion. This is implemented in main/ovms_notify.{h,cpp}.
The main static control object is MyNotify.
Producers need to register the notification types they will be using. This only needs to be done once per type, via the MyNotify.RegisterType(const char*) function call. The notification system itself registers types 'info' and 'alert'.
Producers can call the simple functions MyNotify.NotifyString and MyNotify.NotifyCommand to raise notifications. We can build more complexity on top of these two basic types, if necessary (for example for vehicle error alerts).
Consumers need to register themselves as 'readers', once. This is done with MyNotify.RegisterReader (passing a caller name (normally logging tag), callback function). The reader will get back an ID it can store and remember. Once readers are registered, they need to be active for the lifetime of the app (until reboot).
When a notification is raised, it will be stored in an ordered list (by type). A uint32_t ID will be incrementally assigned and returned to the caller. All readers will be notified (via their callback function) with the type and event record data. The callback function returns a boolean - a 'true' value will immediately mark the record as read by that particular reader. (this has not been fully implemented yet)
A simple GetValue(int verbosity) virtual function is implemented on the event record. This can simply return the string value, or callback the command, as necessary. (this has not been fully implemented yet, but is abstracted within the notification framework)
A reader can MarkRead() the record at some later time (presumably after it has successfully handled it, and got back an indication that the transaction was successful. Once all readers have marked the record as read, it will be released and removed. (this has not been fully implemented yet)
At the moment, notification records are stored in ram (and this is handled by OvmsNotifyType. We could extend that to spool to disk, or otherwise limit memory growth. We could also permanently persist, if required to survive reboots.
Some textual test commands have been implemented for debugging, under the 'notify' command. Notifications can be manually raised, memory status shown, and tracing turned on/off.
I think this meets the requirements that we have been discussing. Notifications can be raised either as constant text strings, or command callbacks.
Feedback appreciated. In the meantime, I'll try to implement push notifications for ovms_server_v2 using this framework, as a proof of concept.
Regards, Mark.
</pre>
<blockquote type="cite">
<pre wrap="">On 18 Nov 2017, at 11:44 PM, Michael Balzer <<a class="moz-txt-link-abbreviated" href="mailto:dexter@expeedo.de" moz-do-not-send="true">dexter@expeedo.de</a> <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><mailto:dexter@expeedo.de></a>> wrote:
Hi Mark,
apologies as well, I'm quite low on spare time currently :-/
I agree it's better to implement the message broker directly, as adding it later on would need changes to all listeners.
ACK on your thoughts, except a) the fixed alert texts and b) the message producing scheme.
a) Providing default messages for standard alert types is good, but replacing them with custom message text should be possible. I.e. the charge alerts need to be specific for the Twizy. (When using commands for message generation this would automatically be handled by the vehicle specific "stat" command implementation)
b) The point I don't see in the scheme of vehicles sending messages to the broker instead of "get my message" events/calls: the car does not know about the verbosity levels needed by the broker's subscribers. That's why I think it's better to let the broker fetch the messages than to have the producer send them to the broker.
Example: the car raises a custom alert. The car implementation wants to add detail info to the basic alert if possible.
An SMS channel has subscribed to the broker with verbosity=minimal (140), and for example a shell and an ssh session (thanks, Steven!) each have subscribed with verbosity=normal (1024). So the broker knows it will need messages of both 140 and 1024 char verbosity.
The car now sends the "get my alert" event (or call to the broker). The broker in turn requests the alert message from the car twice, one variant tailored for verbosity=140 and another for verbosity=1024. It then queues & delivers the message variants to the listeners according to their verbosity levels and finally discards them.
As the verbosity is already part of the ovms_command callback API, all command implementations should take care of the message length limitation anyway, so can easily be used for alert and notification generation as well.
Regards,
Michael
Am 10.11.2017 um 07:07 schrieb Mark Webb-Johnson:
</pre>
<blockquote type="cite">
<pre wrap="">Apologies for the delay in getting back to you on this.
We're up to the stage now where we have to determine how best to implement this. The rest of the ovms_server_v2 stuff is pretty much done.
So, let's break the requirements down:
Metrics -> v2 messages
Push notifications (car -> server -> apps)
Error notifications (car -> server -> apps)
Historical data submissions (car -> server, and presumably later server -> apps)
Let's go through them one by one.
1. Metrics -> v2 messages
I think this works pretty much ok at the moment. The metrics have modified flags, and ovms_server_v2 (and others) can get notified whenever a metric is modified. We can either issue the update immediately, or delay it for later transmission.
The way we do it at the moment in ovms_server_v2 is to delay until the next poll. I think that should be ok. 99% of the time, if a bunch of metrics are updated, they'll all make it into the same update. If not, it shouldn't do too much harm (just client app screen will get two updates, rather than one). For ovms_server_v3, most likely these are going to go out immediately, via MQTT, as they are changed.
2. Push notifications (car -> server -> apps)
In v2, these are fixed strings sent by net_msg_alert() - but originally raised by net_req_notification(). It is a simple bit - set it and at some time later the alert will be sent then the bit cleared. It seems that the twizy vehicle also manually sends push alerts as well.
For v3, I think we need something similar. The ability to queue a notification message (with a timeout would be helpful) and have it delivered as a push notification. I suggest we have standardised message types, plus the ability to send any custom string. It would be good to de-duplicate based on the message type, and rate limit.
3. Error notifications (car -> server -> apps)
In v2, these are (code,data) sent by net_msg_erroralert() - but originally raised by net_req_notification_error(). There is some logic (that doesn't work very well) to try to stop duplicate (repetitive) alerts, but essentially it is implemented by setting a code, and at some time later the error notification will be sent then the code cleared.
For v3, I think we need something similar. I think this can be merged in with push notifications, as it is pretty similar. It is also interesting how we can deal with OBDII DTCs, as they seem to have a similar requirement.
4. Historical data submissions (car -> server, and presumably later server -> apps)
In v2, the logging.{h,c} module provides functions to put a log message into the queue, and then is polled by the net framework to send outstanding log messages. It seems that the Twizzy vehicle also manually send historical data submissions as well.
For v3, I think we need to make a generic module to handle this. Something that can queue a message, and have it reliably delivered at some later point in time.
v3 approach
I think that we can do something along the lines of what you are suggesting, but that all three should fit into a standardised structure and queueing mechanism.
Looking through the way things were done in v2, and what we now know we need, I suggest that overloading the events system is not necessary. On the producer side, we can simply have commands to raise these alerts/error/historical-data. On the custom consumer side, we expose it to scripts very simply.
However, reviewing the v2 code, we do need something more sophisticated. Some examples:
We really need to de-duplicate and/or rate-limit notifications. Particularly Error notifications. Sometimes the car goes crazy and outputs multiple alerts (not always the same code), and those get delivered as a stream of annoying push notifications. We need something like syslog in unix - where rather than giving you 1,000 lines saying the same thing, it says something like "1,000 copies of X were raised".
It would be good to have timeouts, and possibly time stamps. Sometimes you have no connectivity, then five hours later when it comes back you get an alert that charging was interrupted. It would be good to have a timeout on a message (don't bother to send it if it can't be sent in the next 5 minutes), and timestamps (so if message is not sent in realtime, when it is actually sent, it has a timestamp added to show when).
We need more reliable delivery. Historical data has ACKs for this. I think for PUSH, we can do something similar, using v2 protocol PINGs. The car can send the push notification, followed by a v2 ping. If it gets an ack, it can assume the notification message was received.
The ESP32 platform has plenty of power for this. We just need to be careful with RAM usage. Perhaps we can overflow to flash, if necessary? Anyway, that can be hidden inside the specific implementation.
There are two possible implementations of this, once a producer send in a message.
Each subscriber gets a copy of that message, and is responsible for all the above maintenance as well as queueing for delivery.
The message is maintained and queued centrally. Subscribers then receive notifications of changes.
Implementation #2 is my preference. Having a centralised middle queue between the publishers and subscribers means we only need to queue once. Rather than each subscriber having to handle its own queueing. Given the RAM constraints, I really think this is the best approach. We've already done something similar for metrics, and that is very frugal on RAM for the modification monitors.
Regarding the method of gathering the data to send, I think we can be flexible. Support fixed IDs, static text, and callback commands (just a type+data storage structure). When the subscriber calls the method to retrieve the actual data, the centralised approach retrieves the data and returns it to that specific subscriber. Some examples:
An alert that charging has been interrupted. We just want to queue it using a pre-defined fixed type (associated with a fixed message). We don't need command callback, or anything more complex.
An error alert. Similar to the previous one, we just want to queue the alert code, data.
A piece of historical data. In most cases, the data is a textual string that is just queued when it is raised (for example, a drive log).
A status update. In this case, the command callback makes the most sense.
Does that meet your requirements? I don't mind taking on the implementation of the central queue code. The ovms_server_v2 should be simple.
Regards, Mark.
</pre>
<blockquote type="cite">
<pre wrap="">On 5 Nov 2017, at 10:23 PM, Michael Balzer <<a class="moz-txt-link-abbreviated" href="mailto:dexter@expeedo.de" moz-do-not-send="true">dexter@expeedo.de</a> <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><mailto:dexter@expeedo.de></a>> wrote:
Sorry, I missed to explain my previous thoughts.
First of all, this is not about atomic updates of single metrics, those can be done completely independant of the vehicle implementation through the metrics system. This is about text notifications/alerts and pushing structured data/logging objects (historical messages).
Most of my push notifications, alerts and data updates are vehicle specific, and most do already have a "pull" command interface as well. For example the output of "power report" will also be sent as the trip efficiency notification after a drive, and "batt status" is used for the battery alert. Just like the standard "stat" command is also used by the "charge" notifications (just with slightly different content on the Twizy).
I think that's a pattern, as most textual status command outputs will be usable for notifications and alerts as well. So generalizing this was the main idea. I don't know if using the commands for scripting and logging does offer an advantage, just added that as another potential benefit.
Automatic notifications on metrics change would require all metrics updates to be organized so that the "trigger" metric is updated after all other metrics required by a message. So the vehicle developer always needs to be aware of this implicit logic. Also a vehicle module may find it necessary to send a notification independant of the value change for the "trigger" metric.
Independant of the signal source (event/metric), transmissions of any type cannot be done synchronously in any listener, both because it would be executed in the sender context and because any transmission channel needs to cope with temporary connection losses, so needs to queue the data anyway. So the "send" event listeners should always just fetch and queue the message during event handling. I.e.
ovms_server_v2 has a BufferedShell object as the transmit queue anyway
RT vehicle module sends a "xrt log RT-BAT-C" send.data event
ovms_server_v2 calls the "xrt log RT-BAT-C" command
The command executes and populates BufferedShell with textual data.
ovms_server_v2 flags internally for pending transmission & returns from signal handling
If many such channels turn out to exist in parallel, a message broker may step in place for the queueing. The LogBuffers class already has a producer/consumer pattern, so the broker could manage the buffers for all channels.
The major problem I see with sending the data with the signal is a conflict with the verbosity concept: at the time of the signal creation, the sender is not aware of the verbosity required for the individual listeners. So the sender would need to fill a message structure containing variants for all verbosity levels for any notification, which may be unnecessary overhead for most cases.
That is, unless we drop the verbosity concept for unsolicited messages. It may be irrelevant for notifications and alerts, it sure is for data? An SMS channel for example can use just as much characters as fit, as notifications and alerts will normally contain the most important info at the beginning.
The minor problem I see is again the additional memory needed -- as the channels will need to buffer the message anyway, the sender can write directly to the buffer. That's minor, the memory would just be needed for the signal transport.
Regards,
Michael
Am 05.11.2017 um 13:48 schrieb Mark Webb-Johnson:
</pre>
<blockquote type="cite">
<pre wrap="">For send.info <a class="moz-txt-link-rfc2396E" href="http://send.info/" moz-do-not-send="true"><http://send.info/></a>, I think it would be much better if the ovms_server_v* code could work out for itself what needs to be sent. Have a look at MetricModified() that I've just committed, as a starting point. (Sorry, I'd started work on that last night, but hadn't committed yet. Done now.) Suggestion is to put the logic on ovms_server_v* MetricModified() rather than the individual vehicle modules. Things like if the car is turned on we should notify the apps, are universal.
If there are cases where this is vehicle specific, then the send.info <a class="moz-txt-link-rfc2396E" href="http://send.info/" moz-do-not-send="true"><http://send.info/></a> mechanism you suggest is ok; but I still think it better we don't do this. Remember ovms_server_v3 is going to work differently (individual metrics, rather than groups). So, for vehicle specific cases (in particular for metrics not in metrics_standard.h) this approach is fine, but for standard metrics I suggest we use MetricModified() in ovms_server_v2.
I like the idea of using events for send.alert and send.data. But not sure of the purpose of a command feedback for this. From what I can see, this means (for example):
RT vehicle module sends a "xrt log RT-BAT-C" send.data event.
ovms_server_v2 receives that event, allocates a BufferedShell object, calls the "xrt log RT-BAT-C" command.
The command executes and populates BufferedShell with textual data.
The command returns the data to ovms_server_v2.
ovms_server_v2 retrieves the textual data from BufferedShell, sends it on it's way, then frees the BufferedShell object.
The event signal returns back to RT vehicle module.
All the above is happening on the stack of the RT vehicle module (remember that commands are expensive, memory wise).
My question is what does this give us, vs:
RT vehicle module builds the textual data to send. Then signals event send.data, passing that const char* as the parameter.
ovms_server_v2 receives the send.data event, send the const char* data on it's way, then returns.
The RT vehicle module gets back control, and completes.
The only advantage for using commands, that I see, is being able to test this easily as a developer. But given the overhead, I can't see the benefit outweighing the overhead.
We could, of course, have commands for things like PUSH notifications, that could be used by scripts. Or alternatively, the script could just raise the event itself.
Why not just send the data with the signal? What is the advantage of the command callback arrangement?
Regards, Mark
</pre>
<blockquote type="cite">
<pre wrap="">On 5 Nov 2017, at 5:39 PM, Michael Balzer <<a class="moz-txt-link-abbreviated" href="mailto:dexter@expeedo.de" moz-do-not-send="true">dexter@expeedo.de</a> <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><mailto:dexter@expeedo.de></a>> wrote:
Some thoughts about how to implement vehicle notifications and data logging. Please check & comment.
Concept:
- handle all unsolicited transmissions from vehicles by commands
- vehicle module signals event if a transmission is due
- ... with event data = command line to generate output
- listeners retrieve the transmission content by executing the command
- queueing for async delivery is done by the listeners (i.e. through BufferedShell)
→ any shell command can be used for transmissions
→ all transmissions can also be generated on the shell / by script
→ all transmission output can be sent to any channel
→ data logging on SD can be done transparently by the system
Interactive consoles can output unsolicited transmissions by event type
according to the current log level.
Events:
"send.info <a class="moz-txt-link-rfc2396E" href="http://send.info/" moz-do-not-send="true"><http://send.info/></a>" → send text notification
"send.alert" → send text alert
"send.data" → send data record (CSV, content part of v2 MP)
Scheme:
MyEvents.SignalEvent("send.<type>", "command [args]");
Examples:
MyEvents.SignalEvent("send.info <a class="moz-txt-link-rfc2396E" href="http://send.info/" moz-do-not-send="true"><http://send.info/></a>", "stat");
MyEvents.SignalEvent("send.alert", "xrt batt status");
MyEvents.SignalEvent("send.data", "xrt log RT-BAT-C");
→ command output:
H,RT-BAT-C,4,86400,1,1,4025,3675,4045,5,14,14,19,1
Regards,
Michael
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a> <a class="moz-txt-link-rfc2396E" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true"><mailto:OvmsDev@lists.teslaclub.hk></a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a> <a class="moz-txt-link-rfc2396E" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true"><http://lists.teslaclub.hk/mailman/listinfo/ovmsdev></a>
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a> <a class="moz-txt-link-rfc2396E" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true"><mailto:OvmsDev@lists.teslaclub.hk></a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a> <a class="moz-txt-link-rfc2396E" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true"><http://lists.teslaclub.hk/mailman/listinfo/ovmsdev></a>
</pre>
</blockquote>
<pre wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a> <a class="moz-txt-link-rfc2396E" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true"><mailto:OvmsDev@lists.teslaclub.hk></a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a> <a class="moz-txt-link-rfc2396E" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true"><http://lists.teslaclub.hk/mailman/listinfo/ovmsdev></a>
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a> <a class="moz-txt-link-rfc2396E" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true"><mailto:OvmsDev@lists.teslaclub.hk></a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a> <a class="moz-txt-link-rfc2396E" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true"><http://lists.teslaclub.hk/mailman/listinfo/ovmsdev></a>
</pre>
</blockquote>
<pre wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a> <a class="moz-txt-link-rfc2396E" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true"><mailto:OvmsDev@lists.teslaclub.hk></a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a> <a class="moz-txt-link-rfc2396E" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true"><http://lists.teslaclub.hk/mailman/listinfo/ovmsdev></a>
</pre>
</blockquote>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a> <a class="moz-txt-link-rfc2396E" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true"><mailto:OvmsDev@lists.teslaclub.hk></a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre wrap="">></pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk" moz-do-not-send="true">OvmsDev@lists.teslaclub.hk</a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.teslaclub.hk">OvmsDev@lists.teslaclub.hk</a>
<a class="moz-txt-link-freetext" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</body>
</html>