<html><head></head><body><div style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;"><div></div>
<div>Hi guys, </div><div><br></div><div>I have been using ovms units for logging of trip data and have had some non-consistent experiences when the ovms box is loosing data connections. </div><div>Are there any provisions for synchronization of messages that have not been delivered due to connectivity problems?</div><div><br></div><div>Another even more interesting situation is if there is no SIM in the module, so you go for a drive and when you come home the ovms uses the wifi to sync all the messages and data. </div><div>Or is this already handled? I am not sure I understand all the stuff that some of you guys write.</div><div><br></div><div>Nikolay</div><div><br></div>
<div id="yahoo_quoted_1247077396" class="yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Saturday, November 18, 2017, 4:44:58 PM GMT+1, Michael Balzer <dexter@expeedo.de> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="yiv0795859337"><div>
Hi Mark,<br clear="none">
<br clear="none">
apologies as well, I'm quite low on spare time currently :-/<br clear="none">
<br clear="none">
I agree it's better to implement the message broker directly, as
adding it later on would need changes to all listeners.<br clear="none">
<br clear="none">
ACK on your thoughts, except a) the fixed alert texts and b) the
message producing scheme.<br clear="none">
<br clear="none">
a) Providing default messages for standard alert types is good, but
replacing them with custom message text should be possible. I.e. the
charge alerts need to be specific for the Twizy. (When using
commands for message generation this would automatically be handled
by the vehicle specific "stat" command implementation)<br clear="none">
<br clear="none">
b) The point I don't see in the scheme of vehicles sending messages
to the broker instead of "get my message" events/calls: the car does
not know about the verbosity levels needed by the broker's
subscribers. That's why I think it's better to let the broker fetch
the messages than to have the producer send them to the broker.<br clear="none">
<br clear="none">
Example: the car raises a custom alert. The car implementation wants
to add detail info to the basic alert if possible.<br clear="none">
<br clear="none">
An SMS channel has subscribed to the broker with verbosity=minimal
(140), and for example a shell and an ssh session (thanks, Steven!)
each have subscribed with verbosity=normal (1024). So the broker
knows it will need messages of both 140 and 1024 char verbosity.<br clear="none">
<br clear="none">
The car now sends the "get my alert" event (or call to the broker).
The broker in turn requests the alert message from the car twice,
one variant tailored for verbosity=140 and another for
verbosity=1024. It then queues & delivers the message variants
to the listeners according to their verbosity levels and finally
discards them.<br clear="none">
<br clear="none">
As the verbosity is already part of the ovms_command callback API,
all command implementations should take care of the message length
limitation anyway, so can easily be used for alert and notification
generation as well.<br clear="none">
<br clear="none">
<br clear="none">
Regards,<br clear="none">
Michael<br clear="none">
<br clear="none">
<br clear="none">
<div class="yiv0795859337moz-cite-prefix">Am 10.11.2017 um 07:07 schrieb Mark
Webb-Johnson:<br clear="none">
</div>
<blockquote type="cite">
</blockquote></div><div>
Apologies for the delay in getting back to you on this.
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">We’re up to the stage now where we have to determine
how best to implement this. The rest of the ovms_server_v2 stuff
is pretty much done.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">So, let’s break the requirements down:</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">
<ol class="yiv0795859337MailOutline"><li class="yiv0795859337">Metrics -> v2 messages</li><li class="yiv0795859337">Push notifications (car -> server -> apps)</li><li class="yiv0795859337">Error notifications (car -> server -> apps)</li><li class="yiv0795859337">Historical data submissions (car -> server,
and presumably later server -> apps)</li></ol>
</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">Let’s go through them one by one.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337"><u class="yiv0795859337"><b class="yiv0795859337">1. Metrics -> v2 messages</b></u></div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">I think this works pretty much ok at the moment. The
metrics have modified flags, and ovms_server_v2 (and others) can
get notified whenever a metric is modified. We can either issue
the update immediately, or delay it for later transmission.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">The way we do it at the moment in ovms_server_v2 is
to delay until the next poll. I think that should be ok. 99% of
the time, if a bunch of metrics are updated, they’ll all make it
into the same update. If not, it shouldn’t do too much harm
(just client app screen will get two updates, rather than one).
For ovms_server_v3, most likely these are going to go out
immediately, via MQTT, as they are changed.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337"><u class="yiv0795859337"><b class="yiv0795859337">2. Push notifications (car
-> server -> apps)</b></u></div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">In v2, these are fixed strings sent
by net_msg_alert() - but originally raised by
net_req_notification(). It is a simple bit - set it and at some
time later the alert will be sent then the bit cleared. It seems
that the twizy vehicle also manually sends push alerts as well.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">For v3, I think we need something similar. The
ability to queue a notification message (with a timeout would be
helpful) and have it delivered as a push notification. I suggest
we have standardised message types, plus the ability to send any
custom string. It would be good to de-duplicate based on the
message type, and rate limit.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337"><u class="yiv0795859337"><b class="yiv0795859337">3. Error notifications (car
-> server -> apps)</b></u></div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">In v2, these are (code,data) sent
by net_msg_erroralert() - but originally raised by
net_req_notification_error(). There is some logic (that doesn’t
work very well) to try to stop duplicate (repetitive) alerts,
but essentially it is implemented by setting a code, and at some
time later the error notification will be sent then the code
cleared.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">For v3, I think we need something similar. I think
this can be merged in with push notifications, as it is pretty
similar. It is also interesting how we can deal with OBDII DTCs,
as they seem to have a similar requirement.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337"><u class="yiv0795859337"><b class="yiv0795859337">4. Historical data
submissions (car -> server, and presumably later server
-> apps)</b></u></div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">In v2, the logging.{h,c} module provides functions
to put a log message into the queue, and then is polled by the
net framework to send outstanding log messages. It seems that
the Twizzy vehicle also manually send historical data
submissions as well.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">For v3, I think we need to make a generic module to
handle this. Something that can queue a message, and have it
reliably delivered at some later point in time.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337"><u class="yiv0795859337"><b class="yiv0795859337">v3 approach</b></u></div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">I think that we can do something along the lines of
what you are suggesting, but that all three should fit into a
standardised structure and queueing mechanism.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">Looking through the way things were done in v2, and
what we now know we need, I suggest that overloading the events
system is not necessary. On the producer side, we can simply
have commands to raise these alerts/error/historical-data. On
the custom consumer side, we expose it to scripts very simply.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">However, reviewing the v2 code, we do need something
more sophisticated. Some examples:</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">
<ul class="yiv0795859337MailOutline"><li class="yiv0795859337">We really need to de-duplicate and/or rate-limit
notifications. Particularly Error notifications. Sometimes
the car goes crazy and outputs multiple alerts (not always
the same code), and those get delivered as a stream of
annoying push notifications. We need something like syslog
in unix - where rather than giving you 1,000 lines saying
the same thing, it says something like “1,000 copies of X
were raised”.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
</li><li class="yiv0795859337">It would be good to have timeouts, and possibly
time stamps. Sometimes you have no connectivity, then five
hours later when it comes back you get an alert that
charging was interrupted. It would be good to have a timeout
on a message (don’t bother to send it if it can’t be sent in
the next 5 minutes), and timestamps (so if message is not
sent in realtime, when it is actually sent, it has a
timestamp added to show when).<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
</li><li class="yiv0795859337">We need more reliable delivery. Historical data
has ACKs for this. I think for PUSH, we can do something
similar, using v2 protocol PINGs. The car can send the push
notification, followed by a v2 ping. If it gets an ack, it
can assume the notification message was received.</li></ul>
</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">The ESP32 platform has plenty of power for this. We
just need to be careful with RAM usage. Perhaps we can overflow
to flash, if necessary? Anyway, that can be hidden inside the
specific implementation.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">There are two possible implementations of this, once
a producer send in a message.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">
<ol class="yiv0795859337MailOutline"><li class="yiv0795859337">Each subscriber gets a copy of that message, and
is responsible for all the above maintenance as well as
queueing for delivery.</li><li class="yiv0795859337">The message is maintained and queued centrally.
Subscribers then receive notifications of changes.</li></ol>
</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">Implementation #2 is my preference. Having a
centralised middle queue between the publishers and subscribers
means we only need to queue once. Rather than each subscriber
having to handle its own queueing. Given the RAM constraints, I
really think this is the best approach. We’ve already done
something similar for metrics, and that is very frugal on RAM
for the modification monitors.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">Regarding the method of gathering the data to send,
I think we can be flexible. Support fixed IDs, static text, and
callback commands (just a type+data storage structure). When the
subscriber calls the method to retrieve the actual data, the
centralised approach retrieves the data and returns it to that
specific subscriber. Some examples:</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">
<ul class="yiv0795859337MailOutline"><li class="yiv0795859337">An alert that charging has been interrupted. We
just want to queue it using a pre-defined fixed type
(associated with a fixed message). We don’t need command
callback, or anything more complex.</li><li class="yiv0795859337">An error alert. Similar to the previous one, we
just want to queue the alert code, data.</li><li class="yiv0795859337">A piece of historical data. In most cases, the
data is a textual string that is just queued when it is
raised (for example, a drive log).</li><li class="yiv0795859337">A status update. In this case, the command
callback makes the most sense.</li></ul>
</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">Does that meet your requirements? I don’t mind
taking on the implementation of the central queue code. The
ovms_server_v2 should be simple.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">Regards, Mark.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
<div>
<blockquote class="yiv0795859337" type="cite">
<div class="yiv0795859337">On 5 Nov 2017, at 10:23 PM, Michael Balzer
<<a rel="nofollow" shape="rect" class="yiv0795859337" ymailto="mailto:dexter@expeedo.de" target="_blank" href="mailto:dexter@expeedo.de">dexter@expeedo.de</a>> wrote:</div>
<br clear="none" class="yiv0795859337Apple-interchange-newline">
<div class="yiv0795859337">
</div></blockquote></div></div></div><div><div class="yiv0795859337"> Sorry, I
missed to explain my previous thoughts.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
First of all, this is not about atomic updates of single
metrics, those can be done completely independant of the
vehicle implementation through the metrics system. This
is about text notifications/alerts and pushing
structured data/logging objects (historical messages).<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Most of my push notifications, alerts and data updates
are vehicle specific, and most do already have a "pull"
command interface as well. For example the output of
"power report" will also be sent as the trip efficiency
notification after a drive, and "batt status" is used
for the battery alert. Just like the standard "stat"
command is also used by the "charge" notifications (just
with slightly different content on the Twizy).<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
I think that's a pattern, as most textual status command
outputs will be usable for notifications and alerts as
well. So generalizing this was the main idea. I don't
know if using the commands for scripting and logging
does offer an advantage, just added that as another
potential benefit.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Automatic notifications on metrics change would require
all metrics updates to be organized so that the
"trigger" metric is updated after all other metrics
required by a message. So the vehicle developer always
needs to be aware of this implicit logic. Also a vehicle
module may find it necessary to send a notification
independant of the value change for the "trigger"
metric.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Independant of the signal source (event/metric),
transmissions of any type cannot be done synchronously
in any listener, both because it would be executed in
the sender context and because any transmission channel
needs to cope with temporary connection losses, so needs
to queue the data anyway. So the "send" event listeners
should always just fetch and queue the message during
event handling. I.e.<br clear="none" class="yiv0795859337">
<div class="yiv0795859337">
<ul class="yiv0795859337MailOutline"><li class="yiv0795859337">ovms_server_v2 has a BufferedShell
object as the transmit queue anyway<br clear="none" class="yiv0795859337">
</li><li class="yiv0795859337">RT vehicle module sends a “xrt log
RT-BAT-C” send.data event</li><li class="yiv0795859337">ovms_server_v2 calls the “xrt log
RT-BAT-C” command</li><li class="yiv0795859337">The command executes and populates
BufferedShell with textual data.</li><li class="yiv0795859337">ovms_server_v2 flags internally for
pending transmission & returns from signal
handling<br clear="none" class="yiv0795859337">
</li></ul>
</div>
If many such channels turn out to exist in parallel, a
message broker may step in place for the queueing. The
LogBuffers class already has a producer/consumer
pattern, so the broker could manage the buffers for all
channels.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
The major problem I see with sending the data with the
signal is a conflict with the verbosity concept: at the
time of the signal creation, the sender is not aware of
the verbosity required for the individual listeners. So
the sender would need to fill a message structure
containing variants for all verbosity levels for any
notification, which may be unnecessary overhead for most
cases.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
That is, unless we drop the verbosity concept for
unsolicited messages. It may be irrelevant for
notifications and alerts, it sure is for data? An SMS
channel for example can use just as much characters as
fit, as notifications and alerts will normally contain
the most important info at the beginning.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
The minor problem I see is again the additional memory
needed -- as the channels will need to buffer the
message anyway, the sender can write directly to the
buffer. That's minor, the memory would just be needed
for the signal transport.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Regards,<br clear="none" class="yiv0795859337">
Michael<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
<div class="yiv0795859337moz-cite-prefix">Am 05.11.2017 um 13:48
schrieb Mark Webb-Johnson:<br clear="none" class="yiv0795859337">
</div>
<blockquote class="yiv0795859337" type="cite">
</blockquote></div></div><div>
For <a rel="nofollow" shape="rect" class="yiv0795859337" target="_blank" href="http://send.info/">send.info</a>, I think it
would be much better if the ovms_server_v* code could
work out for itself what needs to be sent. Have a look
at MetricModified() that I’ve just committed, as a
starting point. (Sorry, I’d started work on that last
night, but hadn’t committed yet. Done now.) Suggestion
is to put the logic on ovms_server_v* MetricModified()
rather than the individual vehicle modules. Things
like if the car is turned on we should notify the
apps, are universal.
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">If there are cases where this is vehicle
specific, then the <a rel="nofollow" shape="rect" class="yiv0795859337" target="_blank" href="http://send.info/">send.info</a>
mechanism you suggest is ok; but I still think it
better we don’t do this. Remember ovms_server_v3 is
going to work differently (individual metrics,
rather than groups). So, for vehicle specific cases
(in particular for metrics not in
metrics_standard.h) this approach is fine, but for
standard metrics I suggest we use MetricModified()
in ovms_server_v2.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">I like the idea of using events for
send.alert and send.data. But not sure of the
purpose of a command feedback for this. From what I
can see, this means (for example):</div>
<div class="yiv0795859337">
<ul class="yiv0795859337MailOutline"><li class="yiv0795859337">RT vehicle module sends a “xrt log
RT-BAT-C” send.data event.</li><li class="yiv0795859337">ovms_server_v2 receives that event,
allocates a BufferedShell object, calls the “xrt
log RT-BAT-C” command.</li><li class="yiv0795859337">The command executes and populates
BufferedShell with textual data.</li><li class="yiv0795859337">The command returns the data to
ovms_server_v2.</li><li class="yiv0795859337">ovms_server_v2 retrieves the textual
data from BufferedShell, sends it on it’s way,
then frees the BufferedShell object.</li><li class="yiv0795859337">The event signal returns back to RT
vehicle module.</li></ul>
</div>
<div class="yiv0795859337">All the above is happening on the stack
of the RT vehicle module (remember that commands are
expensive, memory wise).</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">My question is what does this give us,
vs:</div>
<div class="yiv0795859337">
<ul class="yiv0795859337MailOutline"><li class="yiv0795859337">RT vehicle module builds the textual
data to send. Then signals event send.data,
passing that const char* as the parameter.</li><li class="yiv0795859337">ovms_server_v2 receives the send.data
event, send the const char* data on it’s way,
then returns.</li><li class="yiv0795859337">The RT vehicle module gets back
control, and completes.</li></ul>
</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">The only advantage for using commands,
that I see, is being able to test this easily as a
developer. But given the overhead, I can’t see the
benefit outweighing the overhead.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">We could, of course, have commands for
things like PUSH notifications, that could be used
by scripts. Or alternatively, the script could just
raise the event itself.</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
</div>
<div class="yiv0795859337">Why not just send the data with the
signal? What is the advantage of the command
callback arrangement?</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
<div class="yiv0795859337">Regards, Mark</div>
<div class="yiv0795859337"><br clear="none" class="yiv0795859337">
<div class="yiv0795859337">
<blockquote class="yiv0795859337" type="cite">
<div class="yiv0795859337">On 5 Nov 2017, at 5:39 PM,
Michael Balzer <<a rel="nofollow" shape="rect" class="yiv0795859337" ymailto="mailto:dexter@expeedo.de" target="_blank" href="mailto:dexter@expeedo.de">dexter@expeedo.de</a>>
wrote:</div>
<br clear="none" class="yiv0795859337Apple-interchange-newline">
<div class="yiv0795859337">
<div class="yiv0795859337">Some thoughts about how to
implement vehicle notifications and data
logging. Please check & comment.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Concept:<br clear="none" class="yiv0795859337">
- handle all unsolicited transmissions
from vehicles by commands<br clear="none" class="yiv0795859337">
- vehicle module signals event if a
transmission is due<br clear="none" class="yiv0795859337">
- … with event data = command line to
generate output<br clear="none" class="yiv0795859337">
- listeners retrieve the transmission
content by executing the command<br clear="none" class="yiv0795859337">
- queueing for async delivery is done by
the listeners (i.e. through BufferedShell)<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
→ any shell command can be used for
transmissions<br clear="none" class="yiv0795859337">
→ all transmissions can also be generated
on the shell / by script<br clear="none" class="yiv0795859337">
→ all transmission output can be sent to
any channel<br clear="none" class="yiv0795859337">
→ data logging on SD can be done
transparently by the system<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Interactive consoles can output
unsolicited transmissions by event type<br clear="none" class="yiv0795859337">
according to the current log level.<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Events:<br clear="none" class="yiv0795859337">
"<a rel="nofollow" shape="rect" class="yiv0795859337" target="_blank" href="http://send.info/">send.info</a>"
→ send text notification<br clear="none" class="yiv0795859337">
"send.alert" → send text alert<br clear="none" class="yiv0795859337">
"send.data" → send data record (CSV,
content part of v2 MP)<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Scheme:<br clear="none" class="yiv0795859337">
MyEvents.SignalEvent("send.<type>",
"command [args]");<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Examples:<br clear="none" class="yiv0795859337">
MyEvents.SignalEvent("<a rel="nofollow" shape="rect" class="yiv0795859337" target="_blank" href="http://send.info/">send.info</a>",
"stat");<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
MyEvents.SignalEvent("send.alert", "xrt
batt status");<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
MyEvents.SignalEvent("send.data", "xrt
log RT-BAT-C");<br clear="none" class="yiv0795859337">
→ command output:<br clear="none" class="yiv0795859337">
H,RT-BAT-C,4,86400,1,1,4025,3675,4045,5,14,14,19,1<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
Regards,<br clear="none" class="yiv0795859337">
Michael<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
-- <br clear="none" class="yiv0795859337">
Michael Balzer * Helkenberger Weg 9 *
D-58256 Ennepetal<br clear="none" class="yiv0795859337">
Fon 02333 / 833 5735 * Handy 0176 / 206
989 26<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
_______________________________________________<br clear="none" class="yiv0795859337">
OvmsDev mailing list<br clear="none" class="yiv0795859337">
<a rel="nofollow" shape="rect" class="yiv0795859337" ymailto="mailto:OvmsDev@lists.teslaclub.hk" target="_blank" href="mailto:OvmsDev@lists.teslaclub.hk">OvmsDev@lists.teslaclub.hk</a><br clear="none" class="yiv0795859337">
<a rel="nofollow" shape="rect" class="yiv0795859337moz-txt-link-freetext" target="_blank" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a><br clear="none" class="yiv0795859337">
</div>
</div>
</blockquote>
</div>
<br clear="none" class="yiv0795859337">
</div>
</div>
<br clear="none" class="yiv0795859337">
<fieldset class="yiv0795859337mimeAttachmentHeader"></fieldset>
<br clear="none" class="yiv0795859337">
<pre class="yiv0795859337">_______________________________________________
OvmsDev mailing list
<a rel="nofollow" shape="rect" class="yiv0795859337moz-txt-link-abbreviated" ymailto="mailto:OvmsDev@lists.teslaclub.hk" target="_blank" href="mailto:OvmsDev@lists.teslaclub.hk">OvmsDev@lists.teslaclub.hk</a>
<a rel="nofollow" shape="rect" class="yiv0795859337moz-txt-link-freetext" target="_blank" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
<div class="yiv0795859337yqt3424795164" id="yiv0795859337yqtfd50058"><br clear="none" class="yiv0795859337">
<pre class="yiv0795859337moz-signature">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
_______________________________________________<br clear="none" class="yiv0795859337">
OvmsDev mailing list<br clear="none" class="yiv0795859337">
<a rel="nofollow" shape="rect" class="yiv0795859337" ymailto="mailto:OvmsDev@lists.teslaclub.hk" target="_blank" href="mailto:OvmsDev@lists.teslaclub.hk">OvmsDev@lists.teslaclub.hk</a><br clear="none" class="yiv0795859337">
<a rel="nofollow" shape="rect" class="yiv0795859337moz-txt-link-freetext" target="_blank" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a><br clear="none" class="yiv0795859337">
<br clear="none" class="yiv0795859337">
<br clear="none">
<fieldset class="yiv0795859337mimeAttachmentHeader"></fieldset>
<br clear="none">
<pre>_______________________________________________
OvmsDev mailing list
<a rel="nofollow" shape="rect" class="yiv0795859337moz-txt-link-abbreviated" ymailto="mailto:OvmsDev@lists.teslaclub.hk" target="_blank" href="mailto:OvmsDev@lists.teslaclub.hk">OvmsDev@lists.teslaclub.hk</a>
<a rel="nofollow" shape="rect" class="yiv0795859337moz-txt-link-freetext" target="_blank" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a>
</pre>
<br clear="none">
<pre class="yiv0795859337moz-signature">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div></div></div><div class="yqt3424795164" id="yqtfd04996">_______________________________________________<br clear="none">OvmsDev mailing list<br clear="none"><a shape="rect" ymailto="mailto:OvmsDev@lists.teslaclub.hk" href="mailto:OvmsDev@lists.teslaclub.hk">OvmsDev@lists.teslaclub.hk</a><br clear="none"><a shape="rect" href="http://lists.teslaclub.hk/mailman/listinfo/ovmsdev" target="_blank">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev</a><br clear="none"></div></div>
</div>
</div></div></body></html>