<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<blockquote type="cite">
<div>I'm not sure whether the 'max' should be the maximum of the
smoothed value.. or the maximum of the raw value.</div>
</blockquote>
<br>
It should normally be the maximum of the raw value I think, the
maximum of the smoothed value cannot tell about how bad the
processing of an ID can become.<br>
<br>
The naming in the table is a bit confusing I think. (besides: I've
never seen "ave" as the abbreviation for average)<br>
<br>
If I understand you correctly, "time ms per s" is the time share in
per mille, so something in that direction would be more clear, and
"length ms" would then be "time [ms]".<br>
<br>
The totals for all averages in the table foot would also be nice.<br>
<br>
Maybe "Ave" (or avg?) also should be placed on the left, as the
"peak" label now suggests being the peak of the average.<br>
<br>
Btw, keep in mind, not all "edge" users / testers are developers
(e.g. the Twizy driver I'm in contact with), collecting stats
feedback for vehicles from testers should be straight forward. Maybe
add a data/history record, sent automatically on every drive/charge
stop when the poller tracing is on?<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 18.05.24 um 02:28 schrieb Michael
Geddes:<br>
</div>
<blockquote type="cite"
cite="mid:CAH0p7u+aFuxS38qpJVyGNrkLtKgZ4Tb3bqwP_UsTBBUf5LmY0A@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>You did say max/pead value. I also halved the N for both.</div>
<div>I'm not sure whether the 'max' should be the maximum of the
smoothed value.. or the maximum of the raw value.</div>
<div>This is currently the raw-value maximum. </div>
<div>The problem is that the middle column is the maximum of the
{{sum over 10s} / (10*1000,000)</div>
<div>I could easily change the 'period' to 1s and see how that
goes.. was just trying to reduce the larger calculations. </div>
<div><br>
</div>
<div><br>
</div>
<font face="monospace">Usage: poller
[pause|resume|status|times|trace]</font>
<div><font face="monospace"><br>
OVMS# poller time status<br>
Poller timing is: on<br>
Type | Count | Ave time | Ave length<br>
| per s | ms per s | ms <br>
-------------+----------+-----------+-----------<br>
Poll:PRI | 1.00| 0.559| 0.543<br>
peak | | 0.663| 1.528<br>
-------------+----------+-----------+-----------<br>
Poll:SRX | 0.08| 0.009| 0.038<br>
peak | | 0.068| 0.146<br>
-------------+----------+-----------+-----------<br>
CAN1 RX[778] | 0.11| 0.061| 0.280<br>
peak | | 0.458| 1.046<br>
-------------+----------+-----------+-----------<br>
CAN1 RX[7a8] | 0.04| 0.024| 0.124<br>
peak | | 0.160| 0.615<br>
-------------+----------+-----------+-----------<br>
CAN1 TX[770] | 0.05| 0.004| 0.016<br>
peak | | 0.022| 0.102<br>
-------------+----------+-----------+-----------<br>
CAN1 TX[7a0] | 0.02| 0.002| 0.011<br>
peak | | 0.010| 0.098<br>
-------------+----------+-----------+-----------<br>
CAN1 TX[7b3] | 0.01| 0.001| 0.006<br>
peak | | 0.000| 0.099<br>
-------------+----------+-----------+-----------<br>
CAN1 TX[7e2] | 0.02| 0.002| 0.011<br>
peak | | 0.010| 0.099<br>
-------------+----------+-----------+-----------<br>
CAN1 TX[7e4] | 0.08| 0.008| 0.048<br>
peak | | 0.049| 0.107<br>
-------------+----------+-----------+-----------<br>
Cmd:State | 0.00| 0.000| 0.005<br>
peak | | 0.000| 0.094</font><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 17 May 2024 at 15:26,
Michael Geddes <<a
href="mailto:frog@bunyip.wheelycreek.net"
moz-do-not-send="true" class="moz-txt-link-freetext">frog@bunyip.wheelycreek.net</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>This is what I have now.</div>
<div>The one on the end is the one MIchael B was after
using an N of 32. (up for discussion).</div>
<div><br>
</div>
<div>The middle is the time spent in that even t per
second. It accumulates times (in microseconds), and then
every 10s it stores it as smoothed (N=16) value.</div>
<div>The Count is similar (except that we store a value of
'100' as 1 event so it can be still integers and has 2
decimal places).</div>
<div><br>
</div>
<div>Every received poll does a 64bit difference to 32bit
(for the elapsed time) and 64bit comparison (for
end-of-period).</div>
<div>It also does 1x 32bit smoothing and 2x 32bit adds.</div>
<div><br>
</div>
<div>Then at the end of a 10s period, it will do a 64bit add
to get the next end-of-period value, as well as the 2x
32bit smoothing calcs.</div>
<div><br>
</div>
<div>This is from the Ioniq 5 so not any big values yet.
You can certainly see how insignificant the TX callbacks
are.</div>
<div>I'll leave it on for when the car is moving and gets
some faster polling.</div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">OVMS# poll time status<br>
Poller timing is: on<br>
Type | Count | Ave time | Ave Length<br>
| per s | ms per s | ms <br>
-------------+----------+-----------+-----------<br>
Poll:PRI | 1.00| 0.540| 0.539<br>
Poll:SRX | 0.03| 0.004| 0.017<br>
CAN1 RX[778] | 0.06| 0.042| 0.175<br>
CAN1 TX[770] | 0.04| 0.002| 0.008<br>
Cmd:State | 0.01| 0.001| 0.005</font><br>
</div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">----------------------8<--------------------------------</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">Nice smoothing class (forces N
as a power of 2):</font></div>
<div> constexpr unsigned floorlog2(unsigned x)<br>
{<br>
return x == 1 ? 0 : 1+floorlog2(x >> 1);<br>
}<br>
/* Maintain a smoothed average using shifts for
division.<br>
* T should be an integer type<br>
* N needs to be a power of 2<br>
*/<br>
template <typename T, unsigned N><br>
class average_util_t<br>
{<br>
private:<br>
T m_ave;<br>
public:<br>
average_util_t() : m_ave(0) {}<br>
static const uint8_t _BITS = floorlog2(N);<br>
void add( T val)<br>
{<br>
static_assert(N == (1 << _BITS), "N must be
a power of 2");<br>
m_ave = (((N-1) * m_ave) + val) >> _BITS;<br>
}<br>
T get() { return m_ave; }<br>
operator T() { return m_ave; }<br>
};<br>
</div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace"><br>
</font></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 16 May 2024 at
10:29, Michael Geddes <<a
href="mailto:frog@bunyip.wheelycreek.net"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">frog@bunyip.wheelycreek.net</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div>Thanks Michael,</div>
<div dir="ltr"><br>
</div>
My calculations give me ((2^32)-1) / (1000*1000*3600)
= only 1.2 hours of processing time in 32bit. The
initial subtraction is 64bit anyway and I can't see a
further 64bit addition being a problem. I have the
calculations being performed in doubles at print-out
where performance is not really an issue anyway.
(Though apparently doing 64 bit division is worse than
floating point).
<div><br>
<div>In addition</div>
<div>* I currently have this being able to be turned
on and off and reset manually (only do it when
required).</div>
<div><br>
</div>
<div>* For the lower volume commands, the smoothed
average is not going to be useful - the count is
more interesting for different reasons.</div>
<div><br>
</div>
<div>* The total time is quite useful. Ie a high
average time doesn't matter if the count is low.
The things that are affecting performance are
stuff with high total time. Stuff which is
happening 100 times a second needs to be a much
lower average than once a second.</div>
<div>
<div>
<div><br>
</div>
<div>* A measure like 'time per minute/second'
and possibly count per minute/seconds as a
smoothed average would potentially be more
useful. (or in addition?)</div>
<div>I think we could do _that_ in a reasonably
efficient manner using a 64 bit 'last measured
time', a 32 bit accumulated value and the
stored 32 bit rolling average. </div>
<div>It would boils down to some iterative
(integer) sums and multiplications plus a
divide by n ^ (time periods passed) - which
is a shift - and which can be optimised to '0'
if 'time-periods-passed' is more than
32/(bits-per-n) - effectively limiting the
number of iterations.</div>
<div>The one issue I can see is that we need to
calculate 'number of time-periods passed'
which is a 64 bit subtraction followed by a 32
bit division (not optimisable to a simple
shift).</div>
<div><br>
</div>
</div>
</div>
<div>* I'm also happy to keep a rolling (32bit)
average time.<br>
</div>
<div> Even if you assume averages in the 100ms,
32bit is going to happily support an N of 64 or
even 128.</div>
<div>
<div>
<div> Am I right in thinking that the choice
of N is highly dependent on frequency. For
things happening 100 times per second, you
might want an N like 128.. where things
happening once per</div>
<div> second, you might want an N of 4 or 8.
The other things we keep track of in this
manner we have a better idea of the frequency
of the thing.<br>
</div>
</div>
<div><br>
</div>
</div>
<div>How about we have (per record type):</div>
<div> * total count (since last reset?) (32 bit)</div>
<div> * smoothed average of time per instance (32
bit)</div>
<div><br>
</div>
<div> * ?xx? total accumulated time since last
reset (64bit) ?? <-- with the below stats this
is much less useful</div>
<div><br>
</div>
<div> * last-measured-time (64 bit) </div>
<div> * accumulated count since last time-period
(16bit - but maybe 32bit anyway for byte
alignment?)</div>
<div> * smoothed average of count per time-period
(32bit)<br>
</div>
<div> * accumulated time since last time-period
(32bit)</div>
<div> * smoothed average of time per time-period
(32bit)</div>
<div><br>
</div>
<div>It's possible to keep the </div>
<div><br>
</div>
<div>Is this going to be too much per record type?
The number of 'records' we are keeping is quite
low (so 10 to 20 maybe) - so it's not a huge
memory burden.<br>
</div>
</div>
<div><br>
</div>
<div>Thoughts?</div>
<div><br>
</div>
</div>
//.ichael<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 16 May 2024
at 03:09, Michael Balzer via OvmsDev <<a
href="mailto:ovmsdev@lists.openvehicles.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ovmsdev@lists.openvehicles.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> esp_timer_get_time() is the right choice for
precision timing.<br>
<br>
I'd say uint32 is enough though, even if counting
microseconds that can hold a total of more than 71
hours of actual processing time. uint64 has a
significant performance penalty, although I don't
recall the overhead for simple additions.<br>
<br>
Also & more important, the average wouldn't be
my main focus, but the maximum processing time
seen per ID, which seems to be missing in your
draft.<br>
<br>
Second thought on the average… the exact overall
average really has a minor meaning, I'd rather see
the current average, adapting to the current mode
of operation (drive/charge/…). I suggest feeding
the measurements to a low pass filter to get the
smoothed average of the last n measurements.
Pattern:<br>
<br>
runavg = ((N-1) * runavg + newval) / N<br>
<br>
By using a low power of 2 for N (e.g. 8 or 16),
you can replace the division by a simple bit
shift, and have enough headroom to use 32 bit
integers.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div>Am 15.05.24 um 06:51 schrieb Michael Geddes
via OvmsDev:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Formatting aside, I have
implemented what I think Michael B was
suggesting. This is a sample run on the Ioniq
5 (which doesn't have unsolicited RX events).
<div><br>
</div>
<div>This uses the call esp_timer_get_time()
got get a 64bit <b>microseconds</b> since
started value - and works out the time to
execute that way. It's looking at absolute
time and not time in the Task - so other
things going on at the same time in other
tasks will have an effect. (The normal tick
count doesn't have nearly enough resolution
to be useful - any other ideas on
measurement?) I've got total accumulated
time displaying in seconds and the average
in milliseconds currently - but I can change
that easy enough.</div>
<div>The cumulative time is stored as uint64_t
which will be plenty, as 32bit wouldn't be
nearly enough.<br>
<div><br>
</div>
<div><font face="monospace">OVMS# <b>poller
time on</b><br>
Poller timing is now on<br>
</font>
<div><br>
</div>
<div><font face="monospace">OVMS# <b>poller
time status</b><br>
Poller timing is: on<br>
Poll [PRI] : n=390
tot=0.2s ave=0.586ms<br>
Poll [SRX] : n=316
tot=0.1s ave=0.196ms<br>
CAN1 RX[0778] : n=382
tot=0.2s ave=0.615ms<br>
CAN1 RX[07a8] : n=48
tot=0.0s ave=0.510ms<br>
CAN1 RX[07bb] : n=162
tot=0.1s ave=0.519ms<br>
CAN1 RX[07ce] : n=33
tot=0.0s ave=0.469ms<br>
CAN1 RX[07ea] : n=408
tot=0.2s ave=0.467ms<br>
CAN1 RX[07ec] : n=486
tot=0.2s ave=0.477ms<br>
CAN3 RX[07df] : n=769
tot=0.2s ave=0.261ms<br>
CAN1 TX[0770] : n=191
tot=0.0s ave=0.054ms<br>
CAN1 TX[07a0] : n=16
tot=0.0s ave=0.047ms<br>
CAN1 TX[07b3] : n=31
tot=0.0s ave=0.069ms<br>
CAN1 TX[07c6] : n=11
tot=0.0s ave=0.044ms<br>
CAN1 TX[07e2] : n=82
tot=0.0s ave=0.067ms<br>
CAN1 TX[07e4] : n=54
tot=0.0s ave=0.044ms<br>
Set State : n=7
tot=0.0s ave=0.104ms<br>
</font><br>
</div>
</div>
<div>This is probably going to be quite
useful in general! The TX call-backs
don't seem to be significant here. (oh, I
should probably implement a reset of the
values too).</div>
</div>
<div><br>
</div>
<div>//.ichael</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, 12
May 2024 at 22:58, Michael Geddes <<a
href="mailto:frog@bunyip.wheelycreek.net"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">frog@bunyip.wheelycreek.net</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Yeah - I certainly wasn't
going to put a hard limit. Just a log
above a certain time, that being said,
the idea of just collecting stats (being
able to turn it on via a "poller timer"
set of commands) would be much more
useful. I'll look into that.</div>
<div dir="ltr">
<div><br>
</div>
<div>Average time is probably a good
stat - and certainly what we care
about.</div>
<div><br>
</div>
<div>I actually am hopeful that those
couple of things I did might help
reduce that average time quite a bit
(that short-cutting the isotp protocol
handling especially). </div>
<div><br>
</div>
<div>That p/r with logging changes might
help reduce the unproductive log time
further, but also makes it possible to
turn on the poller logging without the
RX task logs kicking in.</div>
<div><br>
</div>
<div>//.ichael</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Sun, 12 May 2024 at 22:29, Michael
Balzer via OvmsDev <<a
href="mailto:ovmsdev@lists.openvehicles.com" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ovmsdev@lists.openvehicles.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> Warning / gathering debug
statistics about slow processing can
be helpful, but there must not be a
hard limit. Frame/poll response
processing may need disk or network
I/O, and the vehicle task may be
starving from punctual high loads on
higher priority tasks (e.g.
networking) or by needing to wait
for some semaphore -- that's outside
the application's control, and must
not lead to termination/recreation
of the task (in case you're heading
towards that direction).<br>
<br>
I have no idea how much processing
time the current vehicles actually
need in their respective worst
cases. Your draft is probably too
lax, poll responses and frames
normally need to be processed much
faster. I'd say 10 ms is already too
slow, but any wait for a
queue/semaphore will already mean at
least 10 ms (FreeRTOS tick).
Probably best to begin with just
collecting stats.<br>
<br>
Btw, to help in narrowing down the
actual problem case, the profiler
could collect max times per RX
message ID.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div>Am 12.05.24 um 10:41 schrieb
Michael Geddes:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>I have a question for
Michael B (or whoever) - I
have a commit lined up that
would add a bit of a time
check to the poller loop. What
do we expect the maximum time
to execute a poller loop
command should be? </div>
<div>This is a rough idea (in
ms) I have.. based on nothing
much really, so any ideas
would be appreciated:<br>
</div>
<div> int
TardyMaxTime_ms(OvmsPoller::OvmsPollEntryType
entry_type)<br>
{<br>
switch (entry_type)<br>
{<br>
case
OvmsPoller::OvmsPollEntryType::Poll:
return 80;<br>
case
OvmsPoller::OvmsPollEntryType::FrameRx:
return 30;<br>
case
OvmsPoller::OvmsPollEntryType::FrameTx:
return 20;<br>
case
OvmsPoller::OvmsPollEntryType::Command:
return 10;<br>
case
OvmsPoller::OvmsPollEntryType::PollState:
return 15;<br>
default: return 80;<br>
}<br>
}<br>
</div>
<div><br>
</div>
<div>//.ichael<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Mon, 6
May 2024 at 07:45, Michael
Geddes <<a
href="mailto:frog@bunyip.wheelycreek.net" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">frog@bunyip.wheelycreek.net</a>> wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="auto">I realise
that I was only using
the standard cable to
test - which probably is
not sufficient - I
haven't looked closely
at how the Leaf OBD to
Db9 cable is different
from standard.<br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Ah, my bad
out the queue length.
We are definitely
queueing more messages
though. From my log of
when the overflow
happened, the poller was
in state 0 which means
OFF - ie nothing was
being sent!!</div>
<div dir="auto"><br>
</div>
<div dir="auto">I'll look
at the TX message thing
- opt in sounds good -
though it shouldn't be
playing that much of a
part here as the TXs are
infrequent in this case
(or zero when the leaf
is off or driving) - On
the ioniq 5 when I'm
using the HUD - I'm
polling quite frequently
- multiple times per
second and that seems to
be fine!.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I did find
an issue with the
throttling .. but it
would still mostly apply
the throttling where it
matters, so again, it
shouldn't be the problem
(also, we aren't
transmitting in the leaf
case).</div>
<div dir="auto">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">The
change I made to the
logging of RX
messages showed how
many in a row were
dropped... and it
was mostly 1 only in
a run - which means
even if it is a
short time between -
that means that the
drops are being
interleaved by at
least one success!</div>
<div><br>
</div>
<div>Sooo.. I'm still
wondering what is
going on. Some
things I'm going to
try:</div>
<div><br>
</div>
<div>* If the number
of messages on the
Can bus (coming in
through RX) means
that the queue is
slowly getting
longer and not quite
catching up, then
making the queue
longer will help it
last longer... but
only pushes the
problem down the
road.</div>
<div> - Add 'current
queue length' to the
poller status
information to see
if this is indeed
the case? </div>
<div> - Add some kind
of alert when the
queue reaches a %
full?</div>
<div>* Once you start
overflowing and
getting overflow log
messages, I wonder
if this is then
contributing to the
problem.</div>
<div> - Push the
overflow logging
into Poller Task
which can look at
how many drops
occurred since last
received item.</div>
<div>* Split up the
flags for the poller
messages into 2:<br>
</div>
<div> - Messages
that are/could be
happening in the
TX/RX tasks</div>
<div> - Other noisy
messages that always
happen in the poller
task.</div>
<div><br>
</div>
<div>Thoughts on what
else we might
measure to figure
out what is going
on?<br>
</div>
<div><br>
</div>
<div>//.ichael</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On
Sun, 5 May 2024, 19:29
Michael Balzer via
OvmsDev, <<a
href="mailto:ovmsdev@lists.openvehicles.com" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ovmsdev@lists.openvehicles.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> Michael,<br>
<br>
the queue size isn't
in bytes, it's in
messages:<br>
<br>
<blockquote
type="cite"><font
face="monospace"> *
@param
uxQueueLength
The maximum
number of items
that the queue
can contain.<br>
*<br>
* @param
uxItemSize The
number of bytes
each item in the
queue will
require.</font></blockquote>
<br>
Also, from the time
stamps in Dereks log
excerpt, there were
quite some dropped
frames in that time
window -- at least
23 frames in 40 ms,
that's bad.<br>
<br>
Queue sizes are
currently:<br>
<br>
<font
face="monospace">CONFIG_OVMS_HW_CAN_RX_QUEUE_SIZE=60<br>
CONFIG_OVMS_VEHICLE_CAN_RX_QUEUE_SIZE=60</font><br>
<br>
The new poller now
channels all TX
callbacks through
the task queue
additionally to RX
and commands. So
setting the queue
size to be larger
than the CAN RX
queue size seems
appropriate.<br>
<br>
Nevertheless, an
overflow with more
than 60 waiting
messages still
indicates some too
long processing time
in the vehicle task.<br>
<br>
TX callbacks
previously were done
directly in the CAN
context, and no
current vehicle
overrides the empty
default handler, so
this imposed almost
no additional
overhead. By
requiring a queue
entry for each TX
callback, this
feature now has a
potentially high
impact for all
vehicles. If passing
these to the task is
actually necessary,
it needs to become
an opt-in feature,
so only vehicles
subscribing to the
callback actually
need to cope with
that additional load
& potential
processing delays
involved.<br>
<br>
Regards,<br>
Michael<br>
<br>
<blockquote
type="cite">
<div
class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div
class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div
lang="EN-AU">
<div>
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<div>
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div
style="border:1pt solid rgb(204,204,204);padding:7pt"> </div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div>
_______________________________________________<br>
OvmsDev mailing list<br>
<a
href="mailto:OvmsDev@lists.openvehicles.com" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">OvmsDev@lists.openvehicles.com</a><br>
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
OvmsDev mailing list
<a href="mailto:OvmsDev@lists.openvehicles.com" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">OvmsDev@lists.openvehicles.com</a>
<a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div>
_______________________________________________<br>
OvmsDev mailing list<br>
<a href="mailto:OvmsDev@lists.openvehicles.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">OvmsDev@lists.openvehicles.com</a><br>
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</body>
</html>