<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Our current implementation lacks a recovery attempt from bus errors
and arbitration losses, we only record the error status.<br>
<br>
The new Espressif CAN driver may contain some useful info on this.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 15.08.2018 um 15:46 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite"
cite="mid:70C03BEE-CA3B-43D4-8283-816DD46F64DE@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Sorry, forgot to mention: another option is to poll the bus
manually if an interrupt hasn’t been received in N seconds. Check
the status registers and if everything is not perfect then reset
the controller.<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 15 Aug 2018, at 9:45 PM, Mark Webb-Johnson
<<a href="mailto:mark@webb-johnson.net" class=""
moz-do-not-send="true">mark@webb-johnson.net</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space;
line-break: after-white-space;" class="">This was my worry
for both esp32 can and mcp2515. We get an interrupt, but
for some reason the status is not showing anything to do,
so we return from the interrupt. Then, the status changes.
Or, for a particular status we had to do something that we
didn’t. The bus is locked up, and the interrupt never
fires again, so we never get called. I wonder if we just
fired the interrupt handler again manually, would it
recover?
<div class=""><br class="">
</div>
<div class="">When this happens, do we need to power off
then on the can bus, or is a ‘can canX stop’ ‘can canX
start …’ sufficient?</div>
<div class=""><br class="">
</div>
<div class="">Perhaps a general solution would be a
watchdog. If the CAN bus does not receive anything for N
seconds (perhaps 60), then restart the controller? We
could simply protect it from restarting twice (a simple
check of the counters), so worse case this would restart
once 60 seconds after a bus normally went idle. Or is
that too kludgy?</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 14 Aug 2018, at 6:57 PM, Michael
Balzer <<a href="mailto:dexter@expeedo.de"
class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type"
content="text/html; charset=utf-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
Mark,<br class="">
<br class="">
Frank has observed another lockup on can1, RX
was stuck, not sure about TX. Status was:<br
class="">
<br class="">
<pre style="padding: 8.5px; font-family: Monaco, Menlo, Consolas, "QuickType Mono", "Lucida Console", "Roboto Mono", "Ubuntu Mono", "DejaVu Sans Mono", "Droid Sans Mono", monospace; font-size: 12px; color: rgb(51, 51, 51); border-radius: 4px; display: block; margin: 0px 0px 9px; line-height: 18px; word-break: break-all; word-wrap: break-word; white-space: pre-wrap; background-color: rgb(245, 245, 245); border: 1px solid rgba(0, 0, 0, 0.15); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;" class="">CAN: can1
Mode: Active
Speed: 500000
Interrupts: 22425486
Rx pkt: 21080181
Rx err: 5
Rx ovrflw: 2054
Tx pkt: 1340517
Tx delays: 335
Tx err: 0
Tx ovrflw: 0
Err flags: 0x00800c5b
</pre>
<br class="Apple-interchange-newline">
That seems to be a bus error IRQ -- but
strangely, the bus error status flag is not set?<br
class="">
<br class="">
ECC 5b = form error on TX in acknowledge
delimiter (?)<br class="">
<br class="">
Can we use some of this info as an indicator
that restarting the interface is necessary?<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 12.06.2018 um
15:44 schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:5412B335-B70B-468C-9470-8078088BC0BA@webb-johnson.net"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=utf-8" class="">
I wrote extensions to the ‘test canrx’ and
‘test cantx’ commands. They now provide high
resolution timings, which can be used to get
an idea of performance. CAN2/CAN3 tx is dog
slow compared to CAN1.
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;"
class="">OVMS# test cantx can2 100</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;"
class="">Testing 100 frames on can2</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;"
class="">Transmitted 100 frames in
0.299612s = 2996us/frame</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;"
class=""><br class="">
</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;"
class="">OVMS# test cantx can1 100</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;"
class="">Testing 100 frames on can1</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;"
class="">Transmitted 100 frames in
0.013691s = 136us/frame</span></font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">I tried adding the "while
(MODULE_ESP32CAN->SR.B.RBS)” into
esp32can, and it seems to work just fine.
Not sure how to replicate the problem, so
not sure if it resolves anything - but it
doesn’t seem to make anything worse so I
committed it. Feel free to revert that
commit if it causes any issues.</div>
<div class=""><br class="">
</div>
<div class="">I’ll continue looking into this
tomorrow.</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 12 Jun 2018, at 2:37
PM, Mark Webb-Johnson <<a
href="mailto:mark@webb-johnson.net"
class="" moz-do-not-send="true">mark@webb-johnson.net</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type"
content="text/html; charset=utf-8"
class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
line-break: after-white-space;"
class="">
<blockquote type="cite" class="">But
how would you check that a frame
is available? I understand from
the documentation, the RX IRQ
isn't raised unless the FIFO has a
valid message and (in<br class="">
PeliCAN mode) has been mapped to
the buffer address. I can't see
how we could check the frame
validity additionally to that.</blockquote>
<div class=""><br class="">
</div>
I suggest to put a check for
MODULE_ESP32CAN->SR.B.RBS at the
start of ESP32CAN_rxframe. That
status bit is documented as
indicating "one or more messages are
available in the RXFIFO”. Presumably
we shouldn’t be reading from the RX
FIFO unless that bit is set.
<div class=""><br class="">
</div>
<div class="">I agree that the
documentation doesn’t talk about
spurious interrupts, but adding
that check should not do any harm.
Reading the receive buffer when
nothing is in it, by comparison,
would produce all sorts of weird
incoming frame results (including
duplicates, and the partial frame
corruptions you mention, due to RX
FIFO wrapping).</div>
<div class=""><br class="">
</div>
<div class="">Looking at page #28 of
the datasheet, we have:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0
40px; border: none; padding: 0px;"
class="">
<div class="">
<div class=""><i class="">4.
After reading the contents
of the receive buffer, the
CPU can release this memory
space in the RXFIFO by
setting the release receive
buffer bit to logic 1. This
may result in another
message becoming immediately
available within the receive
buffer. If there is no other
message available, the
receive interrupt bit is
reset.</i></div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">Does that mean that
we should be checking the
receive buffer
in ESP32CAN_rxframe and looping?
I’m confused because if it did
not then the interrupt would
presumably fire again, and cause
recursion (or is the new
interrupt delayed until the end
of the current ISR execution)?
Perhaps we can simply do a:</div>
<div class=""><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0
40px; border: none; padding: 0px;"
class="">
<div class="">
<div class=""><font class=""
face="Andale Mono"><span
style="font-size: 18px;"
class="">while
(MODULE_ESP32CAN->SR.B.RBS)</span></font></div>
</div>
<div class=""><font class=""
face="Andale Mono"><span
style="font-size: 18px;"
class=""> {</span></font></div>
<div class=""><font class=""
face="Andale Mono"><span
style="font-size: 18px;"
class=""> // Read and
process a message</span></font></div>
<div class=""><font class=""
face="Andale Mono"><span
style="font-size: 18px;"
class=""> }</span></font></div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">in ESP32CAN_rxframe
to process all incoming
messages? The mcp2515 does
something similar (relying on
can.cpp CAN_rxstask to loop
until no more messages
available).</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.</div>
<div class=""><br class="">
<blockquote type="cite"
class="">
<div class="">On 12 Jun
2018, at 5:22 AM, Michael
Balzer <<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div class="">Am
11.06.2018 um 04:07
schrieb Mark
Webb-Johnson:<br
class="">
<blockquote type="cite"
class="">I think the
fault must be internal
(controller, or our
driver code), as the
can bus itself has
error checking.<br
class="">
The only other thing I
noticed on my review
is that in
ESP32CAN_rxframe there
is no check that a
frame is actually
available before
reading and queueing
it. If the RX
interrupt is spurious,
that could cause all
sorts of issues. I
think it would be
safer to have such a
check, and maybe that
is the cause of this
issue?<br class="">
</blockquote>
<br class="">
You mean we could be
reading the RX mailbox
a) without a new signal
and b) before the next
message is fully rolled
in.<br class="">
<br class="">
But how would you check
that a frame is
available? I understand
from the documentation,
the RX IRQ isn't raised
unless the FIFO has a
valid message and (in<br
class="">
PeliCAN mode) has been
mapped to the buffer
address. I can't see how
we could check the frame
validity additionally to
that.<br class="">
<br class="">
Setting CMR.RRB then
allows the chip to fetch
the next message from
the FIFO, which will
trigger another IR.RI as
soon as the buffer is
filled.<br class="">
<br class="">
IR.RI is a level
interrupt, so if stuck
will trigger again. But
reading IR clears it. So
even if the ISR would be
called again due to an
IRQ handling bug, IR.RI<br
class="">
would be 0, so rxframe()
wouldn't be called.<br
class="">
<br class="">
…unless IR.RI was set
due to a hardware bug…<br
class="">
<br class="">
…or IR.RI could also be
set on a data overrun
(or other error) for an
invalid message. That
would mean IR.RI must be
discarded in case of an
error interrupt<br
class="">
coming in with IR.RI,
but I've seen nothing in
the documentation about
something like this.<br
class="">
<br class="">
<blockquote type="cite"
class="">I’m working
on the MCP2515 driver
at the moment, to try
to find out what is
causing the OBDII HUD
lock-up (and others
have seen). Perhaps
I’ll find out
something when
reviewing the driver
and shared CAN
library. Due to the
huge volume of traffic
on my Tesla Model S, I
get a large number of
overruns but don’t see
this problem. Maybe it
is rare, or we are
just not noticing it.<br
class="">
</blockquote>
<br class="">
I assume the problem can
be triggered by poor
Wifi connectivity --
that's the situation at
Frank's home. I assume
the Wifi driver disables
interrupts or context<br
class="">
switches in some
situations, which causes
the CAN rx task
execution pauses.<br
class="">
<br class="">
Just a wild guess
though, I would check
the source for this
pattern if it was open.<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<blockquote type="cite"
class="">Regards, Mark<br
class="">
<br class="">
<blockquote
type="cite" class="">On
10 Jun 2018, at 9:05
PM, Michael Balzer
<<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:<br class="">
<br class="">
Frank Demes reported
a very strange CAN
behaviour. He
managed to get a
CRTD log of one
incident.<br
class="">
<br class="">
The effect is
triggered by an
unknown cause
freezing the CAN rx
task for short
periods of time,
i.e. 70 - 120 ms.
This alone should be
no problem, the
RXFIFO<br class="">
will probaly
overflow, but that
is handled by our
driver, so except
some messages lost
this should not be
an issue.<br
class="">
<br class="">
Unfortunately,
something goes
completely wrong:<br
class="">
<br class="">
a) on the overrun
event, the last
valid message is
delivered multiple
times (i.e. 5 times)<br
class="">
<br class="">
b) after resolving
the overrun, a
garbled message is
received.<br
class="">
<br class="">
The garbled message
contains a valid
byte sequence in the
wrong place, i.e.
bytes 5-8 would be
valid if they were
bytes 1-4. Bytes 1-4
cannot have been
read<br class="">
from the bus, so
something is going
very wrong here.<br
class="">
<br class="">
I have walked
through the esp32can
driver and checked
the SJA1000
documentation but
have no clue what is
going wrong.<br
class="">
<br class="">
Any ideas? CRTD
excerpt below.
(Frame 155 is
reflected by the
module with byte 1
changed from 07 to
01 to limit the
charge current. The
problem is, it also<br
class="">
reflects the garbled
message, which
causes the charger
to stop.)<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
----------------------------------------------------------------------------------------<br
class="">
…<br class="">
274.341 1R11 155 07
97 E4 54 4B A8 00 6C<br
class="">
274.341 1T11 155 01
97 E4 54 4B A8 00 6C<br
class="">
274.351 1R11 155 07
97 E5 54 4B A8 00 6C<br
class="">
274.351 1T11 155 01
97 E5 54 4B A8 00 6C<br
class="">
274.361 1R11 155 07
97 E5 54 4B A8 00 6C<br
class="">
274.361 1T11 155 01
97 E5 54 4B A8 00 6C<br
class="">
274.371 1R11 423 03
22 FF FF 00 E0 00 DB<br
class="">
274.371 1R11 426 00
38 01 00 4D 7E 00<br
class="">
274.371 1R11 597 20
A4 03 B1 2F 00 01 53<br
class="">
274.371 1R11 627 00
00 00<br class="">
274.371 1R11 155 07
97 E5 54 4B A8 00 6C<br
class="">
274.371 1T11 155 01
97 E5 54 4B A8 00 6C<br
class="">
274.381 1R11 155 07
97 E5 54 4B A8 00 6C<br
class="">
274.381 1T11 155 01
97 E5 54 4B A8 00 6C<br
class="">
274.391 1R11 5D7 FF
FF 01 E4 53 80 00<br
class="">
<br class="">
- normal up to here<br
class="">
<br class="">
!!! CAN RX is
blocked here for ~70
ms !!!<br class="">
… interrupts
disabled? by whom?<br
class="">
<br class="">
274.461 1R11 599 00
00 4D 7E 1E 26 FF FF<br
class="">
274.461 1R11 436 00
0E 3C B2 00 00<br
class="">
274.461 1R11 155 07
97 E5 54 4B A8 00 6C<br
class="">
274.461 1R11 424 11
40 15 26 47 64 00 48<br
class="">
274.461 1R11 425 0A
1D 44 FF FE 40 01 1F<br
class="">
274.461 1R11 556 30
63 07 30 73 07 30 7A<br
class="">
274.461 1T11 155 01
97 E5 54 4B A8 00 6C
← reflection
for 155 still works
here<br class="">
<br class="">
!!! next RX pause,
~90 ms this time !!!<br
class="">
<br class="">
!!! ID 599 has a
period of 100 ms,
but we now get it 5
times:<br class="">
<br class="">
274.551 1R11 599 00
00 4D 7E 1E 26 FF FF<br
class="">
<br class="">
274.551 1CEV Error
intr=58445
rxpkt=27132
txpkt=528128
errflags=0x15433
rxerr=0 txerr=0
rxovr=2 txovr=0
txdelay=17<br
class="">
crtd fprintf bug
compensation:<br
class="">
274.551 1CEV Error
rxpkt=58445
txpkt=27132
errflags=528128
intr=0x15433 rxerr=0
txerr=0 rxovr=2
txovr=0 txdelay=17<br
class="">
errflags=528128 =
08 0F 00<br class="">
0x08 = IR: Data
overrun<br class="">
0x0F = SR: Bus
OK, error limit not
yet reached, RX
& TX idle,
RXFIFO overrun, RX
buffer full<br
class="">
0x00 = ECC: -<br
class="">
<br class="">
274.551 1R11 599 00
00 4D 7E 1E 26 FF FF<br
class="">
274.551 1R11 599 00
00 4D 7E 1E 26 FF FF<br
class="">
274.551 1R11 599 00
00 4D 7E 1E 26 FF FF<br
class="">
274.551 1R11 599 00
00 4D 7E 1E 26 FF FF<br
class="">
274.551 1R11 155 07
97 E6 54 4B A8 00 6C
RX "A"<br class="">
274.551 1R11 155 07
97 E6 54 4B A8 00 6C
RX "B"<br class="">
274.551 1R11 155 07
97 E7 54 4B A8 00 6C
RX "C"<br class="">
274.551 1R11 423 03
22 FF FF 00 E0 00 DB<br
class="">
274.551 1R11 426 00
38 01 00 4D 7E 00<br
class="">
274.551 1R11 597 20
A4 03 B1 2F 00 01 53<br
class="">
274.551 1R11 627 00
00 00<br class="">
274.551 1T11 155 01
97 E6 54 4B A8 00 6C
reflection "A"<br class="">
274.551 1CEV
TX_Queue T11 155 01
97 E6 54 4B A8 00 6C
reflection "B"<br
class="">
274.551 1CEV
TX_Queue T11 155 01
97 E7 54 4B A8 00 6C
reflection "C"<br
class="">
274.551 1T11 155 01
97 E8 54 4B A8 00 6C
reflection "D"<br class="">
<br class="">
!!! 120 ms pause
!!!<br class="">
<br class="">
274.671 1R11 155 07
97 E8 54 4B A8 00 6C
RX "D"<br class="">
<br class="">
→ "D" was sent to
the listeners &
reflected by
vehicle::IncomingFrame()<br
class="">
then, before the
LogFrame() call was
processed, the CAN
task was paused for
120 ms<br class="">
<br class="">
and now it's
getting really
weird:<br class="">
<br class="">
274.671 1R11 155 07
08 2A A0 07 97 E7 54
RX "E": invalid frame / garbled message!<br class="">
<br class="">
… this frame
certainly was not on
the bus!<br class="">
bytes 2-4 (08 2A
A0) = complete
nonsense, not
matching any frame
on the bus<br
class="">
bytes 5-8 (07 97
E7 54) = normally
bytes 1-4 (!) of ID
155<br class="">
- copy / memory
error? … very
unlikely<br class="">
- CAN RX handling
error? … doesn't
seem so<br class="">
- RXFIFO message
window bug?<br
class="">
<br class="">
274.671 1CEV Error
intr=58458
rxpkt=27134
txpkt=528128
errflags=0x15456
rxerr=0 txerr=0
rxovr=3 txovr=0
txdelay=19<br
class="">
274.671 1R11 155 07
97 E7 54 4B A8 00 6C<br
class="">
274.671 1R11 155 07
97 E7 54 4B A8 00 6C<br
class="">
274.671 1R11 155 07
97 E7 54 4B A8 00 6C<br
class="">
274.671 1T11 155 01
97 E6 54 4B A8 00 6C<br
class="">
274.671 1CEV
TX_Queue T11 155 01
97 E7 54 4B A8 00 6C<br
class="">
274.671 1R11 155 07
97 F2 54 4B A8 00 6C<br
class="">
274.671 1R11 155 07
97 F3 54 4B A8 00 6C<br
class="">
274.671 1R11 423 03
22 FF FF 00 E0 00 DB<br
class="">
274.671 1R11 426 00
38 01 00 4D 7E 00<br
class="">
274.671 1R11 597 20
A4 03 B1 2F 00 01 53<br
class="">
274.671 1R11 627 00
00 00<br class="">
274.671 1R11 155 07
97 F5 54 4B A8 00 6C<br
class="">
274.671 1R11 155 07
97 F6 54 4B A8 00 6C<br
class="">
274.671 1R11 5D7 FF
FF 01 E4 53 80 00<br
class="">
274.671 1R11 599 00
00 4D 7E 1E 26 FF FF<br
class="">
274.671 1R11 155 07
97 F7 54 4B A8 00 6C<br
class="">
274.671 1R11 436 00
0E 3C B2 00 00<br
class="">
274.671 1R11 041 A0
07 97 F5 54 4B A8 00<br
class="">
274.671 1T11 155 01
97 E7 54 4B A8 00 6C<br
class="">
274.671 1T11 155 01
08 2A A0 07 97 E7 54
reflection of invalid msg "E" → charge terminates due
to error<br class="">
<br class="">
<br class="">
-- <br class="">
Michael Balzer *
Helkenberger Weg 9 *
D-58256 Ennepetal<br
class="">
Fon 02333 / 833 5735
* Handy 0176 / 206
989 26<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class=""
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class=""
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
-- <br class="">
Michael Balzer *
Helkenberger Weg 9 *
D-58256 Ennepetal<br
class="">
Fon 02333 / 833 5735 *
Handy 0176 / 206 989 26<br
class="">
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class=""
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</body>
</html>