<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Michael,<br>
<br>
Ok, so I believe I loaded your Edge build, and the issue remains.
To verify, this is the string reported by the web Status page:<br>
<br>
3.2.007-102-g07440970/ota_1/main (build idf
v3.3-beta3-774-g6652269e1 Dec 12 2019 11:59:14)<br>
<br>
Following my earlier email, I left the configuration as-is,
including launching the obd2ecu task, but with the attached OBDII
module removed, and the crash was resolved. Rebooting the module
had it come up running normally. I then plugged in the OBDII module
into the OVMSv3, and OVMS crashed the instant the first it started
polling. The device is a SyncUp Drive, which is a telematics and
WiFi hotspot. Connecting the OBDII device with 12v power off, as
expected, did nothing. Power it on, wait a few seconds to boot, and
OVMS crashes immediately.<br>
<br>
New console log, attached. It shows the module booting normally
(OBDII device not present), followed by the device being attached
(indicated in the log), and the subsequent crash. I removed the
device right after the boot started, so as to let the OVMS boot
normally.<br>
<br>
So, clearly the obd2ecu task it is receiving frames on CAN3 just
fine, but transmits back out on that same bus aren't working. Is
this something I need to address within obd2ecu, or is it in the
core code? I've not changed obd2ecu in quite some time.<br>
<br>
Greg<br>
<br>
<br>
<div class="moz-cite-prefix">Michael Balzer wrote:<br>
</div>
<blockquote type="cite"
cite="mid:ed1b2b37-eeb8-202f-cca6-81638d6e353a@expeedo.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Fix commit is <a
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/8fef3f5fb6eec9fca26997e01d9c7d08080cdc06"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/8fef3f5fb6eec9fca26997e01d9c7d08080cdc06</a><br>
<br>
Greg, please test.<br>
<br>
New build with SPIRAM fix: <a
href="https://ovms.dexters-web.de/firmware/ota/v3.1/edge/"
moz-do-not-send="true">https://ovms.dexters-web.de/firmware/ota/v3.1/edge/</a><br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 12.12.19 um 11:17 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:7b52c31f-1fe5-cc5a-ec52-56bd6df7553a@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Last line of a2l was missing:<br>
<br>
<tt>#!/bin/bash</tt><tt><br>
</tt><tt>elf=~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/build/ovms3.elf</tt><tt><br>
</tt><tt>for adr in $* ; do</tt><tt><br>
</tt><tt> if [[ "$adr" =~ "elf" ]] ; then</tt><tt><br>
</tt><tt> elf="$adr"</tt><tt><br>
</tt><tt> else</tt><tt><br>
</tt><tt> cmd+=" -ex 'l *${adr/:*/}'"</tt><tt><br>
</tt><tt> fi</tt><tt><br>
</tt><tt>done</tt><tt><br>
</tt><tt>cmd+=" -ex 'q'"</tt><tt><br>
</tt><tt>echo "Using elf file: $elf"</tt><br>
<tt>eval xtensa-esp32-elf-gdb -batch $elf $cmd 2>/dev/null </tt><br>
<br>
<br>
<br>
<div class="moz-cite-prefix">Am 12.12.19 um 11:12 schrieb
Michael Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:eed5cc60-9aa8-8b65-bb80-184125826c24@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Mark,<br>
<br>
1) maybe I missed posting my later version of a2l, which
automatically strips the stack pointers from the ":" address
syntax so you can copy&paste the USB output. Here it is:<br>
<br>
<tt>#!/bin/bash</tt><tt><br>
</tt><tt>elf=~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/build/ovms3.elf</tt><tt><br>
</tt><tt>for adr in $* ; do</tt><tt><br>
</tt><tt> if [[ "$adr" =~ "elf" ]] ; then</tt><tt><br>
</tt><tt> elf="$adr"</tt><tt><br>
</tt><tt> else</tt><tt><br>
</tt><tt> cmd+=" -ex 'l *${adr/:*/}'"</tt><tt><br>
</tt><tt> fi</tt><tt><br>
</tt><tt>done</tt><tt><br>
</tt><tt>cmd+=" -ex 'q'"</tt><tt><br>
</tt><tt>echo "Using elf file: $elf"</tt><tt><br>
</tt><br>
<br>
2) You're right, the bug is tx_frame with null origin
overwriting body.bus in the union. I didn't notice that when
checking Marko's submission.<br>
<br>
tx_frame is a copy of the last frame given to the bus for
transmission. The queue msg is gone when the TX done IRQ comes
in, and Marko needed a copy of the frame the TX IRQ relates
to. I asked him (see PR discussion), he checked and confirmed
that all TX is done sequentially, so a single buffer is
sufficient.<br>
<br>
Swapping the lines would work. The frame.origin also shouldn't
be null, but the handler should tolerate that. …oops, tx_frame
also doesn't get initialized in the canbus constructor, so
there's also potentially garbage in tx_frame if due to some
bug a TX IRQ is generated or processed without a previous tx.<br>
<br>
I'll do the fixes… and also rename tx_frame to m_tx_frame for
consistency.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 12.12.19 um 01:20 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite"
cite="mid:DDE51878-283B-45E6-8423-FF095EF621A6@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<div class="">Two issues:</div>
<div class=""><br class="">
</div>
<div class=""><u class=""><b class="">1] A2L</b></u></div>
<div class=""><br class="">
</div>
My a2l is this:
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none;
padding: 0px;" class="">
<div class="">
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">#!/bin/bash</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">elf=3.2.007.ovms3.elf</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">for adr in $* ; do</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> if [[ "$adr" =~ "elf" ]] ; then</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> elf="$adr"</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> else</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> cmd+=" -ex 'l *$adr'"</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> fi</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">done</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">cmd+=" -ex 'q'"</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">echo "Using elf file: $elf"</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">echo "xtensa-esp32-elf-gdb -batch $elf
$cmd"</span></font></div>
<div class=""><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">eval xtensa-esp32-elf-gdb -batch $elf
$cmd 2>/dev/null #| grep "^0x.* is in "</span></font></div>
</div>
</blockquote>
<div class="">
<div><br class="">
</div>
<div>When I run it, I get:</div>
<div><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border: none;
padding: 0px;" class="">
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">$ a2l 3.2.007.ovms3.elf
0x400d5e4c:0x3ffc5c40 0x7ffffffd:0x3ffc5c90</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">Using elf file: 3.2.007.ovms3.elf</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-size: 14px;" class="">xtensa-esp32-elf-gdb
-batch 3.2.007.ovms3.elf -ex 'l
*0x400d5e4c:0x3ffc5c40' -ex 'l
*0x7ffffffd:0x3ffc5c90' -ex ‘q'</span></font></div>
</blockquote>
<div class="">
<div><br class="">
</div>
<div>And a manual run gives:</div>
<div><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border: none;
padding: 0px;" class="">
<div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">$ xtensa-esp32-elf-gdb -batch
3.2.007.ovms3.elf -ex 'l *0x400d5e4c:0x3ffc5c40'
-ex 'l *0x7ffffffd:0x3ffc5c90' -ex 'q'</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">Junk at end of line specification.</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""><br class="">
</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">$ </span><span style="font-size: 14px;"
class="">xtensa-esp32-elf-gdb 3.2.007.ovms3.elf</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-size: 14px;" class="">GNU gdb
(crosstool-NG crosstool-ng-1.22.0-80-g6c4433a)
7.10</span></font></div>
<div><span style="font-size: 14px; font-family:
"Andale Mono";" class="">(gdb) l
*0x400d5e4c:0x3ffc5c40</span></div>
<div><font class="" face="Andale Mono"><span
style="font-size: 14px;" class="">
<div>Junk at end of line specification.</div>
<div>(gdb) l *0x7ffffffd:0x3ffc5c90</div>
<div>(gdb) quit</div>
</span></font></div>
</div>
</blockquote>
<div class="">
<div><br class="">
</div>
<div><u class=""><b class="">2] Frame origin</b></u></div>
<div><br class="">
</div>
<div>Regarding the CAN_txcallback, I think you are
correct. And both generators of that message set the
frame correctly. So my initial thought was that it is
either a memory corruption, or somewhere else sending a
frame with garbage data.</div>
<div><br class="">
</div>
<div>I do see this technique used in both the mcp2515 and
esp32can drivers:</div>
<div><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border: none;
padding: 0px;" class="">
<div class="">
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">msg.body.bus = me;</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">msg.body.frame = me->tx_frame;</span></font></div>
</div>
</blockquote>
<div class="">
<div><br class="">
</div>
<div>I don’t normally just copy structures over like that.
I memcpy() them, but I guess it must work. However, as
our CAN_queue_msg_t is a union of CAN_frame_t (frame,
with first member of the structure a canbus*) and
'canbus* bus’, that is a little worrying. It seems that
the msg.body.bus will get overwritten with whatever is
in msg.body.frame.origin. I can’t see anywhere that
tx_frame.origin is set - which is scary because that
would mean it is always random junk.</div>
<div><br class="">
</div>
<div>Maybe it works if tx_frame.origin is set to the bus
before anything else, but not in Greg’s circumstances
where something else arrives first. Perhaps related to
obd2ecu? I only see tx_frame set in can.cpp
canbus::Write. I don’t really understand the tx_frame
approach at all, and why the frame is just not passed on
the queue.</div>
<div><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border: none;
padding: 0px;" class="">
<div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">// CAN Frame</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">// Note: Take care changing this
structure, as it is a union with</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">// CAN_log_message_t and position of
'origin' is fixed.</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class="">struct CAN_frame_t</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> {</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> canbus* origin;
// Origin of the frame</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> CanFrameCallback * callback;
// Frame-specific callback. Is called when this
frame is successfully sent (or sending failed)</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> CAN_FIR_t FIR;
// Frame information record</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> uint32_t MsgID;
// Message ID</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> union</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> {</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> uint8_t u8[8];
// Payload byte access</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> uint32_t u32[2];
// Payload u32 access (Att: little endian!)</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> uint64_t u64;
// Payload u64 access (Att: little endian!)</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> } data;</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""><br class="">
</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> esp_err_t Write(canbus* bus=NULL,
TickType_t maxqueuewait=0); // bus: NULL=origin</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""> };</span></font></div>
<div><font class="" face="Andale Mono"><span
style="font-style: normal; font-size: 14px;"
class=""><br class="">
</span></font></div>
<div>
<div><font class="" face="Andale Mono"><span
style="font-size: 14px;" class="">// CAN message<br
class="">
typedef struct<br class="">
{<br class="">
CAN_queue_type_t type;<br class="">
union<br class="">
{<br class="">
CAN_frame_t frame; // CAN_frame<br class="">
canbus* bus;<br class="">
} body;<br class="">
} CAN_queue_msg_t;</span></font></div>
</div>
</div>
</blockquote>
<div class="">
<div><br class="">
</div>
<div>This approach seems to date back to the swcan support
merge (f94ae5a1b).</div>
<div><br class="">
</div>
<div>Should we swap around, and set the msg.body.bus after
msg.body.frame? Or am I missing something…</div>
<div><br class="">
</div>
<div>Regards, Mark.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 12 Dec 2019, at 2:29 AM, Michael
Balzer <<a href="mailto:dexter@expeedo.de"
class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div class=""> Mark,<br class="">
<br class="">
good example why not to use addr2line: I think
that result is wrong. a2l uses gdb which gives:<br
class="">
<br class="">
<tt class=""><a class="moz-txt-link-abbreviated"
href="mailto:balzer@leela:%7E/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3"
moz-do-not-send="true">balzer@leela:~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3</a>>
a2l tmp/3.2.007.ovms3.elf 0x400d5e4c:0x3ffc5c40
0x7ffffffd:0x3ffc5c90<br class="">
Using elf file: tmp/3.2.007.ovms3.elf<br
class="">
0x400d5e4c is in CAN_rxtask(void*)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/can/src/can.cpp:730).<br
class="">
725
me->IncomingFrame(&msg.body.frame);<br
class="">
726 } while (loop);<br class="">
727 break;<br class="">
728 }<br class="">
729 case CAN_txcallback:<br class="">
730
msg.body.bus->TxCallback(&msg.body.frame,
true);<br class="">
731 break;<br class="">
732 case CAN_txfailedcallback:<br
class="">
733
msg.body.bus->TxCallback(&msg.body.frame,
false);<br class="">
734
msg.body.bus->LogStatus(CAN_LogStatus_Error);<br
class="">
</tt><tt class=""><br class="">
</tt>…and that actually makes sense and matches
the register dump.<br class="">
<br class="">
If I read the gdb disassembly correctly, A10 =
msg.body.bus, so Greg's got a CAN_txcallback msg
without a bus.<br class="">
<br class="">
Hardening the rxtask against null here would
probably avoid the crash, but I don't see yet how
that could be possible.<br class="">
Both esp32can and mcp2515 set the bus field to
their object addresses, which cannot be null.<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 11.12.19 um 13:45
schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:1BF79214-A026-4AD0-97D4-5C845FBB0328@webb-johnson.net"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
Can’t get a2l working at the moment. The
addr2line gives:
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class="">addr2line -e 3.2.007.ovms3.elf
0x400d5e4c:0x3ffc5c40
0x7ffffffd:0x3ffc5c90</div>
<div class="">/home/openvehicles/build/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/can/src/can.cpp:551</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">That is:</div>
<div class=""><br class="">
</div>
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class="">
<div class="">
<div class="">void
canbus::LogInfo(CAN_log_type_t type,
const char* text)</div>
<div class=""> {</div>
<div class=""> MyCan.LogInfo(this,
type, text); <—— HERE</div>
<div class=""> }</div>
</div>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">ELF is at:</div>
<div class=""><br class="">
</div>
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class="">
<div class=""><a
href="http://api.openvehicles.com/firmware/ota/v3.2/main/3.2.007.ovms3.elf"
class="" moz-do-not-send="true">http://api.openvehicles.com/firmware/ota/v3.2/main/3.2.007.ovms3.elf</a></div>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">Regards, Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 11 Dec 2019, at 5:20
PM, Michael Balzer <<a
href="mailto:dexter@expeedo.de"
class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8"
class="">
<div class="">
<div class="moz-cite-prefix">Mark,<br
class="">
<br class="">
Greg uses your build, the crash
point seems to be consistent, can
you post the a2l on this?<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
PS: Greg, would you mind switching
to EAP to beta test future
releases?<br class="">
<br class="">
<br class="">
Am 11.12.19 um 04:33 schrieb Greg
D.:<br class="">
</div>
<blockquote type="cite"
cite="mid:48cd3f20-409f-d201-640a-31b6e7788e92@gmail.com"
class="">
<pre class="moz-quote-pre" wrap="">Hi folks,
Well, the module updated to 3.2.007 last night. I just checked on it,
and it appears that it didn't exactly survive. Crashed while running
the autoconfig script. Log with two cycles attached.
I tried renaming the /store/events directory to /store/was_events since
it seems like Duktape was getting in the way, but that didn't resolve
the crash. I manually enabled wifi so I could manage the module, and
moved it back to 3.2.005 from the other partition. Seems stable again.
The car is going into the Service Center tomorrow (the perennial issue
with 1146 alerts), so I need to have the module stable so that I can
keep an eye on it. Going to leave it on 3.2.005 for now, unless someone
has a quick fix in the next few hours...
Otherwise, any ideas for troubleshooting after I get the car back back
(hopefully end of day)?
Greg
</pre>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<br class="">
<pre class="moz-signature" cols="144">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
</body>
</html>