<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr"><br></div><div dir="ltr">Nice catch. I had reviewed that code section this afternoon, but missed it.</div><div dir="ltr"><br></div><div dir="ltr">Shall we make a 3.3.002 and push it to EAP?</div><div dir="ltr"><br></div><div dir="ltr">Regards, Mark</div><div dir="ltr"><br><div dir="ltr"></div><blockquote type="cite">On 6 Mar 2022, at 5:58 PM, Michael Balzer <dexter@expeedo.de> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
I've found & fixed the bug: there was an endless loop in
modem::IncomingMuxData() when the NMEA channel had not been created.<br>
<br>
→
<a class="moz-txt-link-freetext" href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/a6e5b9ab957ab72f56740d6fd4ba484fe1691f80">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/a6e5b9ab957ab72f56740d6fd4ba484fe1691f80</a><br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 05.03.22 um 11:21 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite" cite="mid:2a67ff6a-00c9-03e4-5931-6e9be476adc9@expeedo.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
I've now tried starting the NMEA channel without actually sending
any GPS init commands (also no +CGPS=0).<br>
<br>
That inhibits the crashes, although apparently no NMEA sentences
are coming in. The only communication on MUX channel 1 (NMEA) are
some initial frames:<br>
<br>
<font face="monospace">balzer@leela:~/ovms/v3> grep -A1 CHAN=1
minicom.log<br>
V (26168) gsm-mux: ProcessFrame(CHAN=1, ADDR=07, CTRL=73,
FCS=15, LEN=6)<br>
I (26168) gsm-mux: Channel #1 is open<br>
--<br>
V (32568) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=56, LEN=20)<br>
V (32568) gsm-mux: ProcessFrame(CHAN=2, ADDR=09, CTRL=ff,
FCS=d1, LEN=20)<br>
--<br>
V (32938) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=b5, LEN=21)<br>
V (32938) gsm-mux: ProcessFrame(CHAN=2, ADDR=09, CTRL=ff,
FCS=32, LEN=21)<br>
--<br>
V (36948) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=56, LEN=20)<br>
V (36948) gsm-mux: ProcessFrame(CHAN=2, ADDR=09, CTRL=ff,
FCS=d1, LEN=20)<br>
--<br>
V (38968) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=b5, LEN=21)<br>
V (38968) gsm-mux: ProcessFrame(CHAN=2, ADDR=09, CTRL=ff,
FCS=32, LEN=21)<br>
--<br>
V (42638) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=b5, LEN=21)<br>
V (42638) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=f0, LEN=21)<br>
--<br>
V (44988) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=b5, LEN=21)<br>
V (44988) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=f0, LEN=21)<br>
</font><br>
<br>
<br>
Unrelated to this issue, but possibly to the framing errors: I've
also removed the NMEA log filter from modem::StandardLineHandler()
and found after normal GPS startup, the NMEA sentences always come
in three times via three MUX channels – additionally to channel 1
on channels 3 & 4:<br>
<br>
<font face="monospace">V (130148) gsm-mux: ProcessFrame(CHAN=1,
ADDR=05, CTRL=ff, FCS=bf, LEN=78)<br>
D (130148) gsm-nmea: Incoming RMC:
$GPRMC,095314.00,A,xxxx.139928,N,xxxx.391626,E,0.0,,050322,0.9,W,A*06<br>
V (130158) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=fa, LEN=78)<br>
D (130158) cellular: mux-rx-line #3: $GPRMC,095314.00,A,</font><font face="monospace"><font face="monospace">xxxx</font>.139928,N,</font><font face="monospace"><font face="monospace">xxxx</font>.391626,E,0.0,,050322,0.9,W,A*06<br>
V (130158) gsm-mux: ProcessFrame(CHAN=4, ADDR=11, CTRL=ff,
FCS=f7, LEN=78)<br>
D (130158) cellular: mux-rx-line #4: $GPRMC,095314.00,A,</font><font face="monospace"><font face="monospace">xxxx</font>.139928,N,</font><font face="monospace"><font face="monospace">xxxx</font>.391626,E,0.0,,050322,0.9,W,A*06<br>
V (130168) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=b1, LEN=82)<br>
D (130168) gsm-nmea: Incoming GNS: $GNGNS,095314.00,</font><font face="monospace"><font face="monospace">xxxx</font>.139928,N,</font><font face="monospace"><font face="monospace">xxxx</font>.391626,E,AAN,16,0.7,327.9,47.0,,,V*60<br>
V (130178) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=f4, LEN=82)<br>
D (130178) cellular: mux-rx-line #3: $GNGNS,095314.00,</font><font face="monospace"><font face="monospace">xxxx</font>.139928,N,</font><font face="monospace"><font face="monospace">xxxx</font>.391626,E,AAN,16,0.7,327.9,47.0,,,V*60<br>
V (130178) gsm-mux: ProcessFrame(CHAN=4, ADDR=11, CTRL=ff,
FCS=f9, LEN=82)<br>
D (130178) cellular: mux-rx-line #4: $GNGNS,095314.00,</font><font face="monospace"><font face="monospace">xxxx</font>.139928,N,</font><font face="monospace"><font face="monospace">xxxx</font>.391626,E,AAN,16,0.7,327.9,47.0,,,V*60</font><br>
<br>
I don't know if that's intended behaviour or can be changed, but I
suspect this to add unnecessary load to the serial communication.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 05.03.22 um 09:02 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite" cite="mid:1e6e7654-acf8-47a4-5c3b-4c4e1d0ab93b@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Another theory: I found Mark's comment in the SIM7600 driver
NMEA startup:<br>
<br>
// We need to do this a little differently from the standard,
as SIM7600<br>
// may start GPS on power up, and doesn't like us using
CGPS=1,1 when<br>
// it is already on. So workaround is to first CGPS=0.<br>
<br>
IOW, the SIM7600 may have GPS enabled by default, possibly also
sending NMEA sentences, if not explicitly switched off.<br>
<br>
BTW, that also explains the difficulties in booting a 3.3 module
from USB… :-/<br>
<br>
a) We don't switch off GPS if GPS is disabled.<br>
<br>
b) I just tried switching off GPS manually after a GPS-enabled
startup, and the modem still continued to send NMEA sentences:<br>
<br>
<font face="monospace">OVMS# cellular cmd AT+CGPS=0<br>
MODEM command has been sent.<br>
…<br>
D (475190) gsm-nmea: Incoming RMC: $GPRMC,,V,,,,,,,,,,N*53<br>
V (475190) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=c1, LEN=31)<br>
V (475190) gsm-mux: ProcessFrame(CHAN=4, ADDR=11, CTRL=ff,
FCS=cc, LEN=31)<br>
V (475190) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=60, LEN=32)<br>
D (475190) gsm-nmea: Incoming GNS: $GNGNS,,,,,,NNN,,,,,,*1D<br>
V (475190) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=25, LEN=32)<br>
V (475190) gsm-mux: ProcessFrame(CHAN=4, ADDR=11, CTRL=ff,
FCS=28, LEN=32)<br>
V (480180) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=84, LEN=31)<br>
D (480190) gsm-nmea: Incoming RMC: $GPRMC,,V,,,,,,,,,,N*53<br>
…<br>
OVMS# cellular cmd AT+CGPSNMEA=0<br>
MODEM command has been sent.<br>
D (505190) gsm-nmea: Incoming RMC: $GPRMC,,V,,,,,,,,,,N*53<br>
V (505190) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=c1, LEN=31)<br>
V (505190) gsm-mux: ProcessFrame(CHAN=4, ADDR=11, CTRL=ff,
FCS=cc, LEN=31)<br>
V (505190) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=60, LEN=32)<br>
D (505190) gsm-nmea: Incoming GNS: $GNGNS,,,,,,NNN,,,,,,*1D<br>
V (505190) gsm-mux: ProcessFrame(CHAN=3, ADDR=0d, CTRL=ff,
FCS=25, LEN=32)<br>
V (505190) gsm-mux: ProcessFrame(CHAN=4, ADDR=11, CTRL=ff,
FCS=28, LEN=32)<br>
V (510180) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=84, LEN=31)<br>
D (510190) gsm-nmea: Incoming RMC: $GPRMC,,V,,,,,,,,,,N*53<br>
…<br>
</font><br>
<br>
So without us starting the NMEA subsystem, is it possible these
transmissions are still there, filling up some MUX buffer?<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 05.03.22 um 08:33 schrieb
Michael Balzer:<br>
</div>
<blockquote type="cite" cite="mid:6502c120-1870-c1fc-ea8d-8ac13d2bbda4@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
I've tried to reproduce this with my old 3.2 & hand
soldered 3.3 dev module, with different vehicle modules and
modem GPS disabled.<br>
<br>
Results:<br>
- the vehicle module is irrelevant<br>
- 3.2 (SIM5360E) runs without issues (had this running over
night)<br>
- 3.3 (SIM7600G) consistently crashes by the TWDT within a
couple of minutes after modem init<br>
<br>
This is caused by the modem task hogging core 0 as soon as the
PPP connection is up:<br>
<br>
<font face="monospace">I (87161) cellular: Identified cellular
modem: SIM7600/Experimental support for SIMCOM SIM7600<br>
D (87161) cellular: Remove old 'auto' modem driver<br>
I (87161) cellular: Set modem driver to 'SIM7600'<br>
I (87161) cellular: State: Enter PoweredOn state<br>
OVMS# mo ta<br>
Number of Tasks = 18 Stack: Now Max Total Heap
32-bit SPIRAM C# PRI CPU% BPR/MH<br>
3FFAFB88 1 Blk esp_timer 436 708 4096
41648 644 44544 0 22 0% 22/ 0<br>
3FFC2BD8 2 Blk OVMS DukTape 500 9252 12288
188 0 524288 1 5 0% 5/ 0<br>
3FFC46B4 3 Blk eventTask 480 2032 4608
108 0 0 0 20 0% 20/ 0<br>
3FFC6B38 4 Blk OVMS Events 484 3204 8192
80136 0 79300 1 8 1% 8/ 0<br>
3FFC9C30 5 Blk OVMS CanRx 476 1996 4096
3240 0 50556 0 23 0% 23/ 0<br>
3FFCAA2C 6 Blk ipc0 424 504 1024
7804 0 0 0 24 0% 24/ 0<br>
3FFCB030 7 Blk ipc1 428 524 1024
120 0 0 1 24 0% 24/ 0<br>
<b>3FFCCEA0 10 Rdy IDLE0 412 508 1024
0 0 0 0 0 98% 0/ 0</b><br>
3FFCD438 11 Rdy IDLE1 404 580 1024
0 0 0 1 0 97% 0/ 0<br>
3FFCE1D0 12 Blk Tmr Svc 380 1404 3072
124 0 0 0 20 0% 20/ 0<br>
3FFCC308 17 Blk tiT 532 2036 3072
924 0 1040 1 18 0% 18/ 0<br>
<b>3FFD34E0 20 Blk OVMS Cellular 652 2396 4096
5848 0 0 0 20 0% 20/ 0</b><br>
3FFD656C 21 Blk wifi 472 2840 3584
38080 0 5632 0 22 1% 22/ 1<br>
3FFE3244 22 Blk OVMS Vehicle 496 496 8192
0 0 0 1 10 0% 10/ 0<br>
3FFE7BD4 23 Rdy OVMS Console 848 2688 10240 356
12100 15960 1 5 1% 5/ 1<br>
3FFEA5A0 24 Blk OVMS NetMan 908 2684 10240
832 0 2152 1 5 0% 5/ 0<br>
3FFEBBDC 25 Blk mdns 504 2200 4096
104 0 4 0 1 0% 1/ 0<br>
3FFEFDC8 26 Blk OVMS FileLog 580 1524 3072
36 0 0 1 2 0% 2/ 0<br>
<br>
…<br>
<br>
I (118171) cellular: Network Registration status:
RegisteredRoaming<br>
D (118181) cellular: mux-rx-line #3: +CCLK:
"22/03/05,08:22:23+04"<br>
D (118181) cellular: mux-rx-line #3: +CSQ: 24,0<br>
D (118181) cellular: mux-rx-line #3: +COPS: 0,0,"vodafone.de
Hologram",0<br>
I (118181) cellular: Network Provider is: vodafone.de
Hologram<br>
D (119151) cellular: State transition NetWait => NetStart<br>
I (119151) cellular: State: Enter NetStart state<br>
D (120151) cellular: Netstart
AT+CGDCONT=1,"IP","hologram";+CGDATA="PPP",1<br>
V (120191) gsm-mux: ProcessFrame(CHAN=0, ADDR=01, CTRL=ef,
FCS=79, LEN=11)<br>
V (120201) gsm-mux: ProcessFrame(CHAN=2, ADDR=09, CTRL=ff,
FCS=fb, LEN=24)<br>
D (120201) cellular: mux-rx-line #2: CONNECT 115200<br>
<b>I (120201) cellular: PPP Connection is ready to start</b><br>
V (120751) gsm-mux: ProcessFrame(CHAN=1, ADDR=05, CTRL=ff,
FCS=56, LEN=20)<br>
OVMS# mo ta<br>
Number of Tasks = 18 Stack: Now Max Total Heap
32-bit SPIRAM C# PRI CPU% BPR/MH<br>
3FFAFB88 1 Blk esp_timer 436 708 4096
41648 644 44544 0 22 0% 22/ 0<br>
3FFC2BD8 2 Blk OVMS DukTape 500 9252 12288
284 0 524308 1 5 1% 5/ 0<br>
3FFC46B4 3 Blk eventTask 480 2032 4608
108 0 0 0 20 0% 20/ 0<br>
3FFC6B38 4 Blk OVMS Events 484 3204 8192
80280 0 79680 1 8 1% 8/ 0<br>
3FFC9C30 5 Blk OVMS CanRx 476 1996 4096
3240 0 50556 0 23 0% 23/ 0<br>
3FFCAA2C 6 Blk ipc0 424 504 1024
7804 0 0 0 24 0% 24/ 0<br>
3FFCB030 7 Blk ipc1 428 524 1024
120 0 0 1 24 0% 24/ 0<br>
<b>3FFCCEA0 10 Rdy IDLE0 412 508 1024
0 0 0 0 0 79% 0/ 0</b><br>
3FFCD438 11 Rdy IDLE1 404 580 1024
0 0 0 1 0 95% 0/ 0<br>
3FFCE1D0 12 Blk Tmr Svc 380 1452 3072
124 0 0 0 20 0% 20/ 0<br>
3FFCC308 17 Blk tiT 532 2036 3072
924 0 1044 1 18 0% 18/ 0<br>
<b>3FFD34E0 20 Rdy OVMS Cellular 876 2716 4096
6116 0 10440 0 20 19% 20/ 0</b><br>
3FFD656C 21 Blk wifi 472 2840 3584
38080 0 5632 0 22 1% 22/ 1<br>
3FFE3244 22 Blk OVMS Vehicle 496 496 8192
0 0 0 1 10 0% 10/ 0<br>
3FFE7BD4 23 Rdy OVMS Console 848 2688 10240 356
12100 15960 1 5 3% 5/ 1<br>
3FFEA5A0 24 Blk OVMS NetMan 908 2684 10240
832 0 2152 1 5 0% 5/ 0<br>
3FFEBBDC 25 Blk mdns 504 2200 4096
104 0 4 0 1 0% 1/ 0<br>
3FFEFDC8 26 Blk OVMS FileLog 580 1524 3072
36 0 0 1 2 0% 2/ 0<br>
I (122161) housekeeping: System considered stable (RAM:
8b=64256-64760 32b=184 SPI=3362168-3374964)<br>
D (122231) ovms-duktape: Duktape: Compacting DukTape memory
done in 60 ms<br>
I (123151) ovms-server-v2: Send MP-0
h1,0,*-OVM-DebugCrash,0,2592000,3.3.001-285-g601f2a70/factory/edge
(build idf v3.3.4-848-g1ff5e24b1b Feb 22 2022
20:57:41),1,EarlyCrash,12,12,1,1,abort(),0,,0x4008ddca
0x4008e06<br>
5 0x400e88b8 0x40084176 ,6,Task watchdog,,,0,IDLE0,OVMS WIFI
BLE BT cores=2 rev=ESP32/3; MODEM SIM7600<br>
I (123361) ovms-server-v2: Incoming Msg: MP-0 h1<br>
OVMS# mo ta<br>
Number of Tasks = 18 Stack: Now Max Total Heap
32-bit SPIRAM C# PRI CPU% BPR/MH<br>
3FFAFB88 1 Blk esp_timer 436 708 4096
41648 644 44544 0 22 0% 22/ 0<br>
3FFC2BD8 2 Blk OVMS DukTape 500 9252 12288
188 0 524288 1 5 0% 5/ 0<br>
3FFC46B4 3 Blk eventTask 480 2032 4608
108 0 0 0 20 0% 20/ 0<br>
3FFC6B38 4 Blk OVMS Events 484 3204 8192
80136 0 79300 1 8 1% 8/ 0<br>
3FFC9C30 5 Blk OVMS CanRx 476 1996 4096
3240 0 50556 0 23 0% 23/ 0<br>
3FFCAA2C 6 Blk ipc0 424 504 1024
7804 0 0 0 24 0% 24/ 0<br>
3FFCB030 7 Blk ipc1 428 524 1024
120 0 0 1 24 0% 24/ 0<br>
<b>3FFCCEA0 10 Rdy IDLE0 412 508 1024
0 0 0 0 0 0% 0/ 0</b><br>
3FFCD438 11 Rdy IDLE1 404 580 1024
0 0 0 1 0 95% 0/ 0<br>
3FFCE1D0 12 Blk Tmr Svc 380 1452 3072
124 0 0 0 20 0% 20/ 0<br>
3FFCC308 17 Blk tiT 532 2036 3072
924 0 1044 1 18 0% 18/ 0<br>
<b>3FFD34E0 20 Rdy OVMS Cellular 876 2716 4096
6116 0 10440 0 20 98% 20/ 0</b><br>
3FFD656C 21 Blk wifi 472 2840 3584
38080 0 5632 0 22 1% 22/ 1<br>
3FFE3244 22 Blk OVMS Vehicle 496 496 8192
0 0 0 1 10 0% 10/ 0<br>
3FFE7BD4 23 Rdy OVMS Console 880 2688 10240 356
12100 15960 1 5 4% 5/ 1<br>
3FFEA5A0 24 Blk OVMS NetMan 908 2684 10240
832 0 2152 1 5 0% 5/ 0<br>
3FFEBBDC 25 Blk mdns 504 2200 4096
104 0 4 0 1 0% 1/ 0<br>
3FFEFDC8 26 Blk OVMS FileLog 580 1524 3072
36 0 0 1 2 0% 2/ 0<br>
<br>
…<br>
<br>
OVMS# E (181151) task_wdt: Task watchdog got triggered. The
following tasks did not reset the watchdog in time:<br>
E (181151) task_wdt: - IDLE0 (CPU 0)<br>
E (181151) task_wdt: Tasks currently running:<br>
E (181151) task_wdt: CPU 0: Tmr Svc<br>
E (181151) task_wdt: CPU 1: IDLE1<br>
E (181151) task_wdt: Aborting.</font><br>
<br>
<br>
No other log messages.<br>
<br>
<div class="moz-cite-prefix">Am 04.03.22 um 17:08 schrieb Jeff
Anton:<br>
</div>
<blockquote type="cite" cite="mid:54b0e703-d27c-91dd-6bd9-d61694906633@hesiod.org">I
woke from sleep thinking "interrupt storm." <br>
<br>
Craig's observation makes me wonder... Does the gps system
produce interrupts which are not handled properly when gps
is turned off. Unhandled interrupts might get constantly
repeated causing the IDLE tasks to starve? </blockquote>
<br>
I think that may be the case. Maybe the NMEA MUX channel gets
continuous input from the 7600 when GPS is disabled?<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">Am 04.03.22 um 23:15 schrieb
Craig Leres:<br>
</div>
<blockquote type="cite" cite="mid:46668c3c-b059-4122-012a-3533108ddc52@xse.com">One
more data point; I turned GPS off on my dev module (v3.1
board with a sim7600A running 3.3.001-285-g601f2a70-dirty)
and it started crashing. <br>
<br>
Has anyone outside the US tried this? <br>
<br>
Craig <br>
<br>
ice 12 % ./backtrace.sh 0x400891af:0x3ffb0690
0x40089449:0x3ffb06b0 0x400e8950:0x3ffb06d0
0x40083dde:0x3ffb06f0 <br>
+ xtensa-esp32-elf-addr2line -e build/ovms3.elf
0x400891af:0x3ffb0690 0x40089449:0x3ffb06b0
0x400e8950:0x3ffb06d0 0x40083dde:0x3ffb06f0 <br>
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/esp32/panic.c:736
<br>
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/esp32/panic.c:736
<br>
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/esp32/task_wdt.c:274
<br>
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/freertos/xtensa_vectors.S:1154
<br>
<br>
# esp/openvehicles-xtensa-esp32-elf/components/esp32/panic.c
<br>
733 <br>
734 void _esp_error_check_failed_without_abort(esp_err_t
rc, const char *file, int line, const char *function, const
char *expression) <br>
735 { <br>
736
esp_error_check_failed_print("ESP_ERROR_CHECK_WITHOUT_ABORT",
rc, file, line, function, expression); <br>
737 } <br>
<br>
_______________________________________________ <br>
OvmsDev mailing list <br>
<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
<span>_______________________________________________</span><br><span>OvmsDev mailing list</span><br><span>OvmsDev@lists.openvehicles.com</span><br><span>http://lists.openvehicles.com/mailman/listinfo/ovmsdev</span><br></div></blockquote></body></html>