<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
A slight improvement in terms of stability.<br>
<br>
Crash numbers have decreased a bit, but better yet, the crash
"quality" has improved. Backtraces make sense again.<br>
<br>
Some new backtraces let me check the notification concurrency
handling again, and I have found and resolved two more issues.<br>
<br>
I haven't seen any websocket data corruptions with the new version,
but the overall wifi performance has clearly degraded.<br>
<br>
One user reported wifi connections previously working very reliably
now sometimes drop without apparent reason.<br>
<br>
Overall I think we should keep the workaround and hope Espressif
will solve this asap.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 31.01.19 um 01:50 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite"
cite="mid:50E32417-69AA-4492-9029-B5B468CAAC3A@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Yep, I agree. It looks like core #0 is using the top 2MB and core
#1 the bottom 2MB.
<div class=""><br class="">
</div>
<div class="">The question is will we see more problems with this
workaround than without it?</div>
<div class=""><br class="">
</div>
<div class="">I doubt if anything will come out of Espressif China
for several weeks. Chinese New Year is here now and everything
is shutting down for the break. Factories will be down for a
month - businesses perhaps a fortnight.</div>
<div class=""><br class="">
</div>
<div class="">At least we have confirmed that it is the same issue
as:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class="">
<h1 class="gh-header-title" style="box-sizing: border-box;
font-size: 32px; margin: 0px 150px 0px 0px; font-weight:
400; line-height: 1.125; overflow-wrap: break-word; color:
rgb(36, 41, 46); font-family: -apple-system, system-ui,
"Segoe UI", Helvetica, Arial, sans-serif,
"Apple Color Emoji", "Segoe UI Emoji",
"Segoe UI Symbol"; font-variant-ligatures: normal;
orphans: 2; widows: 2; background-color: rgb(255, 255,
255);"><span class="js-issue-title" style="box-sizing:
border-box;">[TW#28180] PSRAM Cache Issue stills exist </span><span
class="gh-header-number" style="box-sizing: border-box;
color: rgb(163, 170, 177); letter-spacing: -1px;">#2892</span></h1>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Regards, Mark<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 31 Jan 2019, at 5:36 AM, Michael Balzer
<<a href="mailto:dexter@expeedo.de" class=""
moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""> Comparison
made, difference confirmed. Load times:<br class="">
<blockquote class=""><b class=""><tt class="">Pre-Workaround:
</tt></b><br class="">
<tt class="">firmware 32.7 KB 5.08 s</tt><br
class="">
<tt class="">firmware 32.7 KB 4.96 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.08 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.06 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.09 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.17 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.00 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.06 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.04 s</tt><br
class="">
<tt class="">firmware 32.7 KB 4.96 s</tt><br
class="">
<br class="">
<b class=""><tt class="">Post-Workaround: </tt></b><br
class="">
<tt class="">firmware 32.7 KB 6.99 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.05 s</tt><br
class="">
<tt class="">firmware 32.7 KB 4.83 s</tt><br
class="">
<tt class="">firmware 32.7 KB 6.42 s</tt><br
class="">
<tt class="">firmware 32.7 KB 6.50 s</tt><br
class="">
<tt class="">firmware 32.7 KB 6.49 s</tt><br
class="">
<tt class="">firmware 32.7 KB 6.40 s</tt><br
class="">
<tt class="">firmware 32.7 KB 4.94 s</tt><br
class="">
<tt class="">firmware 32.7 KB 5.17 s</tt><br
class="">
<tt class="">firmware 32.7 KB 6.54 s</tt><br
class="">
</blockquote>
<br class="">
This looks like the bug hits the wifi or lwip stack now,
causing frequent retransmits of corrupted packets.<br
class="">
<br class="">
I just also tried running the test project with that 2
MB allocation and starting the test task on core 0: the
corruptions show again. So it's reasonable to assume
core 0 can only work with the lower 2 MB and core 1 only
with the higher 2 MB.<br class="">
<br class="">
Would it be an option to split the SPI RAM into two
heaps and use each depending on the core requesting
memory?<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 30.01.19 um 21:01
schrieb Michael Balzer:<br class="">
</div>
<blockquote type="cite"
cite="mid:0c443a8b-be03-6268-c7bf-170a803a67a7@expeedo.de"
class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
Is it just my impression or has the wifi speed
decreased with that workaround?<br class="">
<br class="">
It's especially apparent on the firmware version info.
That's just 23K, but takes some seconds to load. It
was slow before though, I need to make a comparison.<br
class="">
<br class="">
I hope the workaround doesn't shift the issue to core
0 (running the wifi stack)…<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 30.01.19 um 19:19
schrieb Michael Balzer:<br class="">
</div>
<blockquote type="cite"
cite="mid:98bbdf3e-1fb3-c8f5-c228-ec41c3792360@expeedo.de"
class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
Rolled out to "my" edgers. I'll watch crash counters
during the next days and report.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 30.01.19 um 12:44
schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:822711AA-42DE-4A94-93C7-B247C749FAB2@webb-johnson.net"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
Yes, I suspect there is something in there related
to the core. I worry that (a) it is not as simple
as 2MB/2MB, and (b) we run code on both cores
anyway. We may have just fixed the test case
(which runs on core #1), but not code running on
core #0.
<div class=""><br class="">
</div>
<div class="">Anyway, it is worth a try in the
wild to see if it reduces the volume of errors,
while Espressif continue to work on it.</div>
<div class=""><br class="">
</div>
<div class="">Kludgy wonderfulness is in commit
3.2.001-29-g311307cd.</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 30 Jan 2019, at 7:39 PM,
Michael Balzer <<a
href="mailto:dexter@expeedo.de" class=""
moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8"
class="">
<div text="#000000" bgcolor="#FFFFFF"
class="">
<div class="moz-cite-prefix">Oh my…<br
class="">
<br class="">
Can it be the first 2 MB are taboo for
core 1 and the second 2 MB for core 0?<br
class="">
<br class="">
Anyway, we won't need those 2 MB for
now, and hopefully this will be fixed
by Espressif soon.<br class="">
<br class="">
So I'd say yes, commit.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 30.01.19 um 12:26 schrieb Mark
Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:2F06DE6F-A324-4137-90C7-B7450C61A0BA@webb-johnson.net"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8"
class="">
I tried changing CONFIG_SPIRAM_SIZE in
sdkconfig directly, but it just reset
back to the default 4MB.
<div class=""><br class="">
</div>
<div class="">So changed esp-idf
itself, components/esp32/Kconfig,
and added a prompt for that config
option. I could then change it in
menuconfig and the change persisted.</div>
<div class=""><br class="">
</div>
<div class="">I then tested:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0
40px;
 border:
 none;

padding:
 0px;" class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (0) cpu_start:
App cpu up.</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (715) spiram: SPI
SRAM memory test OK</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (716) heap_init:
Initializing. RAM available
for dynamic allocation:</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (716) heap_init:
At 3FFAE6E0 len 00001920 (6
KiB): DRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (723) heap_init:
At 3FFBA920 len 000256E0
(149 KiB): DRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (728) heap_init:
At 3FFE0440 len 00003BC0 (14
KiB): D/IRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (735) heap_init:
At 3FFE4350 len 0001BCB0
(111 KiB): D/IRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (742) heap_init:
At 400953CC len 0000AC34 (43
KiB): IRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (747) cpu_start:
Pro cpu start user code</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (752) spiram:
Adding pool of 2048K of
external SPI memory to heap
allocator</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (158) cpu_start:
Starting scheduler on PRO
CPU.</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (0) cpu_start:
Starting scheduler on APP
CPU.</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">This is ESP32 chip
with 2 CPU cores,
WiFi/BT/BLE, silicon
revision 1, 16MB external
flash</span></font></div>
</div>
<div class=""><font class=""
face="Andale Mono"><span
style="font-size: 14px;"
class="">…</span></font></div>
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (9566) test_task:
testing mode 1 loopcnt 1000</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">E (9576) test_task:
loop #1: corruption detected
at pos=497 size=498</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019 15:08:23</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">#1: size=498, data
@0x3f901ca0</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019 15:08:23</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">E (9806) test_task:
loop #5: corruption detected
at pos=497 size=498</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019 15:08:23</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">E (10036)
test_task: loop #9:
corruption detected at
pos=829 size=830</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019 15:08:23</span></font></div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">which looks like a
FAIL to me?</div>
<div class=""><br class="">
</div>
<div class="">So, then I tried
something a little funky:</div>
<div class=""><br class="">
</div>
<div class="">
<ol class="MailOutline">
<li class="">Go back to
normal CONFIG_SPIRAM_SIZE 4MB<br
class="">
<br class="">
</li>
<li class="">Add this
to main/test_memory_corruption.cpp
app_main():<br class="">
<br class="">
<font class="" face="Andale
Mono"><span
style="font-size:

14px;" class="">void*
big2mb =
heap_caps_malloc(1024*1024*2,
MALLOC_CAP_SPIRAM);<br
class="">
printf("Big 2MB spi memory
allocation is at
%p\n",big2mb);</span></font><br
class="">
<br class="">
</li>
<li class="">Rebuild and run...</li>
</ol>
</div>
</div>
<div class=""><br class="">
</div>
<div class="">Here is what I get:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0
40px;
 border:
 none;

padding:
 0px;" class="">
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (0) cpu_start:
App cpu up.</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1151) spiram:
SPI SRAM memory test OK</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1152)
heap_init: Initializing.
RAM available for dynamic
allocation:</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1152)
heap_init: At 3FFAE6E0 len
00001920 (6 KiB): DRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1159)
heap_init: At 3FFBA920 len
000256E0 (149 KiB): DRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1164)
heap_init: At 3FFE0440 len
00003BC0 (14 KiB): D/IRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1171)
heap_init: At 3FFE4350 len
0001BCB0 (111 KiB): D/IRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1179)
heap_init: At 400953CC len
0000AC34 (43 KiB): IRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1184)
cpu_start: Pro cpu start
user code</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1188) spiram:
Adding pool of 4096K of
external SPI memory to
heap allocator</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (206)
cpu_start: Starting
scheduler on PRO CPU.</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (0) cpu_start:
Starting scheduler on APP
CPU.</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">...</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">This is ESP32
chip with 2 CPU cores,
WiFi/BT/BLE, silicon
revision 1, 16MB external
flash</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">Big 2MB spi
memory allocation is at
0x3f800028</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">...</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (291) wifi
station:
ESP_WIFI_MODE_APSTA</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (291) wifi:
wifi driver task:
3ffc25c4, prio:23,
stack:3584, core=0</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">...</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (3281) wifi
station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (3281) wifi
station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (6111) wifi
station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (6111) wifi
station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (8461)
mem_stresser: started on
core 0 mem @0x3fa01b98</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (8461)
mem_stresser: started on
core 1 mem @0x3fa81bac</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (8951) wifi
station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (8951) wifi
station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (9461)
test_task: running on core
1</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (9561)
test_task: testing mode 1
loopcnt 1000</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">#1: size=1039,
data @0x3fb01ca4</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty
Jan 27 2019
15:08:23)4348864OVMS WIFI
BLE BT cores=2 rev=ESP32/1</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (11781) wifi
station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (11781) wifi
station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (14621) wifi
station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (14621) wifi
station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (17451) wifi
station: connect to the AP
fail</span></font></div>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">I left it running
for a few minutes, but no faults
shown. Modes 1 through 4 all
clean.</div>
<div class=""><br class="">
</div>
<div class="">So, to take the
funkiness to the next level, I
added the same to ovms_main,
right at the start (actually in
the log level setter, as that
seemed to be the first thing
that initialises). Somebody else
gets in for 28bytes of SPIRAM
(or maybe that is a structure
header), but anyway...</div>
<div class=""><br class="">
</div>
</div>
</div>
<blockquote style="margin: 0 0 0
40px;
 border:
 none;

padding:
 0px;" class="">
<div class="">
<div class="">
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1742)
spiram: SPI SRAM memory
test OK</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1743)
heap_init: Initializing.
RAM available for
dynamic allocation:</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1743)
heap_init: At 3FFAFF10
len 000000F0 (0 KiB):
DRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1750)
heap_init: At 3FFCEDE8
len 00011218 (68 KiB):
DRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1755)
heap_init: At 3FFE0440
len 00003BC0 (14 KiB):
D/IRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1762)
heap_init: At 3FFE4350
len 0001BCB0 (111 KiB):
D/IRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1768)
heap_init: At 4009F530
len 00000AD0 (2 KiB):
IRAM</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1775)
cpu_start: Pro cpu start
user code</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (1779)
spiram: Adding pool of
4096K of external SPI
memory to heap allocator</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (125)
ovms_main: Set default
logging level for * to
INFO</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">I (126)
ovms_main: Pre-allocated
2MB of SPIRAM at
0x3f800028 (workaround
ESP32 bug)</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size: 14px;"
class="">...</span></font></div>
<div class="">
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">I
(17783) webserver:
Launching Web Server</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">I
(17823) ssh: Launching
SSH Server</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">OVMS#
test string 500 2</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">#1:
stdlen = 1027</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">1985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty
Jan 30 2019
19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty
Jan 30 2019
19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty
Jan 30 2019
19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty
Jan 30 2019
19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty
Jan 30 2019
19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty
Jan 30 2019
19:17:20)012.61985504</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class=""><br
class="">
</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">finished</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">OVMS#</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class=""><br
class="">
</span></font></div>
<div class=""><font class=""
face="Andale
 Mono"><span
style="font-size:
14px;" class="">OVMS#
module memory<br
class="">
Free 8-bit
76160/230212, 32-bit
812/2720, SPIRAM
1886108/4194252<br
class="">
--Task-- Total
DRAM D/IRAM IRAM
SPIRAM +/- DRAM
D/IRAM IRAM SPIRAM<br
class="">
main*
8368 0 0
8456 +8368 +0
+0 +8456<br
class="">
no task*
5376 0 0
0 +5376 +0
+0 +0<br
class="">
esp_timer
34808 0
6442123100 +34808
+0 +644+2123100<br
class="">
OVMS Events
36352 41040 0
35120 +36352
+41040 +0 +35120<br
class="">
OVMS DukTape
188 0 0
34668 +188
+0 +0 +34668<br
class="">
OVMS CanRx
3124 0 0
26168 +3124
+0 +0 +26168<br
class="">
ipc0
7776 0 0
0 +7776 +0
+0 +0<br
class="">
ipc1
12 0 0
0 +12 +0
+0 +0<br
class="">
Tmr Svc
0 88 0 0
+0 +88
+0 +0<br class="">
tiT
232 468 0
664 +232 +468
+0 +664<br
class="">
wifi
600 8896 0
3324 +600 +8896
+0 +3324<br
class="">
OVMS Console
0 200 1180
26332 +0
+200 +1180 +26332<br
class="">
mdns
0 104 0
4 +0 +104
+0 +4<br
class="">
OVMS NetMan
0 216 0 84
+0 +216
+0 +84</span></font></div>
</div>
</div>
</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">I think that wins as
the most ugly kludge to date.</div>
<div class=""><br class="">
</div>
<div class="">Shall I commit?</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 30 Jan 2019, at
5:00 PM, Mark Webb-Johnson
<<a
href="mailto:mark@webb-johnson.net"
class=""
moz-do-not-send="true">mark@webb-johnson.net</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<meta
http-equiv="Content-Type"
content="text/html;
charset=UTF-8" class="">
<div style="word-wrap:

break-word;



-webkit-nbsp-mode:

space;


line-break:

after-white-space;" class=""><br
class="">
<div class=""><br class="">
<blockquote type="cite"
class="">
<div class="">On 30 Jan
2019, at 4:41 PM,
Michael Balzer <<a
href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<meta
http-equiv="Content-Type"
content="text/html;
charset=UTF-8"
class="">
<div text="#000000"
bgcolor="#FFFFFF"
class="">
<div
class="moz-cite-prefix">Mark,<br
class="">
<br class="">
issue #2892
mentions using
only the lower 2
MB of SPI RAM as a
workaround. Where
are our
allocations placed
by the allocator?
Does it fill from
the middle or end?
If not we don't
use the upper half
yet.<br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
I saw that, and had a
look. I don’t see anything
in makeconfig to limit to
2MB (not 4MB).</div>
<div class=""><br class="">
</div>
<div class="">I do see
a CONFIG_SPIRAM_SIZE=4194304
in sdkconfig, but not sure
how it gets there. Perhaps
we can just change there
to 2MB and try?</div>
<div class=""><br class="">
</div>
<div class="">I did see this
in the documentation:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0
0 0

 40px;

border: none;


padding:
 0px;"
class="">
<div class=""><span
style="caret-color:


rgb(64,
 64,

64); color:

rgb(64,

 64,
64);

font-family:

Lato,



proxima-nova,


"Helvetica

 Neue",
 Arial,


sans-serif;


font-size:

16px;



background-color:


rgb(252,

252,
 252);"
class="">During
ESP-IDF startup,
external RAM is mapped
into the data address
space starting at at
address 0x3F800000
(byte-accessible). The
length of this region
is the same as the
SPIRAM size (up to the
limit of 4MiB).</span></div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">So, maybe we
can look at the address
our allocations come from
to see their offset from
0x3F800000? If it is
coming from the top 2MB,
then perhaps we start with
a big 2MB allocation that
we never use?</div>
<div class=""> <br class="">
<blockquote type="cite"
class="">
<div class="">
<div text="#000000"
bgcolor="#FFFFFF"
class="">
<div
class="moz-cite-prefix">A
possible
workaround for the
most apparent
issues with this
(string assembly)
could be to use
char buffers
instead of
std::string. I was
thinking about
doing that at
least for the
websocket stream.<br
class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">It seems
strange that we are only
seeing it with
std::string. Maybe that
just stresses the system
more, or does a lot of
reallocations?</div>
<br class="">
<blockquote type="cite"
class="">
<div class="">
<div text="#000000"
bgcolor="#FFFFFF"
class="">
<div
class="moz-cite-prefix">
But if this really
is a hardware
issue it can
affect all objects
in SPI RAM,
appending to
std::string then
only triggers this
more often.<br
class="">
<br class="">
Another workaround
could be to run
everything on core
0.<br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Urgh.</div>
<div class=""><br class="">
<blockquote type="cite"
class="">
<div class="">
<div text="#000000"
bgcolor="#FFFFFF"
class="">
<div
class="moz-cite-prefix">Regards,<br
class="">
Michael<br
class="">
<br class="">
<br class="">
Am 30.01.19 um
03:33 schrieb Mark
Webb-Johnson:<br
class="">
</div>
<blockquote
type="cite"
cite="mid:4737F304-A344-4BD8-9B0C-95366DB7A40F@webb-johnson.net"
class="">
<meta
http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
Michael,
<div class=""><br
class="">
</div>
<div class="">Espressif’s
response (and
linking to that
other issue)
sounds like
others are
seeing this.</div>
<div class=""><br
class="">
</div>
<div class="">Lousy
timing (with
Chinese New Year
next week), so
don’t expect
anything quick
from Espressif.
I guess we’ll
just have to
live with it
until they can
find a
workaround. It
sounds like the
issue is at the
hardware level
and a compiler
patch will be
needed.</div>
<div class=""><br
class="">
</div>
<div class="">Regards,
Mark.<br
class="">
<div class=""><br
class="">
<blockquote
type="cite"
class="">
<div class="">On
30 Jan 2019,
at 3:07 AM,
Michael Balzer
<<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<meta
http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
<div
text="#000000"
bgcolor="#FFFFFF" class=""> → <a class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/issues/3006"
moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/3006</a><br
class="">
<br class="">
Regards,<br
class="">
Michael<br
class="">
<br class="">
<br class="">
<div
class="moz-cite-prefix">Am
28.01.19 um
20:46 schrieb
Michael
Balzer:<br
class="">
</div>
<blockquote
type="cite"
cite="mid:b193b172-9a83-6df5-7b61-9e0cbfa46281@expeedo.de"
class="">
<meta
http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
To clarify:
the bug is
most likely
not restricted
to the case of
building a
message in a
buffer. It can
possibly cause
corruptions in
any RAM
section, so
can well be
responsible
for many/most
of the
unidentified
crashes and
stack/heap
corruptions
we're
experiencing.<br
class="">
<br class="">
Regards,<br
class="">
Michael<br
class="">
<br class="">
<br class="">
<div
class="moz-cite-prefix">Am
28.01.19 um
20:43 schrieb
Michael
Balzer:<br
class="">
</div>
<blockquote
type="cite"
cite="mid:06d0ea65-f4d5-e2c5-9ca1-b0ad63bfa3b1@expeedo.de"
class="">
<meta
http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
For those not
following the
github
discussion:
I'm pretty
sure I've
nailed the bug
down.<br
class="">
<br class="">
I have
reproduced the
bug in a
simple test
project and
intend to
raise an issue
with Espressif
on this.<br
class="">
<br class="">
<a
class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457965435"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457965435</a><br
class="">
<br class="">
If you'd like
to test this
on your
module,
configure the
project to
your wifi
credentials,
then use "make
flash". As the
test project
is small, this
normally will
not erase your
OVMS config
partition, but
a backup is
always
recommended.<br
class="">
<br class="">
Regards,<br
class="">
Michael<br
class="">
<br class="">
<br class="">
<div
class="moz-cite-prefix">Am
24.01.19 um
21:19 schrieb
Michael
Balzer:<br
class="">
</div>
<blockquote
type="cite"
cite="mid:e4159a1c-b283-25b0-f1b5-ce570b529a1c@expeedo.de"
class="">
<meta
http-equiv="content-type"
content="text/html; charset=UTF-8" class="">
Everyone
please have a
look at…<br
class="">
<br class="">
<a
class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457334248"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457334248</a><br
class="">
<br class="">
Please try to
reproduce the
bug on your
modules.<br
class="">
<br class="">
I'm open for
explanations.<br
class="">
<br class="">
I thought this
might be some
copy-on-write
bug with
std::string,
but the gcc
5.x libstdc++
does no longer
use that
implementation
(wouldn't be
C++11
compliant as
well). I also
tried moving
all strings to
temporary
buffers, but
modes 5 &
6 eliminated
this
explanation as
well.<br
class="">
<br class="">
My remaining
theories:<br
class="">
<ul class="">
<li class="">A
task writing
out of bounds
(but only
0-bytes?)</li>
<li class="">A
hardware issue
only affecting
some modules<br
class="">
</li>
</ul>
A hardware
issue only
affecting some
percentage of
ESP32 could
explain this
as well as the
strange heap
corruptions
that seem to
affect some
modules
especially
often.<br
class="">
<br class="">
Regards,<br
class="">
Michael<br
class="">
<br class="">
</blockquote>
</blockquote>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class="">
<pre class="moz-signature" cols="144">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</body>
</html>