<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div>I’m seeing this also, when I stress-test by sending in a large canbus dump into retools:<div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">CORRUPT HEAP: Bad head at 0x3f80e3c0. Expected 0xabba1234 got 0x3f80eb00</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">assertion "head != NULL" failed: file "/Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c", line 229, function: multi_heap_realloc</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">abort() was called at PC 0x401015e7 on core 1</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x401015e7: __assert_func at /home/jeroen/esp8266/esp32/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:63 (discriminator 8)</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Backtrace: 0x40092cec:0x3ffe50f0 0x40092ee7:0x3ffe5110 0x401015e7:0x3ffe5130 0x4009294e:0x3ffe5160 0x40084e59:0x3ffe5180 0x400e4a51:0x3ffe51a0 0x400fb78a:0x3ffe51c0 0x400fb7da:0x3ffe51e0 0x400fb7f0:0x3ffe5200 0x400fe51a:0x3ffe5220 0x4013abd9:0x3ffe5260 0x4013c29e:0x3ffe52b0 0x4013c2b9:0x3ffe52f0</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x40092cec: invoke_abort at /Users/mark/esp/esp-idf/components/esp32/./panic.c:669</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x40092ee7: abort at /Users/mark/esp/esp-idf/components/esp32/./panic.c:669</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x401015e7: __assert_func at /home/jeroen/esp8266/esp32/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:63 (discriminator 8)</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x4009294e: multi_heap_realloc at /Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c:326</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x40084e59: heap_caps_realloc at /Users/mark/esp/esp-idf/components/heap/./heap_caps.c:388</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x400e4a51: ExternalRamRealloc at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/main/./ovms_malloc.c:65</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x400fb78a: mbuf_insert at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x400fb7da: mbuf_append at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x400fb7f0: mg_socket_if_tcp_send at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x400fe51a: mg_send at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">0x4013abd9: re::DoServe(CAN_frame_t*) at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/retools/src/retools.cpp:312</span></font></div></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">…</span></font></div></blockquote><div class=""><div><br class=""></div><div>That code path does a lot of realloc’s.</div><div><br class=""></div><div>If I switch back to use normal malloc/calloc/realloc, it still crashes.</div><div><br class=""></div><div>If I disable the malloc #defined in mongoose mg_locals.h, it still crashes.</div><div><br class=""></div><div>The only way I can avoid this is to consume ALL the data on the MG_EV_RECV event, which means realloc is presumably not called (so often?) within mongoose.</div><div><br class=""></div><div><div>I assumed the conclusion is that fundamentally realloc is broken in the ESP IDF version we use.</div><div class=""><br class=""></div></div><div><div>I tried updating our ESP-IDF to espressif/master, but get compile errors in build/heap:</div></div><div><br class=""></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">CC build/heap/multi_heap_poisoning.o</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">In file included from /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:27:0:</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:66:32: error: unknown type name 'TaskHandle_t'</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> #define MULTI_HEAP_BLOCK_OWNER TaskHandle_t task;</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> ^</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:50:5: note: in expansion of macro 'MULTI_HEAP_BLOCK_OWNER'</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> MULTI_HEAP_BLOCK_OWNER</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> ^</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'poison_allocated_region':</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:67:57: error: implicit declaration of function 'xTaskGetCurrentTaskHandle' [-Werror=implicit-function-declaration]</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> #define MULTI_HEAP_SET_BLOCK_OWNER(HEAD) (HEAD)->task = xTaskGetCurrentTaskHandle()</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> ^</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:71:5: note: in expansion of macro 'MULTI_HEAP_SET_BLOCK_OWNER'</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> MULTI_HEAP_SET_BLOCK_OWNER(head);</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> ^</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'multi_heap_get_block_owner':</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:68:42: warning: return makes pointer from integer without a cast [-Wint-conversion]</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> #define MULTI_HEAP_GET_BLOCK_OWNER(HEAD) ((HEAD)->task)</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> ^</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:284:12: note: in expansion of macro 'MULTI_HEAP_GET_BLOCK_OWNER'</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> return MULTI_HEAP_GET_BLOCK_OWNER((poison_head_t*)multi_heap_get_block_address_impl(block));</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""> ^</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">cc1: some warnings being treated as errors</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">make[1]: *** [multi_heap_poisoning.o] Error 1</span></font></div></div></div></blockquote><div class=""><div><br class=""></div><div>I then tried:</div><div><br class=""></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div><div>+CONFIG_HEAP_POISONING_DISABLED=y</div><div>+CONFIG_HEAP_POISONING_LIGHT=</div><div>-CONFIG_HEAP_TASK_TRACKING=y</div></div></div></blockquote><div class=""><div><br class=""></div><div><div>But no difference. Still crashes.</div><div class=""><br class=""></div></div><div>Also tried:</div><div><br class=""></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div><div>-CONFIG_FREERTOS_USE_TRACE_FACILITY=y</div><div>-CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS=</div><div>+CONFIG_FREERTOS_USE_TRACE_FACILITY=</div></div></div></blockquote><div class=""><div><br class=""></div><div><div>But no difference. Still crashes.</div><div class=""><br class=""></div></div><div><div>Although not in a simply way:</div><div><br class=""></div></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">OVMS# module memory</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4141392/4194252</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">OVMS# test realloc</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">First check heap integrity...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Now allocate 4KB RAM...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Check heap integrity...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Now re-allocate bigger, 1,000 times...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Check heap integrity...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Now re-allocate smaller, 1,000 times...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Check heap integrity...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">And free the buffer...</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Final check of heap integrity…</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class=""><br class=""></span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">OVMS# module memory</span></font></div><div><font face="Andale Mono" class=""><span style="font-size: 14px;" class="">Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4140892/4194252</span></font></div></div></div></blockquote><div class=""><div><div class=""><br class=""></div><div class="">Maybe something inside mongoose itself? As that is the only place we are seeing this.</div><div class=""><br class=""></div></div><div>Regards, Mark.</div><div><br class=""><blockquote type="cite" class=""><div class="">On 25 Apr 2018, at 3:18 PM, Stephen Casner <<a href="mailto:casner@acm.org" class="">casner@acm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Done, but now I get a corrupt heap abort. That does not make sense<br class="">because on my v3.0 hardware the new functions should boil down to<br class="">vanilla malloc, calloc, realloc. More tomorrow when I'm more awake.<br class=""><br class=""> -- Steve<br class=""><br class="">On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:<br class=""><br class=""><blockquote type="cite" class="">ok<br class=""><br class=""><br class=""><blockquote type="cite" class="">On 25 Apr 2018, at 2:34 PM, Stephen Casner <<a href="mailto:casner@acm.org" class="">casner@acm.org</a>> wrote:<br class=""><br class="">Mark,<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Presumably defining those in mg_locals.h? Then I guess<br class="">ovms.{h,cpp} needs our ExternalRamMalloc extended with<br class="">ExternalRamCalloc and ExternalRamRealloc.<br class=""></blockquote></blockquote><br class="">I already did this, but ran into the problem that ovms.{h,cpp} is C++<br class="">code, whereas mongoose is C. I started down a path of making<br class="">duplicate C code just for mongoose, but I was trying to make this<br class="">change without modifying mongoose.{h,c}. I think the right approach<br class="">would be to extract ExternalRamMalloc etc. into a separate pair of<br class="">files ovms_malloc.{h,c} that are C and then include them from ovms.h.<br class=""><br class=""> -- Steve<br class=""><br class="">On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?<br class=""></blockquote><br class="">Let me know if you want me to do this first. Should be relatively simple as ESP-IDF provides equivalent functions.<br class=""><br class="">Regards, Mark.<br class=""><br class=""><blockquote type="cite" class="">On 25 Apr 2018, at 1:30 PM, Mark Webb-Johnson <<a href="mailto:mark@webb-johnson.net" class="">mark@webb-johnson.net</a>> wrote:<br class=""><br class="">Steve,<br class=""><br class="">I see:<br class=""><br class="">#ifndef MG_MALLOC<br class="">#define MG_MALLOC malloc<br class="">#endif<br class=""><br class="">#ifndef MG_CALLOC<br class="">#define MG_CALLOC calloc<br class="">#endif<br class=""><br class="">#ifndef MG_REALLOC<br class="">#define MG_REALLOC realloc<br class="">#endif<br class=""><br class="">#ifndef MG_FREE<br class="">#define MG_FREE free<br class="">#endif<br class=""><br class="">Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.<br class=""><br class="">Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?<br class=""><br class="">Regards, Mark.<br class=""><br class=""><blockquote type="cite" class="">On 25 Apr 2018, at 1:20 PM, Stephen Casner <<a href="mailto:casner@acm.org" class="">casner@acm.org</a> <<a href="mailto:casner@acm.org" class="">mailto:casner@acm.org</a>>> wrote:<br class=""><br class="">Mark,<br class=""><br class="">OK, I'll change mongoose to explicitly allocate from SPIRAM when<br class="">available. I won't be able to completely test it, though, since I<br class="">only have v3.0.<br class=""><br class=""> -- Steve<br class=""><br class="">On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:<br class=""><br class=""><blockquote type="cite" class="">Steve,<br class=""><br class="">Because of things like this:<br class=""><br class=""><a href="https://github.com/espressif/esp-idf/issues/1492" class="">https://github.com/espressif/esp-idf/issues/1492</a> <<a href="https://github.com/espressif/esp-idf/issues/1492" class="">https://github.com/espressif/esp-idf/issues/1492</a>> <<a href="https://github.com/espressif/esp-idf/issues/1492" class="">https://github.com/espressif/esp-idf/issues/1492</a> <<a href="https://github.com/espressif/esp-idf/issues/1492" class="">https://github.com/espressif/esp-idf/issues/1492</a>>><br class=""><br class="">I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup...<br class=""><br class="">Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc).<br class=""><br class="">Last time I tried it (earlier this year), the ESP-IDF just simply didn't work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.<br class=""><br class="">I know Espressif have been working on it, and perhaps they've fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):<br class=""><br class=""><a href="https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4" class="">https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4</a> <<a href="https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4" class="">https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4</a>> <<a href="https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4" class="">https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4</a> <<a href="https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4" class="">https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4</a>>><br class=""><br class="">IDF 3.0 has officially been released last night.<br class=""><br class="">Regards, Mark.<br class=""><br class=""><blockquote type="cite" class="">On 25 Apr 2018, at 4:40 AM, Stephen Casner <<a href="mailto:casner@acm.org" class="">casner@acm.org</a> <<a href="mailto:casner@acm.org" class="">mailto:casner@acm.org</a>>> wrote:<br class=""><br class="">We've been dancing around the external RAM allocation decision with<br class="">our own ExtenalRamMalloc() function and extensions to the C++ new<br class="">allocator to try SPIRAM first but then back off to internal memory if<br class="">that fails (presumably because of v3.0 hardware rather than because<br class="">all 4MB of SPIRAM is used up).<br class=""><br class="">Do we need all that? There is already a mechanism in the multi-heap<br class="">memory system for the defaut malloc to try allocating from SPIRAM for<br class="">any request with size larger than malloc_alwaysinternal_limit which<br class="">can be set dynamically with heap_caps_malloc_extmem_enable(). If the<br class="">allocation from SPIRAM fails (again, presumably only on v3.0 hardware)<br class="">then the allocation is retried without the MALLOC_CAP_SPIRAM<br class="">capability, which is just what our ExternalRamMalloc does.<br class=""><br class="">If there are only a few allocations such as stacks that must come from<br class="">internal memory, and if those allocations (in system code) already use<br class="">heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then<br class="">we could probably just set malloc_alwaysinternal_limit to a small<br class="">number (perhaps 0) so that everything else comes from SPIRAM.<br class=""><br class="">I don't know if any of our application memory uses are sufficiently<br class="">sensitive to memory performance that we would want to ensure that the<br class="">memory is internal. My guess is that those would be few and therefore<br class="">it makes sense to take special action for them rather than for<br class="">everything else.<br class=""><br class="">For continued support of v3.0 hardware we could even test whether<br class="">SPIRAM is present and leave malloc_alwaysinternal_limit at its default<br class="">value of -1 if not.<br class=""><br class="">Opinions?<br class=""><br class=""> -- Steve<br class=""><br class="">On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:<br class=""><br class=""><blockquote type="cite" class="">Michael,<br class=""><br class="">Do we need ovms_extram.h? Can we just put the namespace directly<br class="">into ovms.h (where all the other external ram allocation stuff is)?<br class=""><br class="">Regards, Mark<br class=""></blockquote>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a> <<a href="mailto:OvmsDev@lists.openvehicles.com" class="">mailto:OvmsDev@lists.openvehicles.com</a>><br class=""><a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class=""></blockquote><br class=""></blockquote>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a> <<a href="mailto:OvmsDev@lists.openvehicles.com" class="">mailto:OvmsDev@lists.openvehicles.com</a>><br class=""><a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class=""></blockquote><br class=""></blockquote><br class=""></blockquote>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></blockquote><br class="">_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""><br class=""><br class=""></blockquote>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></div></div></blockquote></div><br class=""></div></body></html>