<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div><blockquote type="cite" class=""><span style="font-family: Menlo-Regular; font-size: 14px;" class=""> But why are you seeing this crash now when we have</span><br style="font-family: Menlo-Regular; font-size: 14px;" class=""><span style="font-family: Menlo-Regular; font-size: 14px;" class="">not seen it before? </span></blockquote><div><br class=""></div><div>I haven’t really done much with that code path in the past. It is a high volume of data being uploaded to the module, and due to the way my code was handling it (badly), it resulted in a lot of realloc’s.</div><div><br class=""></div><div>For CRTD format, I read line-by-line. The issue is that the mongoose buffer presented may not have a complete line in it.</div><div><br class=""></div><div><ul class="MailOutline"><li class="">Mongoose was sending me a MG_EV_RECV with about 1KB in the buffer.</li><li class="">I removed about 30 bytes (one line), and returned.</li><li class="">That meant mongoose realloc’d down to 1KB - 30 bytes, then called me again. Rinse and repeat.</li></ul></div><div><br class=""></div><div>I first noticed this because SPI RAM memory consumption was going through the roof during retools canbus dump simulations. I was handling the incoming messages so slowly, that they were coming in faster than they were handled. Lesson learned: when mongoose tells you it has data, consume it otherwise mongoose will keep receiving (a sensible framework would back off the receiving if the consumer was not fast enough).</div><div><br class=""></div><div>My change now is to consume all I can on the MG_EV_RECV (in a while loop), but if there is half a line left over that gets left in the mongoose buffer and that causes a realloc.</div><div><br class=""></div><div>To get around this, I’ll have to do my own line buffering (so I can completely drain the mongoose buffer on MG_EV_RECV). A hassle, but I’ll try to get it done tonight. That will resolve my problem, but I don’t think it addresses the core mongoose heap corruption issue.</div><div><br class=""></div><div>My code is in retools.cpp re::MongooseHandler MG_EV_RECV part.</div><div><br class=""></div><div><blockquote type="cite" class=""><span style="font-family: Menlo-Regular; font-size: 14px;" class="">You say it still crashes, but you don't show a crash here?</span></blockquote></div><div><br class=""></div><div>I can get it to crash in retools. Repeatedly. Easy and every time.</div><div><br class=""></div><div><ol class="MailOutline"><li class="">Start a wifi</li><li class="">Start a vehicle mode, on can1 (maybe not necessary, but I do it anyway)</li><li class="">OVMS# re start</li><li class="">OVMS# re serve mode simulate</li><li class="">$ cat my-canbus-dump.crtd | nc TESTBOX.local 3000</li><li class="">Sit back and watch the crash</li></ol></div><div><br class=""></div><div>If you don’t have a CRTD file, just use this snippet:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><div>593.138 1R11 7E8 00 00 00 00 00 00 00 00</div><div>593.138 1R11 202 4D 5F 07 9D C4 09 3F 23</div><div>593.138 1R11 20A 00 01 01 83 D1 3E 00 00</div><div>593.138 1R11 228 40 C0 30 F8</div><div>593.138 1R11 6F2 1F 18 08 F9 A1 80 08 1F</div><div>593.138 1R11 212 D8 09 12 71 80</div></div></blockquote><div><br class=""></div><div>Make 1,000 copies of it, appending them all to one .crtd file. That can be your test data.</div><div><br class=""></div><div>So, I then wrote a ‘test realloc’ in test_framework.cpp to see if I could repeat it. I couldn’t. Which leads me to suspect the issue is in mongoose (or with the way I am only consuming part of the mongoose buffer).</div><div><br class=""></div><div>Regards, Mark.</div><div><br class=""><blockquote type="cite" class=""><div class="">On 26 Apr 2018, at 2:16 PM, Stephen Casner <<a href="mailto:casner@acm.org" class="">casner@acm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">On Thu, 26 Apr 2018, Mark Webb-Johnson wrote:<br class=""><br class=""><blockquote type="cite" class="">I’m seeing this also, when I stress-test by sending in a large<br class="">canbus dump into retools:<br class=""><br class="">CORRUPT HEAP: Bad head at 0x3f80e3c0. Expected 0xabba1234 got 0x3f80eb00<br class="">assertion "head != NULL" failed: file "/Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c", line 229, function: multi_heap_realloc<br class=""></blockquote><br class="">Indeed, this is the same error signature as what I was seeing, but the<br class="">reason I saw it was because of a mistake in my code. I had not<br class="">included the "if (size == 0)" test in ExternalRamRealloc(). Since<br class="">mongoose calls MG_REALLOC with size == 0, that caused<br class="">heap_caps_realloc() to do heap_caps_free() and return NULL, so my code<br class="">called realloc() on that same ptr that had just been freed.<br class=""><br class="">But before my commit this morning I fixed that bug by adding my own test for<br class="">size==0 so I could do my own heap_caps_free() and not call calling<br class="">heap_caps_realloc().<br class=""><br class="">So, as I stare at the code now, I don't see anything yet. Also, your<br class="">crash occurred when MG_REALLOC was called with nonzero size.<br class=""><br class=""><blockquote type="cite" class="">That code path does a lot of realloc’s.<br class=""><br class="">If I switch back to use normal malloc/calloc/realloc, it still crashes.<br class=""><br class="">If I disable the malloc #defined in mongoose mg_locals.h, it still crashes.<br class=""><br class="">The only way I can avoid this is to consume ALL the data on the<br class="">MG_EV_RECV event, which means realloc is presumably not called (so<br class="">often?) within mongoose.<br class=""><br class="">I assumed the conclusion is that fundamentally realloc is broken in<br class="">the ESP IDF version we use.<br class=""></blockquote><br class="">So, by switching back to the old code you've also verified that it was<br class="">not my new code. But why are you seeing this crash now when we have<br class="">not seen it before? Is there any way that my moving<br class="">ExternalRamMalloc() to a C namespace could cause trouble in some<br class="">allocation outside of mongoose leaving a bad heap for the mongoose<br class="">call to trip over?<br class=""><br class=""><blockquote type="cite" class="">I tried updating our ESP-IDF to espressif/master, but get compile<br class="">errors in build/heap:<br class=""><br class="">CC build/heap/multi_heap_poisoning.o<br class="">In file included from /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:27:0:<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:66:32: error: unknown type name 'TaskHandle_t'<br class=""> #define MULTI_HEAP_BLOCK_OWNER TaskHandle_t task;<br class=""> ^<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:50:5: note: in expansion of macro 'MULTI_HEAP_BLOCK_OWNER'<br class=""> MULTI_HEAP_BLOCK_OWNER<br class=""> ^<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'poison_allocated_region':<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:67:57: error: implicit declaration of function 'xTaskGetCurrentTaskHandle' [-Werror=implicit-function-declaration]<br class=""> #define MULTI_HEAP_SET_BLOCK_OWNER(HEAD) (HEAD)->task = xTaskGetCurrentTaskHandle()<br class=""> ^<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:71:5: note: in expansion of macro 'MULTI_HEAP_SET_BLOCK_OWNER'<br class=""> MULTI_HEAP_SET_BLOCK_OWNER(head);<br class=""> ^<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'multi_heap_get_block_owner':<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:68:42: warning: return makes pointer from integer without a cast [-Wint-conversion]<br class=""> #define MULTI_HEAP_GET_BLOCK_OWNER(HEAD) ((HEAD)->task)<br class=""> ^<br class="">/Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:284:12: note: in expansion of macro 'MULTI_HEAP_GET_BLOCK_OWNER'<br class=""> return MULTI_HEAP_GET_BLOCK_OWNER((poison_head_t*)multi_heap_get_block_address_impl(block));<br class=""> ^<br class="">cc1: some warnings being treated as errors<br class="">make[1]: *** [multi_heap_poisoning.o] Error 1<br class=""></blockquote><br class="">Those three macros are part of the code I provided in the pull<br class="">request, but in 4c532a59b2503dd8bc60d437bb65961676db9e55 we already<br class="">updated to a version of expressif/esp-idf master with the pull request<br class="">included. It seems that the references in the macros are to other<br class="">code in ESP-IDF that used to be valid but now may not be. Perhaps<br class="">they made a change but didn't test compilation with<br class="">CONFIG_HEAP_TASK_TRACKING=y?<br class=""><br class=""><blockquote type="cite" class="">I then tried:<br class=""><br class="">+CONFIG_HEAP_POISONING_DISABLED=y<br class="">+CONFIG_HEAP_POISONING_LIGHT=<br class="">-CONFIG_HEAP_TASK_TRACKING=y<br class=""><br class="">But no difference. Still crashes.<br class=""></blockquote><br class="">Do you mean that it then compiles OK, but when you run the stress test<br class="">it crashes? If it compiles OK, then my hypothesis that they made a<br class="">change and didn't test with CONFIG_HEAP_TASK_TRACKING=y might be<br class="">right.<br class=""><br class=""><blockquote type="cite" class="">Also tried:<br class=""><br class="">-CONFIG_FREERTOS_USE_TRACE_FACILITY=y<br class="">-CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS=<br class="">+CONFIG_FREERTOS_USE_TRACE_FACILITY=<br class=""><br class="">But no difference. Still crashes.<br class=""><br class="">Although not in a simply way:<br class=""><br class="">OVMS# module memory<br class="">Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4141392/4194252<br class=""><br class="">OVMS# test realloc<br class="">First check heap integrity...<br class="">Now allocate 4KB RAM...<br class="">Check heap integrity...<br class="">Now re-allocate bigger, 1,000 times...<br class="">Check heap integrity...<br class="">Now re-allocate smaller, 1,000 times...<br class="">Check heap integrity...<br class="">And free the buffer...<br class="">Final check of heap integrity…<br class=""><br class="">OVMS# module memory<br class="">Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4140892/4194252<br class=""><br class="">Maybe something inside mongoose itself? As that is the only place we are seeing this.<br class=""></blockquote><br class="">You say it still crashes, but you don't show a crash here?<br class=""><br class=""> -- Steve_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></div></div></blockquote></div><br class=""></body></html>