<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div>Not sure if it is related, but I am hitting this with my work on canlog. I’m seeing:<div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">0x400f7af1: CheckQueueOverflow(char const*, char*) at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_events.cpp:381</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">ELF file SHA256: 57f584b0e3fe542a</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">Backtrace: 0x4008e6f3:0x3ffcce00 0x4008e95d:0x3ffcce20 0x400f7af1:0x3ffcce40 0x400f8051:0x3ffcce80 0x400f91d0:0x3ffcceb0 0x40092387:0x3ffccf00</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">0x4008e6f3: invoke_abort at /Users/mark/esp/esp-idf/components/esp32/panic.c:736</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">0x4008e95d: abort at /Users/mark/esp/esp-idf/components/esp32/panic.c:736</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">0x400f7af1: CheckQueueOverflow(char const*, char*) at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_events.cpp:381</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">0x400f8051: OvmsEvents::SignalEvent(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void*, void (*)(char const*, void*), unsigned int) at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_events.cpp:459</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">0x400f91d0: HousekeepingTicker1(void*) at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_housekeeping.cpp:122</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""><br class=""></span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class="">0x40092387: prvProcessExpiredTimer at /Users/mark/esp/esp-idf/components/freertos/timers.c:485</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""> (inlined by) prvProcessTimerOrBlockTask at /Users/mark/esp/esp-idf/components/freertos/timers.c:571</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 14px;" class=""> (inlined by) prvTimerTask at /Users/mark/esp/esp-idf/components/freertos/timers.c:544</span></font></div></div></blockquote><div class=""><div><br class=""></div><div>So that is the CheckQueueOverflow() system within events, that handles when an event cannot be queued. In this case it is aborting the system (to cause a restart). Not a watchdog timer.</div><div><br class=""></div><div>In my case, using gdb stubs gets into the debugger nicely. But I get:</div><div><br class=""></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><div>Info thread gives:</div><div><br class=""></div><div>…</div><div> 3 Thread 2 (OVMS Events CPU1) 0x40081a28 in esp_crosscore_int_send_yield (core_id=1)</div><div> at /Users/mark/esp/esp-idf/components/esp32/crosscore_int.c:112</div><div> 2 Thread 1 (OVMS NetMan CPU1) 0x7f400992 in ?? ()</div><div>* 1 Remote target 0x4008e6f3 in invoke_abort () at /Users/mark/esp/esp-idf/components/esp32/panic.c:156(gdb) thread 1</div><div><br class=""></div><div>and:</div><div><br class=""></div><div>[Switching to thread 1 (Remote target)]</div><div>#0 0x4008e6f3 in invoke_abort () at /Users/mark/esp/esp-idf/components/esp32/panic.c:156</div><div>156 *((int *) 0) = 0;</div><div><br class=""></div><div>(gdb) bt</div><div>#0 0x4008e6f3 in invoke_abort () at /Users/mark/esp/esp-idf/components/esp32/panic.c:156</div><div>#1 0x4008e960 in abort () at /Users/mark/esp/esp-idf/components/esp32/panic.c:171</div><div>Backtrace stopped: previous frame identical to this frame (corrupt stack?)</div><div><br class=""></div><div><div>[Switching to thread 2 (Thread 1)]<span class="Apple-tab-span" style="white-space:pre"> </span>MWJ: This is NETMAN</div><div>#0 0x7f400992 in ?? ()</div><div><br class=""></div></div><div><div>(gdb) bt</div><div>#0 0x7f400992 in ?? ()</div><div>#1 0x7ffe6583 in ?? ()</div><div>Backtrace stopped: previous frame identical to this frame (corrupt stack?)</div><div><br class=""></div><div><div>(gdb) thread 3<span class="Apple-tab-span" style="white-space:pre"> </span>MWJ: This is EVENTS</div><div>[Switching to thread 3 (Thread 2)]</div><div>#0 0x40081a28 in esp_crosscore_int_send_yield (core_id=1) at /Users/mark/esp/esp-idf/components/esp32/crosscore_int.c:112</div><div>112<span class="Apple-tab-span" style="white-space:pre"> </span> esp_crosscore_int_send(core_id, REASON_YIELD);</div><div>(gdb) bt</div><div>#0 0x40081a28 in esp_crosscore_int_send_yield (core_id=1) at /Users/mark/esp/esp-idf/components/esp32/crosscore_int.c:112</div><div>#1 0x4009030c in xQueueGenericReceive (xQueue=0x3ffc01cc, pvBuffer=0x3ffc23f0, xTicksToWait=500, xJustPeeking=0)</div><div> at /Users/mark/esp/esp-idf/components/freertos/queue.c:1592</div><div>#2 0x400f85c2 in OvmsEvents::EventTask (this=0x3ffb5dec <MyEvents>)</div><div> at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_events.cpp:213</div><div>#3 0x400f8660 in EventLaunchTask (pvParameters=0x3ffb5dec <MyEvents>)</div><div> at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_events.cpp:80</div><div>Backtrace stopped: previous frame identical to this frame (corrupt stack?)</div></div></div></div></blockquote><div class=""><div><br class=""></div><div>Seems stack is corrupted somewhere. I’m still looking. But the GDB stub does seem helpful for this.</div><div><br class=""></div><div>Mark</div><div><br class=""><blockquote type="cite" class=""><div class="">On 25 Apr 2021, at 12:47 AM, Stephen Casner <<a href="mailto:casner@acm.org" class="">casner@acm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Michael,<br class=""><br class="">I confirm the same lack of crashes, and I agree that the problem is<br class="">not strictly with WolfSSL.<br class=""><br class="">It seems that a task WDT ought to be able to report where the task was<br class="">executing at the time. That would certainly be helpful in narrowing<br class="">down the type of problem. I'm not sure if the trap to gdb mode would<br class="">provide that info since I observed no response on the USB console when<br class="">the module went comatose<br class=""><br class=""> -- Steve<br class=""><br class="">On Sat, 24 Apr 2021, Michael Balzer wrote:<br class=""><br class=""><blockquote type="cite" class="">Steve,<br class=""><br class="">in the two weeks since disabling TLS on the server V2 connection I haven't had<br class="">a single crash. While that's not a proof yet the watchdog issue is TLS<br class="">related, it's at least a strong indicator.<br class=""><br class="">The watchdog triggers if the idle task on a core doesn't get a CPU share for<br class="">120 seconds. If the TLS functions block a CPU for more than a few seconds,<br class="">that's already pretty bad, as that means TLS will cause delays in CAN<br class="">processing (disrupting protocol transfers) and can possibly cause frame drops<br class="">and queue overflows. Blocking the whole system for more than 120 seconds is<br class="">totally unacceptable.<br class=""><br class="">This doesn't feel like a calculation / math performance issue, it rather feels<br class="">like a bug - and that may imply a security issue as well.<br class=""><br class="">But I don't think this is caused by WolfSSL, as the issue has been present<br class="">with mbedTLS as well, just didn't occur that frequently. Maybe some race<br class="">condition with the LwIP task?<br class=""><br class="">Regards,<br class="">Michael<br class=""><br class=""><br class="">Am 11.04.21 um 09:44 schrieb Michael Balzer:<br class=""><blockquote type="cite" class="">Steve,<br class=""><br class="">I can confirm an increase of these events since we changed to WolfSSL, about<br class="">once every three days currently for me. The frequency was much lower before,<br class="">more like once or twice per month.<br class=""><br class="">I've disabled TLS on my module now and will report if that helps.<br class=""><br class="">Regards,<br class="">Michael<br class=""><br class=""><br class="">Am 10.04.21 um 21:20 schrieb Stephen Casner:<br class=""><blockquote type="cite" class="">Michael,<br class=""><br class="">As you saw from my earlier emails, I was getting these crashes<br class="">typically after less than 24 hours of operation. I changed my config<br class="">to disable TLS on server v2 and rebooted 2021-04-05 23:36:04.648 PDT<br class="">and there has not been a crash since. So it definitely appears to be<br class="">correlated with the additional processing to support TLS.<br class=""><br class=""> -- Steve<br class=""><br class="">On Sun, 4 Apr 2021, Michael Balzer wrote:<br class=""><br class=""><blockquote type="cite" class="">Steve,<br class=""><br class="">that's the problem with this issue, it's totally unclear what causes<br class="">this.<br class=""><br class="">The signal dropping begins when the queue is full, which happens after<br class="">the<br class="">task has been blocked for ~as many seconds as the queue is big. So there<br class="">is no<br class="">logged activity that could cause this, your module basically went into<br class="">this<br class="">from idling.<br class=""><br class="">Regards,<br class="">Michael<br class=""></blockquote><br class=""></blockquote><br class=""><br class="">_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></blockquote><br class="">--<br class="">Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal<br class="">Fon 02333 / 833 5735 * Handy 0176 / 206 989 26<br class=""></blockquote>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></div></div></blockquote></div><br class=""></div></body></html>