<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Michael,<div class=""><br class=""></div><div class="">Neat. I agree that the stack track is normally sufficient.</div><div class=""><br class=""></div><div class="">I’ve started to store the production PUSH firmwares in a standardised way. You can find them at:</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><a href="http://api.openvehicles.com/firmware/ota/" class="">http://api.openvehicles.com/firmware/ota/</a><a href="http://api.openvehicles.com/firmware/ota/v3.0/main/" class="">(v3.0|v3.1)/main/</a></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><ver>.ovms3.bin</div><div class=""><ver>.ovms3.elf</div><div class=""><ver>.ovms3.map</div><div class=""><br class=""></div></blockquote>Where <ver> is the version in format ‘major.minor.sequence' such as 3.1.003.</blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><br class=""></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class="">So, for example:</blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><br class=""></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><a href="http://api.openvehicles.com/firmware/ota/v3.1/main/3.1.003.ovms3.elf" class="">http://api.openvehicles.com/firmware/ota/v3.1/main/3.1.003.ovms3.elf</a></blockquote></blockquote><div class=""><br class=""></div><div class="">Regards, Mark</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 8 Apr 2018, at 8:56 PM, Michael Balzer <<a href="mailto:dexter@expeedo.de" class="">dexter@expeedo.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">I've pushed an update to enable saving crash data over a reset and automatically send that to the server using the history record "*-OVM-DebugCrash".<br class=""><br class="">Note: you will need to pull the esp-idf as well. As the support for exception handlers was insufficient for our needs, I have added a separate error handler<br class="">registration that catches all three kinds of crashes (exceptions, panics and aborts).<br class=""><br class="">Crash reason, registers (if available) and backtrace are available via "boot status", with "make monitor" automatically mapping the addresses:<br class=""><br class="">OVMS# boot status<br class="">Last boot was 16 second(s) ago<br class=""> This is reset #1 since last power cycle<br class=""> Detected boot reason: Crash<br class=""> Crash counters: 1 total, 0 early<br class=""> CPU#0 boot reason was 12<br class=""> CPU#1 boot reason was 12<br class=""><br class="">Last crash: abort() was called on core 1<br class=""> Backtrace:<br class=""> 0x4008f698 0x4008f86f 0x400e7027 0x400edb76 0x400edc8d 0x400edc7f 0x400edcb5 0x400e3908 0x400f11c9 0x400f1230 0x400e3937 0x401cb613 0x400e3f49 0x400e82e5<br class="">0x400e84d1 0x400e3df5 0x400e3e04 0x400e69dd<br class="">0x4008f698: invoke_abort at /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br class="">0x4008f86f: abort at /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br class="">0x400e7027: module_fault(int, OvmsWriter*, OvmsCommand*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_module.cpp:823<br class="">0x400edb76: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400edc8d: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400edc7f: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400edcb5: OvmsCommandApp::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400e3908: Execute(microrl*, int, char const* const*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:47<br class="">0x400f11c9: new_line_handler at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:620<br class="">0x400f1230: microrl_insert_char at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:668<br class="">0x400e3937: OvmsShell::ProcessChar(char) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:70<br class="">0x401cb613: OvmsShell::ProcessChars(char const*, int) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:77<br class="">(discriminator 2)<br class="">0x400e3f49: ConsoleAsync::HandleDeviceEvent(void*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:169<br class="">0x400e82e5: OvmsConsole::Poll(unsigned int, void*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:152<br class="">0x400e84d1: OvmsConsole::Service() at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:132 (discriminator 1)<br class="">0x400e3df5: ConsoleAsync::Service() at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:80<br class="">0x400e3e04: non-virtual thunk to ConsoleAsync::Service() at ??:?<br class="">0x400e69dd: TaskBase::Task(void*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./task_base.cpp:156<br class=""><br class=""><br class="">The "*-OVM-DebugCrash" record has this structure:<br class=""><br class=""> // H type "*-OVM-DebugCrash"<br class=""> // ,<firmware_version><br class=""> // ,<bootcount>,<bootreason_name>,<bootreason_cpu0>,<bootreason_cpu1><br class=""> // ,<crashcnt>,<earlycrashcnt>,<crashtype>,<crashcore>,<registers>,<backtrace><br class=""><br class="">…with registers and backtraces separated by spaces.<br class=""><br class="">Example:<br class=""><br class="">*-OVM-DebugCrash,0,2592000,3.1.003-11-g37c5f4b/factory/main (build idf v3.1-dev-453-g0f978bcb-dirty Apr 8 2018<br class="">14:42:07),1,Crash,12,12,1,0,abort(),1,,0x4008f698 0x4008f86f 0x400e7027 0x400edb76 0x400edc8d 0x400edc7f 0x400edcb5 0x400e3908 0x400f11c9 0x400f1230 0x400e3937<br class="">0x401cb613 0x400e3f49 0x400e82e5 0x400e84d1 0x400e3df5 0x400e3e04 0x400e69dd<br class=""><br class="">To decode the backtrace, feed it to addr2line:<br class=""><br class="">balzer@leela:~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3> xtensa-esp32-elf-addr2line -pfiaC -e build/ovms3.elf 0x4008f698 0x4008f86f 0x400e7027<br class="">0x400edb76 0x400edc8d 0x400edc7f 0x400edcb5 0x400e3908 0x400f11c9 0x400f1230 0x400e3937 0x401cb613 0x400e3f49 0x400e82e5 0x400e84d1 0x400e3df5 0x400e3e04<br class="">0x400e69dd<br class="">0x4008f698: invoke_abort at /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br class="">0x4008f86f: abort at /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br class="">0x400e7027: module_fault(int, OvmsWriter*, OvmsCommand*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_module.cpp:823<br class="">0x400edb76: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400edc8d: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400edc7f: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400edcb5: OvmsCommandApp::Execute(int, OvmsWriter*, int, char const* const*) at<br class="">/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br class="">0x400e3908: Execute(microrl*, int, char const* const*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:47<br class="">0x400f11c9: new_line_handler at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:620<br class="">0x400f1230: microrl_insert_char at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:668<br class="">0x400e3937: OvmsShell::ProcessChar(char) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:70<br class="">0x401cb613: OvmsShell::ProcessChars(char const*, int) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:77<br class="">(discriminator 2)<br class="">0x400e3f49: ConsoleAsync::HandleDeviceEvent(void*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:169<br class="">0x400e82e5: OvmsConsole::Poll(unsigned int, void*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:152<br class="">0x400e84d1: OvmsConsole::Service() at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:132 (discriminator 1)<br class="">0x400e3df5: ConsoleAsync::Service() at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:80<br class="">0x400e3e04: non-virtual thunk to ConsoleAsync::Service() at ??:?<br class="">0x400e69dd: TaskBase::Task(void*) at /home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./task_base.cpp:156<br class=""><br class=""><br class="">There is also an option to write core dumps in esp-idf, but a core has 64K, and crashes are still too many. I think the backtrace is sufficient in most situations.<br class=""><br class="">We should keep a central archive of .elf files for the releases rolled out, so we don't need to recompile for debugging.<br class=""><br class="">Regards,<br class="">Michael<br class=""><br class="">-- <br class="">Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal<br class="">Fon 02333 / 833 5735 * Handy 0176 / 206 989 26<br class=""><br class=""><br class="">_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class=""><a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class=""></div></div></blockquote></div><br class=""></div></body></html>