<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
gdb is better, but also not exact. Line numbers reported are usually
around 2-5 lines too low.<br>
<br>
I'm pretty sure now the crashes in mg_send_dns_query() are also
memory related, it looks like MG_CALLOC() fails.<br>
<br>
I think this happens because we still buffer notifications in RAM.
The Twizy firmware sends a lot of data notifications which may add
up on a temporary loss of connection.<br>
<br>
I'm looking forward to testing with the v3.1 module. The first has
arrived in germany, mine has been routed to the customs department…
let's see what happens next…<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 09.04.2018 um 00:27 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:679d8d05-0bee-a6e9-8788-e52069120b62@expeedo.de">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Att: addr2line reports wrong line numbers.<br>
<br>
That's a known issue: <a class="moz-txt-link-freetext"
href="https://github.com/jcmvbkbc/binutils-gdb-xtensa/issues/5"
moz-do-not-send="true">https://github.com/jcmvbkbc/binutils-gdb-xtensa/issues/5</a><br>
<br>
Workaround is using gdb. I've made a little bash script named
"a2l" for this:<br>
<br>
<tt>#!/bin/bash</tt><tt><br>
</tt><tt>elf=~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/build/ovms3.elf</tt><tt><br>
</tt><tt>for adr in $* ; do cmd+=" -ex 'l *$adr'" ; done</tt><tt><br>
</tt><tt>cmd+=" -ex 'q'"</tt><tt><br>
</tt><tt>eval xtensa-esp32-elf-gdb -batch $elf $cmd 2>/dev/null
| grep " is in "</tt><tt><br>
</tt><br>
You can also leave out the final grep if you want to see the
source context.<br>
<br>
<br>
80% of the crashes I have recorded today have been out of memory
issues.<br>
<br>
The remaining 20% have all been along this path:<br>
<br>
<tt>0x400f8432 is in mg_send_dns_query
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11168).</tt><tt><br>
</tt><tt>0x400f85d9 is in mg_resolve_async_eh
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11628).</tt><tt><br>
</tt><tt>0x400f644b is in mg_call
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:2277).</tt><tt><br>
</tt><tt>0x400f6536 is in mg_if_poll
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:2318).</tt><tt><br>
</tt><tt>0x400f7a56 is in mg_mgr_handle_conn
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:3729).</tt><tt><br>
</tt><tt>0x400f7ca9 is in mg_socket_if_poll
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:3916).</tt><tt><br>
</tt><tt>0x400f4471 is in mg_mgr_poll
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:2446).</tt><tt><br>
</tt><tt>0x400eb9e6 is in OvmsNetManager::MongooseTask()
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_netmanager.cpp:382).</tt><tt><br>
</tt><tt>0x400eba25 is in MongooseRawTask(void*)
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_netmanager.cpp:370).</tt><tt><br>
</tt><br>
…which is what Greg reported as "webserver status crash" but also
occurs without user interaction.<br>
<br>
This is related to network changes, for example losing contact to
a wifi station quite often triggers this. It looks like mongoose
continues to use some object that has been deleted by another
task, but I need to have a closer look tomorrow.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 08.04.2018 um 16:11 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite"
cite="mid:E0CF9F48-8B8F-4C16-A8A5-003BE805BF6F@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
Michael,
<div class=""><br class="">
</div>
<div class="">Neat. I agree that the stack track is normally
sufficient.</div>
<div class=""><br class="">
</div>
<div class="">I’ve started to store the production PUSH
firmwares in a standardised way. You can find them at:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class=""><a
href="http://api.openvehicles.com/firmware/ota/" class=""
moz-do-not-send="true">http://api.openvehicles.com/firmware/ota/</a><a
href="http://api.openvehicles.com/firmware/ota/v3.0/main/"
class="" moz-do-not-send="true">(v3.0|v3.1)/main/</a></div>
</blockquote>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class=""><ver>.ovms3.bin</div>
<div class=""><ver>.ovms3.elf</div>
<div class=""><ver>.ovms3.map</div>
<div class=""><br class="">
</div>
</blockquote>
Where <ver> is the version in format
‘major.minor.sequence' such as 3.1.003.</blockquote>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class=""><br class="">
</blockquote>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">So, for example:</blockquote>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class=""><br class="">
</blockquote>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class=""><a
href="http://api.openvehicles.com/firmware/ota/v3.1/main/3.1.003.ovms3.elf"
class="" moz-do-not-send="true">http://api.openvehicles.com/firmware/ota/v3.1/main/3.1.003.ovms3.elf</a></blockquote>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Regards, Mark</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 8 Apr 2018, at 8:56 PM, Michael Balzer
<<a href="mailto:dexter@expeedo.de" class=""
moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">I've pushed an update to enable saving
crash data over a reset and automatically send that to
the server using the history record
"*-OVM-DebugCrash".<br class="">
<br class="">
Note: you will need to pull the esp-idf as well. As
the support for exception handlers was insufficient
for our needs, I have added a separate error handler<br
class="">
registration that catches all three kinds of crashes
(exceptions, panics and aborts).<br class="">
<br class="">
Crash reason, registers (if available) and backtrace
are available via "boot status", with "make monitor"
automatically mapping the addresses:<br class="">
<br class="">
OVMS# boot status<br class="">
Last boot was 16 second(s) ago<br class="">
This is reset #1 since last power cycle<br class="">
Detected boot reason: Crash<br class="">
Crash counters: 1 total, 0 early<br class="">
CPU#0 boot reason was 12<br class="">
CPU#1 boot reason was 12<br class="">
<br class="">
Last crash: abort() was called on core 1<br class="">
Backtrace:<br class="">
0x4008f698 0x4008f86f 0x400e7027 0x400edb76
0x400edc8d 0x400edc7f 0x400edcb5 0x400e3908 0x400f11c9
0x400f1230 0x400e3937 0x401cb613 0x400e3f49 0x400e82e5<br
class="">
0x400e84d1 0x400e3df5 0x400e3e04 0x400e69dd<br
class="">
0x4008f698: invoke_abort at
/home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
class="">
0x4008f86f: abort at
/home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
class="">
0x400e7027: module_fault(int, OvmsWriter*,
OvmsCommand*, int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_module.cpp:823<br
class="">
0x400edb76: OvmsCommand::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400edc8d: OvmsCommand::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400edc7f: OvmsCommand::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400edcb5: OvmsCommandApp::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400e3908: Execute(microrl*, int, char const* const*)
at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:47<br
class="">
0x400f11c9: new_line_handler at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:620<br
class="">
0x400f1230: microrl_insert_char at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:668<br
class="">
0x400e3937: OvmsShell::ProcessChar(char) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:70<br
class="">
0x401cb613: OvmsShell::ProcessChars(char const*, int)
at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:77<br
class="">
(discriminator 2)<br class="">
0x400e3f49: ConsoleAsync::HandleDeviceEvent(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:169<br
class="">
0x400e82e5: OvmsConsole::Poll(unsigned int, void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:152<br
class="">
0x400e84d1: OvmsConsole::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:132
(discriminator 1)<br class="">
0x400e3df5: ConsoleAsync::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:80<br
class="">
0x400e3e04: non-virtual thunk to
ConsoleAsync::Service() at ??:?<br class="">
0x400e69dd: TaskBase::Task(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./task_base.cpp:156<br
class="">
<br class="">
<br class="">
The "*-OVM-DebugCrash" record has this structure:<br
class="">
<br class="">
// H type "*-OVM-DebugCrash"<br class="">
// ,<firmware_version><br class="">
//
,<bootcount>,<bootreason_name>,<bootreason_cpu0>,<bootreason_cpu1><br
class="">
//
,<crashcnt>,<earlycrashcnt>,<crashtype>,<crashcore>,<registers>,<backtrace><br
class="">
<br class="">
…with registers and backtraces separated by spaces.<br
class="">
<br class="">
Example:<br class="">
<br class="">
*-OVM-DebugCrash,0,2592000,3.1.003-11-g37c5f4b/factory/main (build idf
v3.1-dev-453-g0f978bcb-dirty Apr 8 2018<br class="">
14:42:07),1,Crash,12,12,1,0,abort(),1,,0x4008f698
0x4008f86f 0x400e7027 0x400edb76 0x400edc8d 0x400edc7f
0x400edcb5 0x400e3908 0x400f11c9 0x400f1230 0x400e3937<br
class="">
0x401cb613 0x400e3f49 0x400e82e5 0x400e84d1 0x400e3df5
0x400e3e04 0x400e69dd<br class="">
<br class="">
To decode the backtrace, feed it to addr2line:<br
class="">
<br class="">
<a class="moz-txt-link-abbreviated"
href="mailto:balzer@leela:%7E/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3"
moz-do-not-send="true">balzer@leela:~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3</a>>
xtensa-esp32-elf-addr2line -pfiaC -e build/ovms3.elf
0x4008f698 0x4008f86f 0x400e7027<br class="">
0x400edb76 0x400edc8d 0x400edc7f 0x400edcb5 0x400e3908
0x400f11c9 0x400f1230 0x400e3937 0x401cb613 0x400e3f49
0x400e82e5 0x400e84d1 0x400e3df5 0x400e3e04<br
class="">
0x400e69dd<br class="">
0x4008f698: invoke_abort at
/home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
class="">
0x4008f86f: abort at
/home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
class="">
0x400e7027: module_fault(int, OvmsWriter*,
OvmsCommand*, int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_module.cpp:823<br
class="">
0x400edb76: OvmsCommand::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400edc8d: OvmsCommand::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400edc7f: OvmsCommand::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400edcb5: OvmsCommandApp::Execute(int, OvmsWriter*,
int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
class="">
0x400e3908: Execute(microrl*, int, char const* const*)
at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:47<br
class="">
0x400f11c9: new_line_handler at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:620<br
class="">
0x400f1230: microrl_insert_char at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:668<br
class="">
0x400e3937: OvmsShell::ProcessChar(char) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:70<br
class="">
0x401cb613: OvmsShell::ProcessChars(char const*, int)
at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:77<br
class="">
(discriminator 2)<br class="">
0x400e3f49: ConsoleAsync::HandleDeviceEvent(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:169<br
class="">
0x400e82e5: OvmsConsole::Poll(unsigned int, void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:152<br
class="">
0x400e84d1: OvmsConsole::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:132
(discriminator 1)<br class="">
0x400e3df5: ConsoleAsync::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:80<br
class="">
0x400e3e04: non-virtual thunk to
ConsoleAsync::Service() at ??:?<br class="">
0x400e69dd: TaskBase::Task(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./task_base.cpp:156<br
class="">
<br class="">
<br class="">
There is also an option to write core dumps in
esp-idf, but a core has 64K, and crashes are still too
many. I think the backtrace is sufficient in most
situations.<br class="">
<br class="">
We should keep a central archive of .elf files for the
releases rolled out, so we don't need to recompile for
debugging.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
-- <br class="">
Michael Balzer * Helkenberger Weg 9 * D-58256
Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26<br
class="">
<br class="">
<br class="">
_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</body>
</html>