<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Att: addr2line reports wrong line numbers.<br>
    <br>
    That's a known issue:
    <a class="moz-txt-link-freetext" href="https://github.com/jcmvbkbc/binutils-gdb-xtensa/issues/5">https://github.com/jcmvbkbc/binutils-gdb-xtensa/issues/5</a><br>
    <br>
    Workaround is using gdb. I've made a little bash script named "a2l"
    for this:<br>
    <br>
    <tt>#!/bin/bash</tt><tt><br>
    </tt><tt>elf=~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/build/ovms3.elf</tt><tt><br>
    </tt><tt>for adr in $* ; do cmd+=" -ex 'l *$adr'" ; done</tt><tt><br>
    </tt><tt>cmd+=" -ex 'q'"</tt><tt><br>
    </tt><tt>eval xtensa-esp32-elf-gdb -batch $elf $cmd 2>/dev/null |
      grep " is in "</tt><tt><br>
    </tt><br>
    You can also leave out the final grep if you want to see the source
    context.<br>
    <br>
    <br>
    80% of the crashes I have recorded today have been out of memory
    issues.<br>
    <br>
    The remaining 20% have all been along this path:<br>
    <br>
    <tt>0x400f8432 is in mg_send_dns_query
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11168).</tt><tt><br>
    </tt><tt>0x400f85d9 is in mg_resolve_async_eh
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11628).</tt><tt><br>
    </tt><tt>0x400f644b is in mg_call
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:2277).</tt><tt><br>
    </tt><tt>0x400f6536 is in mg_if_poll
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:2318).</tt><tt><br>
    </tt><tt>0x400f7a56 is in mg_mgr_handle_conn
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:3729).</tt><tt><br>
    </tt><tt>0x400f7ca9 is in mg_socket_if_poll
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:3916).</tt><tt><br>
    </tt><tt>0x400f4471 is in mg_mgr_poll
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:2446).</tt><tt><br>
    </tt><tt>0x400eb9e6 is in OvmsNetManager::MongooseTask()
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_netmanager.cpp:382).</tt><tt><br>
    </tt><tt>0x400eba25 is in MongooseRawTask(void*)
(/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_netmanager.cpp:370).</tt><tt><br>
    </tt><br>
    …which is what Greg reported as "webserver status crash" but also
    occurs without user interaction.<br>
    <br>
    This is related to network changes, for example losing contact to a
    wifi station quite often triggers this. It looks like mongoose
    continues to use some object that has been deleted by another task,
    but I need to have a closer look tomorrow.<br>
    <br>
    Regards,<br>
    Michael<br>
    <br>
    <br>
    <div class="moz-cite-prefix">Am 08.04.2018 um 16:11 schrieb Mark
      Webb-Johnson:<br>
    </div>
    <blockquote type="cite"
      cite="mid:E0CF9F48-8B8F-4C16-A8A5-003BE805BF6F@webb-johnson.net">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      Michael,
      <div class=""><br class="">
      </div>
      <div class="">Neat. I agree that the stack track is normally
        sufficient.</div>
      <div class=""><br class="">
      </div>
      <div class="">I’ve started to store the production PUSH firmwares
        in a standardised way. You can find them at:</div>
      <div class=""><br class="">
      </div>
      <blockquote style="margin: 0 0 0 40px; border: none; padding:
        0px;" class="">
        <div class=""><a
            href="http://api.openvehicles.com/firmware/ota/" class=""
            moz-do-not-send="true">http://api.openvehicles.com/firmware/ota/</a><a
            href="http://api.openvehicles.com/firmware/ota/v3.0/main/"
            class="" moz-do-not-send="true">(v3.0|v3.1)/main/</a></div>
      </blockquote>
      <blockquote style="margin: 0 0 0 40px; border: none; padding:
        0px;" class="">
        <blockquote style="margin: 0 0 0 40px; border: none; padding:
          0px;" class="">
          <div class=""><ver>.ovms3.bin</div>
          <div class=""><ver>.ovms3.elf</div>
          <div class=""><ver>.ovms3.map</div>
          <div class=""><br class="">
          </div>
        </blockquote>
        Where <ver> is the version in format
        ‘major.minor.sequence' such as 3.1.003.</blockquote>
      <blockquote style="margin: 0 0 0 40px; border: none; padding:
        0px;" class=""><br class="">
      </blockquote>
      <blockquote style="margin: 0 0 0 40px; border: none; padding:
        0px;" class="">So, for example:</blockquote>
      <blockquote style="margin: 0 0 0 40px; border: none; padding:
        0px;" class=""><br class="">
      </blockquote>
      <blockquote style="margin: 0 0 0 40px; border: none; padding:
        0px;" class="">
        <blockquote style="margin: 0 0 0 40px; border: none; padding:
          0px;" class=""><a
href="http://api.openvehicles.com/firmware/ota/v3.1/main/3.1.003.ovms3.elf"
            class="" moz-do-not-send="true">http://api.openvehicles.com/firmware/ota/v3.1/main/3.1.003.ovms3.elf</a></blockquote>
      </blockquote>
      <div class=""><br class="">
      </div>
      <div class="">Regards, Mark</div>
      <div class=""><br class="">
        <div>
          <blockquote type="cite" class="">
            <div class="">On 8 Apr 2018, at 8:56 PM, Michael Balzer <<a
                href="mailto:dexter@expeedo.de" class=""
                moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
            <br class="Apple-interchange-newline">
            <div class="">
              <div class="">I've pushed an update to enable saving crash
                data over a reset and automatically send that to the
                server using the history record "*-OVM-DebugCrash".<br
                  class="">
                <br class="">
                Note: you will need to pull the esp-idf as well. As the
                support for exception handlers was insufficient for our
                needs, I have added a separate error handler<br class="">
                registration that catches all three kinds of crashes
                (exceptions, panics and aborts).<br class="">
                <br class="">
                Crash reason, registers (if available) and backtrace are
                available via "boot status", with "make monitor"
                automatically mapping the addresses:<br class="">
                <br class="">
                OVMS# boot status<br class="">
                Last boot was 16 second(s) ago<br class="">
                  This is reset #1 since last power cycle<br class="">
                  Detected boot reason: Crash<br class="">
                  Crash counters: 1 total, 0 early<br class="">
                  CPU#0 boot reason was 12<br class="">
                  CPU#1 boot reason was 12<br class="">
                <br class="">
                Last crash: abort() was called on core 1<br class="">
                  Backtrace:<br class="">
                  0x4008f698 0x4008f86f 0x400e7027 0x400edb76 0x400edc8d
                0x400edc7f 0x400edcb5 0x400e3908 0x400f11c9 0x400f1230
                0x400e3937 0x401cb613 0x400e3f49 0x400e82e5<br class="">
                0x400e84d1 0x400e3df5 0x400e3e04 0x400e69dd<br class="">
                0x4008f698: invoke_abort at
                /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
                  class="">
                0x4008f86f: abort at
                /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
                  class="">
                0x400e7027: module_fault(int, OvmsWriter*, OvmsCommand*,
                int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_module.cpp:823<br
                  class="">
                0x400edb76: OvmsCommand::Execute(int, OvmsWriter*, int,
                char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400edc8d: OvmsCommand::Execute(int, OvmsWriter*, int,
                char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400edc7f: OvmsCommand::Execute(int, OvmsWriter*, int,
                char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400edcb5: OvmsCommandApp::Execute(int, OvmsWriter*,
                int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400e3908: Execute(microrl*, int, char const* const*)
                at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:47<br
                  class="">
                0x400f11c9: new_line_handler at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:620<br
                  class="">
                0x400f1230: microrl_insert_char at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:668<br
                  class="">
                0x400e3937: OvmsShell::ProcessChar(char) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:70<br
                  class="">
                0x401cb613: OvmsShell::ProcessChars(char const*, int) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:77<br
                  class="">
                (discriminator 2)<br class="">
                0x400e3f49: ConsoleAsync::HandleDeviceEvent(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:169<br
                  class="">
                0x400e82e5: OvmsConsole::Poll(unsigned int, void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:152<br
                  class="">
                0x400e84d1: OvmsConsole::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:132
                (discriminator 1)<br class="">
                0x400e3df5: ConsoleAsync::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:80<br
                  class="">
                0x400e3e04: non-virtual thunk to ConsoleAsync::Service()
                at ??:?<br class="">
                0x400e69dd: TaskBase::Task(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./task_base.cpp:156<br
                  class="">
                <br class="">
                <br class="">
                The "*-OVM-DebugCrash" record has this structure:<br
                  class="">
                <br class="">
                    // H type "*-OVM-DebugCrash"<br class="">
                    //  ,<firmware_version><br class="">
                    // 
,<bootcount>,<bootreason_name>,<bootreason_cpu0>,<bootreason_cpu1><br
                  class="">
                    // 
,<crashcnt>,<earlycrashcnt>,<crashtype>,<crashcore>,<registers>,<backtrace><br
                  class="">
                <br class="">
                …with registers and backtraces separated by spaces.<br
                  class="">
                <br class="">
                Example:<br class="">
                <br class="">
*-OVM-DebugCrash,0,2592000,3.1.003-11-g37c5f4b/factory/main (build idf
                v3.1-dev-453-g0f978bcb-dirty Apr  8 2018<br class="">
                14:42:07),1,Crash,12,12,1,0,abort(),1,,0x4008f698
                0x4008f86f 0x400e7027 0x400edb76 0x400edc8d 0x400edc7f
                0x400edcb5 0x400e3908 0x400f11c9 0x400f1230 0x400e3937<br
                  class="">
                0x401cb613 0x400e3f49 0x400e82e5 0x400e84d1 0x400e3df5
                0x400e3e04 0x400e69dd<br class="">
                <br class="">
                To decode the backtrace, feed it to addr2line:<br
                  class="">
                <br class="">
<a class="moz-txt-link-abbreviated" href="mailto:balzer@leela:~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3">balzer@leela:~/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3</a>>
                xtensa-esp32-elf-addr2line -pfiaC -e build/ovms3.elf
                0x4008f698 0x4008f86f 0x400e7027<br class="">
                0x400edb76 0x400edc8d 0x400edc7f 0x400edcb5 0x400e3908
                0x400f11c9 0x400f1230 0x400e3937 0x401cb613 0x400e3f49
                0x400e82e5 0x400e84d1 0x400e3df5 0x400e3e04<br class="">
                0x400e69dd<br class="">
                0x4008f698: invoke_abort at
                /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
                  class="">
                0x4008f86f: abort at
                /home/balzer/esp/esp-idf/components/esp32/./panic.c:669<br
                  class="">
                0x400e7027: module_fault(int, OvmsWriter*, OvmsCommand*,
                int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_module.cpp:823<br
                  class="">
                0x400edb76: OvmsCommand::Execute(int, OvmsWriter*, int,
                char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400edc8d: OvmsCommand::Execute(int, OvmsWriter*, int,
                char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400edc7f: OvmsCommand::Execute(int, OvmsWriter*, int,
                char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400edcb5: OvmsCommandApp::Execute(int, OvmsWriter*,
                int, char const* const*) at<br class="">
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_command.cpp:94<br
                  class="">
                0x400e3908: Execute(microrl*, int, char const* const*)
                at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:47<br
                  class="">
                0x400f11c9: new_line_handler at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:620<br
                  class="">
                0x400f1230: microrl_insert_char at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/components/microrl/./microrl.c:668<br
                  class="">
                0x400e3937: OvmsShell::ProcessChar(char) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:70<br
                  class="">
                0x401cb613: OvmsShell::ProcessChars(char const*, int) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_shell.cpp:77<br
                  class="">
                (discriminator 2)<br class="">
                0x400e3f49: ConsoleAsync::HandleDeviceEvent(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:169<br
                  class="">
                0x400e82e5: OvmsConsole::Poll(unsigned int, void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:152<br
                  class="">
                0x400e84d1: OvmsConsole::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./ovms_console.cpp:132
                (discriminator 1)<br class="">
                0x400e3df5: ConsoleAsync::Service() at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./console_async.cpp:80<br
                  class="">
                0x400e3e04: non-virtual thunk to ConsoleAsync::Service()
                at ??:?<br class="">
                0x400e69dd: TaskBase::Task(void*) at
/home/balzer/esp/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/./task_base.cpp:156<br
                  class="">
                <br class="">
                <br class="">
                There is also an option to write core dumps in esp-idf,
                but a core has 64K, and crashes are still too many. I
                think the backtrace is sufficient in most situations.<br
                  class="">
                <br class="">
                We should keep a central archive of .elf files for the
                releases rolled out, so we don't need to recompile for
                debugging.<br class="">
                <br class="">
                Regards,<br class="">
                Michael<br class="">
                <br class="">
                -- <br class="">
                Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal<br
                  class="">
                Fon 02333 / 833 5735 * Handy 0176 / 206 989 26<br
                  class="">
                <br class="">
                <br class="">
                _______________________________________________<br
                  class="">
                OvmsDev mailing list<br class="">
                <a href="mailto:OvmsDev@lists.openvehicles.com" class=""
                  moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                  class="">
                <a
                  href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                  class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                  class="">
              </div>
            </div>
          </blockquote>
        </div>
        <br class="">
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
  </body>
</html>