[Ovmsdev] frequent crashes

[Craig Leres]

I've been running a v3.1 module in my Cadillac for a few weeks now and 
recently realized that it crashes frequently. I see this on the status page:

     Last boot was 12879 second(s) ago
     Time at boot: 2019-06-26 11:31:51 PDT
       This is reset #10 since last power cycle
       Detected boot reason: Crash (8/14)
       Crash counters: 8 total, 0 early

     Last crash: abort() was called on core 0
       Backtrace:
       0x40096068 0x40096263 0x40115aa7 0x40232e95 0x4023312d 0x4023315d 
0x40229d52 0x4022a780 0x40226576 0x402276d6 0x40227ad1
       Version: 3.2.002-89-g0207c45e-dirty/ota_0/main (build idf 
v3.1-dev-2835-g151269458 Jun 22 2019 14:12:19)
     Firmware:          3.2.002-89-g0207c45e-dirty/ota_0/main (build idf 
v3.1-dev-2835-g151269458 Jun 22 2019 14:12:19)
     Running partition: ota_0
     Boot partition:    ota_0
     Factory image:     3.2.002
     OTA_O image:       3.2.002-89-g0207c45e-dirty
     OTA_1 image:       3.2.002-89-g0207c45e-dirty

Which translates to:

     + xtensa-esp32-elf-addr2line -e build/ovms3.elf 0x40096068 
0x40096263 0x40115aa7 0x40232e95 0x4023312d 0x4023315d 0x40229d52 
0x4022a780 0x40226576 0x402276d6 0x40227ad1
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/esp32/panic.c:670
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/esp32/panic.c:670
 
/home/jeroen/esp8266/esp32/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:63 
(discriminator 8)
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/core/ipv4/ip4.c:846 
(discriminator 1)
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/core/ipv4/ip4.c:810
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/core/ipv4/ip4.c:787
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/core/tcp_out.c:1272 
(discriminator 4)
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/core/tcp_out.c:1073
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/api/api_msg.c:1875 
(discriminator 6)
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/api/api_msg.c:1875 
(discriminator 6)
 
/home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/api/tcpip.c:474

I was beginning to think that my battery tender maintenance charger was 
the cause but I disconnected it a few days ago.

Is this psram cache issue related or something else? I haven't written 
much code yet and it doesn't do a lot, especially when the car is off.

Is there an easy way to poll the module for crash related info?

I never understood what the boot reason numbers were:

       Detected boot reason: Crash (8/14)

but I see now that there is a number for each core and are defined in 
the esp-idf/components/esp32/include/rom/rtc.h:

typedef enum {
     NO_MEAN                =  0,
     POWERON_RESET          =  1,    /**<1, Vbat power on reset*/
     SW_RESET               =  3,    /**<3, Software reset digital core*/
     OWDT_RESET             =  4,    /**<4, Legacy watch dog reset 
digital core*/
     DEEPSLEEP_RESET        =  5,    /**<3, Deep Sleep reset digital core*/
     SDIO_RESET             =  6,    /**<6, Reset by SLC module, reset 
digital core*/
     TG0WDT_SYS_RESET       =  7,    /**<7, Timer Group0 Watch dog reset 
digital core*/
     TG1WDT_SYS_RESET       =  8,    /**<8, Timer Group1 Watch dog reset 
digital core*/
     RTCWDT_SYS_RESET       =  9,    /**<9, RTC Watch dog Reset digital 
core*/
     INTRUSION_RESET        = 10,    /**<10, Instrusion tested to reset 
CPU*/
     TGWDT_CPU_RESET        = 11,    /**<11, Time Group reset CPU*/
     SW_CPU_RESET           = 12,    /**<12, Software reset CPU*/
     RTCWDT_CPU_RESET       = 13,    /**<13, RTC Watch dog Reset CPU*/
     EXT_CPU_RESET          = 14,    /**<14, for APP CPU, reseted by PRO 
CPU*/
     RTCWDT_BROWN_OUT_RESET = 15,    /**<15, Reset when the vdd voltage 
is not stable*/
     RTCWDT_RTC_RESET       = 16     /**<16, RTC Watch dog reset digital 
core and rtc module*/
} RESET_REASON;

so my most recent reset was due to 8: "Timer Group1 Watch dog reset 
digital core" (and the 14 is probably cpu0 resetting cpu1 when the watch 
dog hit?) I am using Ticker10 based on Ticker1 code I saw in 
vehicle_nissanleaf.cpp. But I get crashes even if I ifdef out that code out.

I've also attached a screen grab from my hologram.io account, I think 
it's showing just how frequently the crashes are.

I could also eliminate my FreeBSD xtensa toolchain by doing a build 
under ubuntu, would that be worthwhile?

		Craig
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hologram.png
Type: image/png
Size: 37611 bytes
Desc: not available
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20190626/43c822bb/attachment-0002.png>