[Ovmsdev] Investigating crashes. And, OVMS stopped collecting data
Michael Balzer
dexter at expeedo.de
Fri Jan 1 21:41:05 HKT 2021
Steve,
writing the core dump to flash isn't possible. The only option to attach
gdb is to…
1. build with CONFIG_ESP32_PANIC_GDBSTUB enabled, then
2. produce the crash with "make monitor" running connected via USB.
"make monitor" will run a (very) simple python terminal which checks the
output for the GDBSTUB crash info and automatically invokes gdb.
.elf files of public builds are available in the respective firmware
folders on the servers.
The xtensa toolkit has the full range of GCC tools, check out the "bin"
directory.
Regards,
Michael
Am 01.01.21 um 13:55 schrieb Steve Davies:
> Hi,
>
> Thanks for the advice - managed to get backtraces but looks like I
> need to keep old ovms3.elfs.
>
> I didn't realise there is a full gdb - I found an option in 'make
> menuconfig" to write coredump to "flash" - will that work? Is Is
> there enough room?
>
> If I can get a full core file I can scratch in the stack frames and
> perhaps better understand that went wrong.
>
> Steve
>
>
> On Fri, 1 Jan 2021 at 12:54, Michael Balzer <dexter at expeedo.de
> <mailto:dexter at expeedo.de>> wrote:
>
> Steve,
>
> decode your backtraces using the "a2l" script (see support folder)
> and the .elf file of the build the crash occured with. The script
> by default uses the latest .elf file.
>
> You can find the crash backtraces of previous crashes on the
> server in table "*-OVM-DebugCrash" (kept for 30 days).
>
> Known crash issues:
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues?q=is%3Aissue+is%3Aopen+crash
> <https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues?q=is%3Aissue+is%3Aopen+crash>
>
> Any help with finding crash reasons is appreciated. I still am
> under the impression the PSRAM bug isn't solved for all edge
> cases, but also am certain we can find fatal bugs in our code as well.
>
> Regards,
> Michael
>
>
> Am 01.01.21 um 08:47 schrieb Steve Davies:
>> Hi Ethan,
>>
>> Thanks for the info. Naively doesn't seem like a javascript
>> program should be able to crash the whole system.
>>
>> I did think about rather posting to ABRP from the server side -
>> IE as data is sent to the server then code on the server sends
>> the update to ABRP.
>>
>> But maybe I buried the lead, which really was to find out how I
>> can get a lower level view of what crashed the board, and how to
>> turn those memory addresses into a better idea as to where in the
>> code the crash occurred.
>>
>> Do you know, or Michael can you help?
>>
>> Thanks,
>> Steve
>>
>>
>>
>> On Fri, 1 Jan 2021 at 03:11, Ethan Rose <ethan at ethanrose.co.nz
>> <mailto:ethan at ethanrose.co.nz>> wrote:
>>
>> I too have crashes often when running the ABRP plugin on my
>> NL ‘13. I’ve resorted to using the LeafSpy/BT option instead
>> on long drives as it’s very unreliable.
>>
>> Regards,
>>
>> Ethan Rose
>>
>>> On 1/01/2021, at 08:52, Steve Davies <steve at telviva.co.za
>>> <mailto:steve at telviva.co.za>> wrote:
>>>
>>>
>>> Well something had obviously gone off the rails since
>>> eventually the OVMS seemed to crash - couldn't ping it - and
>>> didn't come back.
>>>
>>> I power-cycled it and it came back and got a GPS lock.
>>>
>>> Was there ever discussion of a watchdog to reset things if
>>> it goes wonky?
>>>
>>> Steve
>>>
>>> On Thu, 31 Dec 2020 at 19:37, Steve Davies
>>> <steve at telviva.co.za <mailto:steve at telviva.co.za>> wrote:
>>>
>>> Hi,
>>>
>>> I went for a scenic drive partly to test my OVMS I3
>>> stuff and feeding data to ABRP.
>>>
>>> When I got home I found that ABRP had my drive ending
>>> say 15km from home. And when I checked OVMS it also
>>> thought I was in the same spot.
>>>
>>> [6315S] v.p.altitude 65.4m
>>> [6315S] v.p.direction 97.5°
>>> [6315S] v.p.gpshdop 1
>>> [9999S] v.p.gpslock yes
>>> [6315S] v.p.gpsmode AN
>>> [6315S] v.p.gpsspeed 18.7052km/h
>>> [6315S] v.p.latitude -34.0808
>>> [-----] v.p.location
>>> [6315S] v.p.longitude 18.4327
>>> [5282S] v.p.odometer 14901km
>>> [6315S] v.p.satcount 9
>>> [5195S] v.p.speed 0km/h
>>> [5195S] v.p.trip 85km
>>>
>>> You can see that the metrics are very old (I show more
>>> digits in my version).
>>>
>>> I then looked further and no metrics seemed to be
>>> updating - for instance the m.time.utc wasn't updating.
>>>
>>> I was starting to look at it when my OVMS rebooted
>>> (which seems to happen sometimes. I've also had the
>>> logging stop - I noticed one less task so I presume
>>> that's related).
>>>
>>>
>>> Last boot was 127 second(s) ago
>>> Time at boot: 2020-12-31 19:10:09 SAST
>>> This is reset #8 since last power cycle
>>> Detected boot reason: Crash (12/12)
>>> Reset reason: Exception/panic (4)
>>> Crash counters: 1 total, 0 early
>>>
>>> Last crash: LoadProhibited exception on core 1
>>> Registers:
>>> PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 :
>>> 0x3ffe5920
>>> A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 :
>>> 0x00000001
>>> A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 :
>>> 0x3ffe5910
>>> A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 :
>>> 0x00000001
>>> A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004
>>> EXCCAUSE: 0x0000001c
>>> EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01
>>> LCOUNT : 0xfffffffd
>>> Backtrace:
>>> 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02
>>> 0x400fe8af 0x4013a432
>>> Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build
>>> idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52)
>>>
>>> After this crash and restart m.time.utc was empty. I
>>> rebooted and its still missing - does it only appear
>>> once NTP syncs or something?
>>>
>>> Also I have no GPS position, gpsmode or anything like
>>> that (but I have a 3G session with the hologram sim and
>>> roaming on one of the local cellular operators).
>>>
>>> I've made sure the GPS puck is out in the open and I
>>> guess I'll wait - does the time come from the GPS? And
>>> any ideas why I'm not getting a GPS lock suddenly?
>>>
>>> Apart from that - how do I debug crashes? I went
>>> looking for some sort of loader memory map in the build
>>> directory to try turn the PC and backtrace addresses
>>> into something more useful but didn't pick up anything.
>>>
>>> Thanks
>>> Steve
>>>
>>>
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> <mailto:OvmsDev at lists.openvehicles.com>
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>> <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>> <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210101/98349b9c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210101/98349b9c/attachment-0001.sig>
More information about the OvmsDev
mailing list