Investigating crashes. And, OVMS stopped collecting data
Hi, I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP. When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot. [6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km You can see that the metrics are very old (I show more digits in my version). I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating. I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related). Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52) After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something? Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators). I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly? Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything. Thanks Steve
Well something had obviously gone off the rails since eventually the OVMS seemed to crash - couldn't ping it - and didn't come back. I power-cycled it and it came back and got a GPS lock. Was there ever discussion of a watchdog to reset things if it goes wonky? Steve On Thu, 31 Dec 2020 at 19:37, Steve Davies <steve@telviva.co.za> wrote:
Hi,
I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP.
When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot.
[6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km
You can see that the metrics are very old (I show more digits in my version).
I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating.
I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related).
Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early
Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52)
After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something?
Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators).
I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly?
Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything.
Thanks Steve
I too have crashes often when running the ABRP plugin on my NL ‘13. I’ve resorted to using the LeafSpy/BT option instead on long drives as it’s very unreliable. Regards, Ethan Rose
On 1/01/2021, at 08:52, Steve Davies <steve@telviva.co.za> wrote:
Well something had obviously gone off the rails since eventually the OVMS seemed to crash - couldn't ping it - and didn't come back.
I power-cycled it and it came back and got a GPS lock.
Was there ever discussion of a watchdog to reset things if it goes wonky?
Steve
On Thu, 31 Dec 2020 at 19:37, Steve Davies <steve@telviva.co.za> wrote: Hi,
I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP.
When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot.
[6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km
You can see that the metrics are very old (I show more digits in my version).
I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating.
I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related).
Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early
Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52)
After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something?
Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators).
I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly?
Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything.
Thanks Steve
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Hi Ethan, Thanks for the info. Naively doesn't seem like a javascript program should be able to crash the whole system. I did think about rather posting to ABRP from the server side - IE as data is sent to the server then code on the server sends the update to ABRP. But maybe I buried the lead, which really was to find out how I can get a lower level view of what crashed the board, and how to turn those memory addresses into a better idea as to where in the code the crash occurred. Do you know, or Michael can you help? Thanks, Steve On Fri, 1 Jan 2021 at 03:11, Ethan Rose <ethan@ethanrose.co.nz> wrote:
I too have crashes often when running the ABRP plugin on my NL ‘13. I’ve resorted to using the LeafSpy/BT option instead on long drives as it’s very unreliable.
Regards,
Ethan Rose
On 1/01/2021, at 08:52, Steve Davies <steve@telviva.co.za> wrote:
Well something had obviously gone off the rails since eventually the OVMS seemed to crash - couldn't ping it - and didn't come back.
I power-cycled it and it came back and got a GPS lock.
Was there ever discussion of a watchdog to reset things if it goes wonky?
Steve
On Thu, 31 Dec 2020 at 19:37, Steve Davies <steve@telviva.co.za> wrote:
Hi,
I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP.
When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot.
[6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km
You can see that the metrics are very old (I show more digits in my version).
I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating.
I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related).
Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early
Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52)
After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something?
Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators).
I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly?
Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything.
Thanks Steve
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Steve, decode your backtraces using the "a2l" script (see support folder) and the .elf file of the build the crash occured with. The script by default uses the latest .elf file. You can find the crash backtraces of previous crashes on the server in table "*-OVM-DebugCrash" (kept for 30 days). Known crash issues: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues?q=is... Any help with finding crash reasons is appreciated. I still am under the impression the PSRAM bug isn't solved for all edge cases, but also am certain we can find fatal bugs in our code as well. Regards, Michael Am 01.01.21 um 08:47 schrieb Steve Davies:
Hi Ethan,
Thanks for the info. Naively doesn't seem like a javascript program should be able to crash the whole system.
I did think about rather posting to ABRP from the server side - IE as data is sent to the server then code on the server sends the update to ABRP.
But maybe I buried the lead, which really was to find out how I can get a lower level view of what crashed the board, and how to turn those memory addresses into a better idea as to where in the code the crash occurred.
Do you know, or Michael can you help?
Thanks, Steve
On Fri, 1 Jan 2021 at 03:11, Ethan Rose <ethan@ethanrose.co.nz <mailto:ethan@ethanrose.co.nz>> wrote:
I too have crashes often when running the ABRP plugin on my NL ‘13. I’ve resorted to using the LeafSpy/BT option instead on long drives as it’s very unreliable.
Regards,
Ethan Rose
On 1/01/2021, at 08:52, Steve Davies <steve@telviva.co.za <mailto:steve@telviva.co.za>> wrote:
Well something had obviously gone off the rails since eventually the OVMS seemed to crash - couldn't ping it - and didn't come back.
I power-cycled it and it came back and got a GPS lock.
Was there ever discussion of a watchdog to reset things if it goes wonky?
Steve
On Thu, 31 Dec 2020 at 19:37, Steve Davies <steve@telviva.co.za <mailto:steve@telviva.co.za>> wrote:
Hi,
I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP.
When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot.
[6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km
You can see that the metrics are very old (I show more digits in my version).
I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating.
I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related).
Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early
Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52)
After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something?
Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators).
I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly?
Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything.
Thanks Steve
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Hi, Thanks for the advice - managed to get backtraces but looks like I need to keep old ovms3.elfs. I didn't realise there is a full gdb - I found an option in 'make menuconfig" to write coredump to "flash" - will that work? Is Is there enough room? If I can get a full core file I can scratch in the stack frames and perhaps better understand that went wrong. Steve On Fri, 1 Jan 2021 at 12:54, Michael Balzer <dexter@expeedo.de> wrote:
Steve,
decode your backtraces using the "a2l" script (see support folder) and the .elf file of the build the crash occured with. The script by default uses the latest .elf file.
You can find the crash backtraces of previous crashes on the server in table "*-OVM-DebugCrash" (kept for 30 days).
Known crash issues: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues?q=is...
Any help with finding crash reasons is appreciated. I still am under the impression the PSRAM bug isn't solved for all edge cases, but also am certain we can find fatal bugs in our code as well.
Regards, Michael
Am 01.01.21 um 08:47 schrieb Steve Davies:
Hi Ethan,
Thanks for the info. Naively doesn't seem like a javascript program should be able to crash the whole system.
I did think about rather posting to ABRP from the server side - IE as data is sent to the server then code on the server sends the update to ABRP.
But maybe I buried the lead, which really was to find out how I can get a lower level view of what crashed the board, and how to turn those memory addresses into a better idea as to where in the code the crash occurred.
Do you know, or Michael can you help?
Thanks, Steve
On Fri, 1 Jan 2021 at 03:11, Ethan Rose <ethan@ethanrose.co.nz> wrote:
I too have crashes often when running the ABRP plugin on my NL ‘13. I’ve resorted to using the LeafSpy/BT option instead on long drives as it’s very unreliable.
Regards,
Ethan Rose
On 1/01/2021, at 08:52, Steve Davies <steve@telviva.co.za> wrote:
Well something had obviously gone off the rails since eventually the OVMS seemed to crash - couldn't ping it - and didn't come back.
I power-cycled it and it came back and got a GPS lock.
Was there ever discussion of a watchdog to reset things if it goes wonky?
Steve
On Thu, 31 Dec 2020 at 19:37, Steve Davies <steve@telviva.co.za> wrote:
Hi,
I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP.
When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot.
[6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km
You can see that the metrics are very old (I show more digits in my version).
I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating.
I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related).
Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early
Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52)
After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something?
Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators).
I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly?
Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything.
Thanks Steve
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing listOvmsDev@lists.openvehicles.comhttp://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Steve, writing the core dump to flash isn't possible. The only option to attach gdb is to… 1. build with CONFIG_ESP32_PANIC_GDBSTUB enabled, then 2. produce the crash with "make monitor" running connected via USB. "make monitor" will run a (very) simple python terminal which checks the output for the GDBSTUB crash info and automatically invokes gdb. .elf files of public builds are available in the respective firmware folders on the servers. The xtensa toolkit has the full range of GCC tools, check out the "bin" directory. Regards, Michael Am 01.01.21 um 13:55 schrieb Steve Davies:
Hi,
Thanks for the advice - managed to get backtraces but looks like I need to keep old ovms3.elfs.
I didn't realise there is a full gdb - I found an option in 'make menuconfig" to write coredump to "flash" - will that work? Is Is there enough room?
If I can get a full core file I can scratch in the stack frames and perhaps better understand that went wrong.
Steve
On Fri, 1 Jan 2021 at 12:54, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Steve,
decode your backtraces using the "a2l" script (see support folder) and the .elf file of the build the crash occured with. The script by default uses the latest .elf file.
You can find the crash backtraces of previous crashes on the server in table "*-OVM-DebugCrash" (kept for 30 days).
Known crash issues: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues?q=is... <https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues?q=is%3Aissue+is%3Aopen+crash>
Any help with finding crash reasons is appreciated. I still am under the impression the PSRAM bug isn't solved for all edge cases, but also am certain we can find fatal bugs in our code as well.
Regards, Michael
Am 01.01.21 um 08:47 schrieb Steve Davies:
Hi Ethan,
Thanks for the info. Naively doesn't seem like a javascript program should be able to crash the whole system.
I did think about rather posting to ABRP from the server side - IE as data is sent to the server then code on the server sends the update to ABRP.
But maybe I buried the lead, which really was to find out how I can get a lower level view of what crashed the board, and how to turn those memory addresses into a better idea as to where in the code the crash occurred.
Do you know, or Michael can you help?
Thanks, Steve
On Fri, 1 Jan 2021 at 03:11, Ethan Rose <ethan@ethanrose.co.nz <mailto:ethan@ethanrose.co.nz>> wrote:
I too have crashes often when running the ABRP plugin on my NL ‘13. I’ve resorted to using the LeafSpy/BT option instead on long drives as it’s very unreliable.
Regards,
Ethan Rose
On 1/01/2021, at 08:52, Steve Davies <steve@telviva.co.za <mailto:steve@telviva.co.za>> wrote:
Well something had obviously gone off the rails since eventually the OVMS seemed to crash - couldn't ping it - and didn't come back.
I power-cycled it and it came back and got a GPS lock.
Was there ever discussion of a watchdog to reset things if it goes wonky?
Steve
On Thu, 31 Dec 2020 at 19:37, Steve Davies <steve@telviva.co.za <mailto:steve@telviva.co.za>> wrote:
Hi,
I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP.
When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot.
[6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km
You can see that the metrics are very old (I show more digits in my version).
I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating.
I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related).
Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early
Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52)
After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something?
Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators).
I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly?
Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything.
Thanks Steve
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
participants (3)
-
Ethan Rose -
Michael Balzer -
Steve Davies