[Ovmsdev] Crash report server statistics (dexters-web)
Michael Balzer
dexter at expeedo.de
Tue Sep 8 15:18:27 HKT 2020
Everyone,
to enable all developers to retrieve the crash reports from my server,
I've added a cron job to create a CSV report.
The CSV report can be downloaded from here:
* https://ovms.dexters-web.de/firmware/developer/
* …direct URL:
https://ovms.dexters-web.de/firmware/developer/crashreport.csv
The report covers all crash reports of the last 10 days and is updated
once per hour.
As explained before, to analyse the backtraces you will need a copy of
the build's ELF file (build/ovms.elf) to use with the "a2l" script.
The ELF files of my OTA builds can be downloaded from the respective OTA
release directory:
* https://ovms.dexters-web.de/firmware/ota/
Regards,
Michael
Am 07.09.20 um 20:42 schrieb Michael Balzer:
> Soko,
>
> I know, but cannot tell you who is running that VWUP.OBD module
> without permission of that user. I can tell him you'd like to get in
> contact if you want me to do that.
>
> Crashes show in the boot status and are transmitted to the server (if
> running the v2/v3 client). Download the records "*-OVM-DebugCrash" to
> check your history, these records are kept on the server for 30 days.
>
> The reset counter in the boot status is for reboots of any kind.
>
> You can help by inspecting "your" backtraces using the .elf files and
> a2l. You probably can't do that on the "3.2.015-81-ga90d3a4c-dirty"
> build used by the VWUP.OBD user, that seems to be a local build of
> that user. You can generally ask your beta testers to send their crash
> reports to you. Elf files of the public releases are available in the
> corresponding download directory in bzip2 compressed form, example:
>
> https://ovms.dexters-web.de/firmware/ota/v3.1/3.2.015-47-g2d9d1215/
>
> Regards,
> Michael
>
>
> Am 07.09.20 um 18:06 schrieb Soko:
>>
>> Hi Michael,
>>
>> Interesting... especially the crash of VWUP.OBD do you know who's
>> running this? I'm not quite sure how such crashes show but as far as
>> I interpret my current status I haven't had any crash since 7 days:
>> ------------------------
>> Last boot was 533173 second(s) ago Time at boot: 2020-09-01 13:58:04
>> CEST This is reset #7 since last power cycle Detected boot reason:
>> SoftReset (12/12) Reset reason: Unknown/unset (0) Crash counters: 0
>> total, 0 early
>>
>> ------------------------------------------------------------------------
>>
>> Firmware: 3.2.014-126-gf4c489b5-dirty/ota_0/main (build idf
>> v3.3.2-881-g22d636b7b Sep 1 2020 12:48:35) Running partition: ota_0
>> Boot partition: ota_0 Factory image: 3.2.013-65-gb8088aa9-dirty OTA_O
>> image: 3.2.014-126-gf4c489b5-dirty OTA_1 image:
>> 3.2.014-126-gf4c489b5-dirty ------------------------
>>
>> The #7 reset is from flashing the new firmware via OTA, correct?
>>
>> Anything I can activate/look-out-for to help here?
>>
>> Soko
>>
>> On 07.09.2020 11:33, Michael Balzer wrote:
>>> Everyone,
>>>
>>> TL;DR: we've got a huge improvement with 3.2.015, but still quite a
>>> lot crashes, with some focus on Kia Niro & Smart ED.
>>>
>>> Main crash reason now is the watchdog triggering, meaning some task
>>> running more or less continuously for over 60 seconds, so the low
>>> prio tasks don't get any CPU share. That crash type seems to happen
>>> on all car types.
>>>
>>> Total crash report counts per version and car type for cars
>>> registered at dexters-web.de, based on DebugCrash records of the
>>> last 3 days:
>>>
>>> crash_version c_type crash_count
>>> 3.2.008-101-ga8361db5 NL 66
>>> 3.2.009-7-gc57fb4c7-dirty MI 2
>>> 3.2.010-187-g54fa43ac NL 1
>>> 3.2.011-4-g9de2ecd9-dirty KN 1
>>> 3.2.013 TR2S 22
>>> 3.2.013-223-g7c41d3a6 KN 2
>>> 3.2.013-223-g7c41d3a6 RT 3
>>> 3.2.013-223-g7c41d3a6 SE 1
>>> 3.2.013-45-g10f2f729-dirty KN 2
>>> 3.2.014 KN 272
>>> 3.2.014 MI 1
>>> 3.2.014 NL 70
>>> 3.2.014 RT 15
>>> 3.2.014 SE 8
>>> 3.2.014 TR2N 1
>>> 3.2.014 TR2S 1
>>> 3.2.014-136-g06dbaf15-dirty SE 2
>>> 3.2.014-42-g6c884ce6 TR1N 1
>>> 3.2.014-84-g07f25e35 NL 8
>>> 3.2.015 KN 40
>>> 3.2.015 NL 1
>>> 3.2.015 NONE 2
>>> 3.2.015 RZ 2
>>> 3.2.015 SE 11
>>> 3.2.015-11-g5f5d9c8b-dirty MG 1
>>> 3.2.015-25-g3397409b KN 1
>>> 3.2.015-25-g3397409b NL 1
>>> 3.2.015-28-g13c10bcf-dirty VWUP.OBD 2
>>> 3.2.015-30-gffd0cace-dirty RT 1
>>> 3.2.015-39-g0b2819b9 SE 1
>>> 3.2.015-39-g0b2819b9 SQ 1
>>> 3.2.015-46-ge93058df MI 1
>>> 3.2.015-81-ga90d3a4c-dirty VWUP.OBD 1
>>>
>>>
>>>
>>> Same query extended by crash type (reset reason & backtrace):
>>>
>>> crash_version c_type crash_count crash_resetreason crash_backtrace
>>> 3.2.008-101-ga8361db5 NL 66 Task watchdog 0x4008ae97 0x4008b12d
>>> 0x400e6fe4 0x40083d92
>>> 3.2.009-7-gc57fb4c7-dirty MI 2 Exception/panic 0x4008b013
>>> 0x4008b2a9 0x4010ea3b 0x4009792b 0x400840b9 0x40084721 0x4000bec7
>>> 0x401c8ad9 0x400f1f45 0x400f2b47 0x400f2b40 0x400f2b40 0x4015a971
>>> 0x4015981a 0x400f535d 0x400f5555 0x400f55bd 0x400f55cd
>>> 3.2.010-187-g54fa43ac NL 1 Exception/panic 0x400ff929 0x40161011
>>> 0x401616f5 0x4015e70a 0x4015ea45 0x4015f9be 0x4015fad5
>>> 3.2.011-4-g9de2ecd9-dirty KN 1 Task watchdog 0x4008b75f
>>> 0x4008b9f9 0x400e5a94 0x4008405a
>>> 3.2.013 TR2S 22 Task watchdog 0x4008b72f 0x4008b9c9 0x400e6abc
>>> 0x4008404a
>>> 3.2.013-223-g7c41d3a6 KN 2 Task watchdog 0x4008b03f 0x4008b2d5
>>> 0x400e70a4 0x40083d96
>>> 3.2.013-223-g7c41d3a6 RT 3 Task watchdog 0x4008b03f 0x4008b2d5
>>> 0x400e70a4 0x40083d96
>>> 3.2.013-223-g7c41d3a6 SE 1 Task watchdog 0x4008b03f 0x4008b2d5
>>> 0x400e70a4 0x40083d96
>>> 3.2.013-45-g10f2f729-dirty KN 1 Task watchdog 0x4008b75f
>>> 0x4008b9f9 0x400e58d0 0x4008405a
>>> 3.2.013-45-g10f2f729-dirty KN 1 Exception/panic 0x401087f3
>>> 0x40109f78 0x40134aa6 0x40134aec 0x40134751 0x400f5b09 0x400f5d22
>>> 0x400f5d99 0x400f5da9
>>> 3.2.014 KN 151 Task watchdog 0x4008e627 0x4008e8c1 0x400e6e34
>>> 0x40083f6a
>>> 3.2.014 KN 57 Unknown/unset
>>> 3.2.014 KN 34 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c
>>> 0x40083f6a
>>> 3.2.014 KN 13 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 KN 12 Exception/panic 0x4008e627 0x4008e8c1 0x4008e8d8
>>> 0x40090920 0x40092270 0x40092226
>>> 3.2.014 KN 2 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x40131ff9 0x4013617e 0x40108302 0x40108572
>>> 0x401085e4 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 KN 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f83fd 0x40160734 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>> 0x401678a0 0x40168484 0x40161d62 0x401622ce 0x4015f821 0x400f86ca
>>> 0x400f8865 0x400f894c 0x400f89ad
>>> 3.2.014 KN 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff
>>> 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 MI 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c
>>> 0x40083f6a
>>> 3.2.014 MI 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>> 0x4010a0c2 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 NL 35 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>> 0x40175f25 0x40176355 0x401625cd 0x40293d02
>>> 3.2.014 NL 21 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f83fd 0x4016054c 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>> 0x40175f25 0x401763c3 0x401625cd 0x40293d02
>>> 3.2.014 NL 4 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c
>>> 0x40083f6a
>>> 3.2.014 NL 3 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4
>>> 0x400f3145 0x40160684 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
>>> 0x40176cb5 0x40177153 0x40162705 0x40293bc2
>>> 3.2.014 NL 2 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>> 0x40175f25 0x40176379 0x401625cd 0x40293d02
>>> 3.2.014 NL 2 Exception/panic 0x4008e627 0x4008e8c1 0x400e090c
>>> 0x401c5a79 0x401c60c6 0x4012c361 0x4012c6fd 0x4012cca3
>>> 3.2.014 NL 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4
>>> 0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
>>> 0x40176cb5 0x40177109 0x40162705 0x40293bc2
>>> 3.2.014 NL 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4
>>> 0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
>>> 0x40176cb5 0x401770e5 0x40162705 0x40293bc2
>>> 3.2.014 NL 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4
>>> 0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1
>>> 0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed
>>> 3.2.014 RT 9 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c
>>> 0x40083f6a
>>> 3.2.014 RT 2 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x4013676b 0x401367ab 0x4013856e 0x4010b6c2 0x40108302
>>> 0x40109cb5 0x4010a10d 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x4013676b 0x40136c43 0x40138006 0x40136302 0x400f86ca
>>> 0x400f8865 0x400f894c 0x400f89ad
>>> 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff
>>> 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x400f8675 0x400f86a9 0x4020dc91
>>> 3.2.014 SE 7 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c
>>> 0x40083f6a
>>> 3.2.014 SE 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 TR2N 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98
>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>> 3.2.014 TR2S 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4
>>> 0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1
>>> 0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed
>>> 3.2.014-136-g06dbaf15-dirty SE 1 Exception/panic 0x4008e627
>>> 0x4008e8c1 0x400f40c4 0x400f46c9 0x40132107 0x4013684d 0x40108aba
>>> 0x40108d2a 0x40108dad 0x4010a63a 0x4010ab8a 0x40106c7a 0x400fb4ef
>>> 0x400fb59d
>>> 3.2.014-136-g06dbaf15-dirty SE 1 Task watchdog 0x4008e627
>>> 0x4008e8c1 0x400e72d8 0x40083f6a
>>> 3.2.014-42-g6c884ce6 TR1N 1 Exception/panic 0x4008e627
>>> 0x4008e8c1 0x400f85b8 0x400f8bbd 0x40132183 0x40136913 0x40108a1e
>>> 0x40108b90 0x4010ab13 0x40106bde 0x400f27d0 0x400f286d
>>> 3.2.014-84-g07f25e35 NL 7 Task watchdog 0x4008e627 0x4008e8c1
>>> 0x400e61e4 0x40083f6a
>>> 3.2.014-84-g07f25e35 NL 1 Exception/panic 0x4008e627 0x4008e8c1
>>> 0x400f8608 0x400f8b6d 0x4016132a 0x40295749 0x400f6e81 0x400f6edd
>>> 0x40294b15 0x401770cd 0x40177759 0x40163561 0x402956c6
>>> 3.2.015 KN 33 Unknown/unset
>>> 3.2.015 KN 5 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200
>>> 0x40083f6a
>>> 3.2.015 KN 2 Task watchdog 0x4008e627 0x4008e8c1 0x400e6f08
>>> 0x40083f6a
>>> 3.2.015 NL 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e6f08
>>> 0x40083f6a
>>> 3.2.015 NONE 2 Task watchdog 0x4008e627 0x4008e8c1 0x400e6f08
>>> 0x40083f6a
>>> 3.2.015 RZ 2 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200
>>> 0x40083f6a
>>> 3.2.015 SE 11 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200
>>> 0x40083f6a
>>> 3.2.015-11-g5f5d9c8b-dirty MG 1 Exception/panic 0x4017314a
>>> 0x4017324b 0x4017333a 0x4017334e 0x401644df 0x40164510 0x401645e4
>>> 0x400f541f 0x400f502e 0x400f4a16 0x400f4bb1 0x400f4c98 0x400f4cf9
>>> 3.2.015-25-g3397409b KN 1 Task watchdog 0x4008e627 0x4008e8c1
>>> 0x400e6200 0x40083f6a
>>> 3.2.015-25-g3397409b NL 1 Task watchdog 0x4008e627 0x4008e8c1
>>> 0x400e6200 0x40083f6a
>>> 3.2.015-28-g13c10bcf-dirty VWUP.OBD 2 Task watchdog 0x4008e627
>>> 0x4008e8c1 0x400e7050 0x40083f6a
>>> 3.2.015-30-gffd0cace-dirty RT 1 Exception/panic 0x4008e627
>>> 0x4008e8c1 0x400847e3 0x40084909 0x4011b3e9 0x4010d241 0x4010cd3d
>>> 0x4009389f 0x4008427d 0x400846e5 0x4000bec7 0x400ef696 0x400ef6a1
>>> 0x400eff52 0x400f0064 0x4014c7fd 0x4014c821 0x4014ced2 0x4014d2b7
>>> 0x4014db0c 0x40153c29 0x40108c1e 0x40109c4d 0x40108c1e 0x40108fe7
>>> 0x401090de 0x4010aa67 0x4010acee 0x40106dde 0x400f29c7 0x400f2a75
>>> 3.2.015-39-g0b2819b9 SE 1 Task watchdog 0x4008e627 0x4008e8c1
>>> 0x400e6290 0x40083f6a
>>> 3.2.015-39-g0b2819b9 SQ 1 Exception/panic 0x400fec54 0x4015d851
>>> 0x4015de61 0x4015b169 0x4015b49b 0x4015c3d6 0x4015c4e9
>>> 3.2.015-46-ge93058df MI 1 Task watchdog 0x4008e627 0x4008e8c1
>>> 0x400e7490 0x40083f6a
>>> 3.2.015-81-ga90d3a4c-dirty VWUP.OBD 1 Task watchdog 0x4008e627
>>> 0x4008e8c1 0x400e700c 0x40083f6a
>>>
>>>
>>>
>>> Car count by type:
>>>
>>> c_type c_count
>>> DBC 1
>>> KN 28
>>> KS 6
>>> MB 1
>>> MG 3
>>> MI 6
>>> NL 38
>>> NONE 2
>>> O2 2
>>> RT 91
>>> RZ 6
>>> SE 27
>>> SQ 2
>>> TR 1
>>> TR1N 1
>>> TR2N 7
>>> TR2S 9
>>> TS 1
>>> VAA12 1
>>> VAA13 1
>>> VAA14 1
>>> VWUP 2
>>> VWUP.OBD 1
>>> XX 2
>>>
>>>
>>>
>>> Regards,
>>> Michael
>>>
>>> --
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20200908/2717032e/attachment.htm>
More information about the OvmsDev
mailing list