[Ovmsdev] Crash report server statistics (dexters-web)

Michael Balzer dexter at expeedo.de
Mon Sep 7 17:33:47 HKT 2020 

Everyone,

TL;DR: we've got a huge improvement with 3.2.015, but still quite a lot
crashes, with some focus on Kia Niro & Smart ED.

Main crash reason now is the watchdog triggering, meaning some task
running more or less continuously for over 60 seconds, so the low prio
tasks don't get any CPU share. That crash type seems to happen on all
car types.

Total crash report counts per version and car type for cars registered
at dexters-web.de, based on DebugCrash records of the last 3 days:

crash_version 	c_type 	crash_count
3.2.008-101-ga8361db5 	NL 	66
3.2.009-7-gc57fb4c7-dirty 	MI 	2
3.2.010-187-g54fa43ac 	NL 	1
3.2.011-4-g9de2ecd9-dirty 	KN 	1
3.2.013 	TR2S 	22
3.2.013-223-g7c41d3a6 	KN 	2
3.2.013-223-g7c41d3a6 	RT 	3
3.2.013-223-g7c41d3a6 	SE 	1
3.2.013-45-g10f2f729-dirty 	KN 	2
3.2.014 	KN 	272
3.2.014 	MI 	1
3.2.014 	NL 	70
3.2.014 	RT 	15
3.2.014 	SE 	8
3.2.014 	TR2N 	1
3.2.014 	TR2S 	1
3.2.014-136-g06dbaf15-dirty 	SE 	2
3.2.014-42-g6c884ce6 	TR1N 	1
3.2.014-84-g07f25e35 	NL 	8
3.2.015 	KN 	40
3.2.015 	NL 	1
3.2.015 	NONE 	2
3.2.015 	RZ 	2
3.2.015 	SE 	11
3.2.015-11-g5f5d9c8b-dirty 	MG 	1
3.2.015-25-g3397409b 	KN 	1
3.2.015-25-g3397409b 	NL 	1
3.2.015-28-g13c10bcf-dirty 	VWUP.OBD 	2
3.2.015-30-gffd0cace-dirty 	RT 	1
3.2.015-39-g0b2819b9 	SE 	1
3.2.015-39-g0b2819b9 	SQ 	1
3.2.015-46-ge93058df 	MI 	1
3.2.015-81-ga90d3a4c-dirty 	VWUP.OBD 	1



Same query extended by crash type (reset reason & backtrace):

crash_version 	c_type 	crash_count 	crash_resetreason 	crash_backtrace
3.2.008-101-ga8361db5 	NL 	66 	Task watchdog 	0x4008ae97 0x4008b12d
0x400e6fe4 0x40083d92
3.2.009-7-gc57fb4c7-dirty 	MI 	2 	Exception/panic 	0x4008b013 0x4008b2a9
0x4010ea3b 0x4009792b 0x400840b9 0x40084721 0x4000bec7 0x401c8ad9
0x400f1f45 0x400f2b47 0x400f2b40 0x400f2b40 0x4015a971 0x4015981a
0x400f535d 0x400f5555 0x400f55bd 0x400f55cd
3.2.010-187-g54fa43ac 	NL 	1 	Exception/panic 	0x400ff929 0x40161011
0x401616f5 0x4015e70a 0x4015ea45 0x4015f9be 0x4015fad5
3.2.011-4-g9de2ecd9-dirty 	KN 	1 	Task watchdog 	0x4008b75f 0x4008b9f9
0x400e5a94 0x4008405a
3.2.013 	TR2S 	22 	Task watchdog 	0x4008b72f 0x4008b9c9 0x400e6abc
0x4008404a
3.2.013-223-g7c41d3a6 	KN 	2 	Task watchdog 	0x4008b03f 0x4008b2d5
0x400e70a4 0x40083d96
3.2.013-223-g7c41d3a6 	RT 	3 	Task watchdog 	0x4008b03f 0x4008b2d5
0x400e70a4 0x40083d96
3.2.013-223-g7c41d3a6 	SE 	1 	Task watchdog 	0x4008b03f 0x4008b2d5
0x400e70a4 0x40083d96
3.2.013-45-g10f2f729-dirty 	KN 	1 	Task watchdog 	0x4008b75f 0x4008b9f9
0x400e58d0 0x4008405a
3.2.013-45-g10f2f729-dirty 	KN 	1 	Exception/panic 	0x401087f3
0x40109f78 0x40134aa6 0x40134aec 0x40134751 0x400f5b09 0x400f5d22
0x400f5d99 0x400f5da9
3.2.014 	KN 	151 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6e34
0x40083f6a
3.2.014 	KN 	57 	Unknown/unset 	
3.2.014 	KN 	34 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a
3.2.014 	KN 	13 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
3.2.014 	KN 	12 	Exception/panic 	0x4008e627 0x4008e8c1 0x4008e8d8
0x40090920 0x40092270 0x40092226
3.2.014 	KN 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x40131ff9 0x4013617e 0x40108302 0x40108572
0x401085e4 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
3.2.014 	KN 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f83fd 0x40160734 0x40293d7d 0x400f6711 0x400f676d 0x40293151
0x401678a0 0x40168484 0x40161d62 0x401622ce 0x4015f821 0x400f86ca
0x400f8865 0x400f894c 0x400f89ad
3.2.014 	KN 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff
0x401064be 0x400f20c4 0x400f2161
3.2.014 	MI 	1 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a
3.2.014 	MI 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
0x4010a0c2 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
3.2.014 	NL 	35 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151
0x40175f25 0x40176355 0x401625cd 0x40293d02
3.2.014 	NL 	21 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f83fd 0x4016054c 0x40293d7d 0x400f6711 0x400f676d 0x40293151
0x40175f25 0x401763c3 0x401625cd 0x40293d02
3.2.014 	NL 	4 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a
3.2.014 	NL 	3 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
0x400f3145 0x40160684 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
0x40176cb5 0x40177153 0x40162705 0x40293bc2
3.2.014 	NL 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151
0x40175f25 0x40176379 0x401625cd 0x40293d02
3.2.014 	NL 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400e090c
0x401c5a79 0x401c60c6 0x4012c361 0x4012c6fd 0x4012cca3
3.2.014 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
0x40176cb5 0x40177109 0x40162705 0x40293bc2
3.2.014 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
0x40176cb5 0x401770e5 0x40162705 0x40293bc2
3.2.014 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1
0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed
3.2.014 	RT 	9 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a
3.2.014 	RT 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x4013676b 0x401367ab 0x4013856e 0x4010b6c2 0x40108302
0x40109cb5 0x4010a10d 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x4013676b 0x40136c43 0x40138006 0x40136302 0x400f86ca
0x400f8865 0x400f894c 0x400f89ad
3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff
0x401064be 0x400f20c4 0x400f2161
3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x400f8675 0x400f86a9 0x4020dc91
3.2.014 	SE 	7 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a
3.2.014 	SE 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
3.2.014 	TR2N 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
3.2.014 	TR2S 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1
0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed
3.2.014-136-g06dbaf15-dirty 	SE 	1 	Exception/panic 	0x4008e627
0x4008e8c1 0x400f40c4 0x400f46c9 0x40132107 0x4013684d 0x40108aba
0x40108d2a 0x40108dad 0x4010a63a 0x4010ab8a 0x40106c7a 0x400fb4ef
0x400fb59d
3.2.014-136-g06dbaf15-dirty 	SE 	1 	Task watchdog 	0x4008e627 0x4008e8c1
0x400e72d8 0x40083f6a
3.2.014-42-g6c884ce6 	TR1N 	1 	Exception/panic 	0x4008e627 0x4008e8c1
0x400f85b8 0x400f8bbd 0x40132183 0x40136913 0x40108a1e 0x40108b90
0x4010ab13 0x40106bde 0x400f27d0 0x400f286d
3.2.014-84-g07f25e35 	NL 	7 	Task watchdog 	0x4008e627 0x4008e8c1
0x400e61e4 0x40083f6a
3.2.014-84-g07f25e35 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1
0x400f8608 0x400f8b6d 0x4016132a 0x40295749 0x400f6e81 0x400f6edd
0x40294b15 0x401770cd 0x40177759 0x40163561 0x402956c6
3.2.015 	KN 	33 	Unknown/unset 	
3.2.015 	KN 	5 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a
3.2.015 	KN 	2 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6f08 0x40083f6a
3.2.015 	NL 	1 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6f08 0x40083f6a
3.2.015 	NONE 	2 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6f08
0x40083f6a
3.2.015 	RZ 	2 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a
3.2.015 	SE 	11 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a
3.2.015-11-g5f5d9c8b-dirty 	MG 	1 	Exception/panic 	0x4017314a
0x4017324b 0x4017333a 0x4017334e 0x401644df 0x40164510 0x401645e4
0x400f541f 0x400f502e 0x400f4a16 0x400f4bb1 0x400f4c98 0x400f4cf9
3.2.015-25-g3397409b 	KN 	1 	Task watchdog 	0x4008e627 0x4008e8c1
0x400e6200 0x40083f6a
3.2.015-25-g3397409b 	NL 	1 	Task watchdog 	0x4008e627 0x4008e8c1
0x400e6200 0x40083f6a
3.2.015-28-g13c10bcf-dirty 	VWUP.OBD 	2 	Task watchdog 	0x4008e627
0x4008e8c1 0x400e7050 0x40083f6a
3.2.015-30-gffd0cace-dirty 	RT 	1 	Exception/panic 	0x4008e627
0x4008e8c1 0x400847e3 0x40084909 0x4011b3e9 0x4010d241 0x4010cd3d
0x4009389f 0x4008427d 0x400846e5 0x4000bec7 0x400ef696 0x400ef6a1
0x400eff52 0x400f0064 0x4014c7fd 0x4014c821 0x4014ced2 0x4014d2b7
0x4014db0c 0x40153c29 0x40108c1e 0x40109c4d 0x40108c1e 0x40108fe7
0x401090de 0x4010aa67 0x4010acee 0x40106dde 0x400f29c7 0x400f2a75
3.2.015-39-g0b2819b9 	SE 	1 	Task watchdog 	0x4008e627 0x4008e8c1
0x400e6290 0x40083f6a
3.2.015-39-g0b2819b9 	SQ 	1 	Exception/panic 	0x400fec54 0x4015d851
0x4015de61 0x4015b169 0x4015b49b 0x4015c3d6 0x4015c4e9
3.2.015-46-ge93058df 	MI 	1 	Task watchdog 	0x4008e627 0x4008e8c1
0x400e7490 0x40083f6a
3.2.015-81-ga90d3a4c-dirty 	VWUP.OBD 	1 	Task watchdog 	0x4008e627
0x4008e8c1 0x400e700c 0x40083f6a



Car count by type:

c_type 	c_count
DBC 	1
KN 	28
KS 	6
MB 	1
MG 	3
MI 	6
NL 	38
NONE 	2
O2 	2
RT 	91
RZ 	6
SE 	27
SQ 	2
TR 	1
TR1N 	1
TR2N 	7
TR2S 	9
TS 	1
VAA12 	1
VAA13 	1
VAA14 	1
VWUP 	2
VWUP.OBD 	1
XX 	2



Regards,
Michael

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20200907/0a3ed17f/attachment-0001.html>

More information about the OvmsDev mailing list