[Ovmsdev] Crash report server statistics (dexters-web)

Everyone, TL;DR: we've got a huge improvement with 3.2.015, but still quite a lot crashes, with some focus on Kia Niro & Smart ED. Main crash reason now is the watchdog triggering, meaning some task running more or less continuously for over 60 seconds, so the low prio tasks don't get any CPU share. That crash type seems to happen on all car types. Total crash report counts per version and car type for cars registered at dexters-web.de, based on DebugCrash records of the last 3 days: crash_version c_type crash_count 3.2.008-101-ga8361db5 NL 66 3.2.009-7-gc57fb4c7-dirty MI 2 3.2.010-187-g54fa43ac NL 1 3.2.011-4-g9de2ecd9-dirty KN 1 3.2.013 TR2S 22 3.2.013-223-g7c41d3a6 KN 2 3.2.013-223-g7c41d3a6 RT 3 3.2.013-223-g7c41d3a6 SE 1 3.2.013-45-g10f2f729-dirty KN 2 3.2.014 KN 272 3.2.014 MI 1 3.2.014 NL 70 3.2.014 RT 15 3.2.014 SE 8 3.2.014 TR2N 1 3.2.014 TR2S 1 3.2.014-136-g06dbaf15-dirty SE 2 3.2.014-42-g6c884ce6 TR1N 1 3.2.014-84-g07f25e35 NL 8 3.2.015 KN 40 3.2.015 NL 1 3.2.015 NONE 2 3.2.015 RZ 2 3.2.015 SE 11 3.2.015-11-g5f5d9c8b-dirty MG 1 3.2.015-25-g3397409b KN 1 3.2.015-25-g3397409b NL 1 3.2.015-28-g13c10bcf-dirty VWUP.OBD 2 3.2.015-30-gffd0cace-dirty RT 1 3.2.015-39-g0b2819b9 SE 1 3.2.015-39-g0b2819b9 SQ 1 3.2.015-46-ge93058df MI 1 3.2.015-81-ga90d3a4c-dirty VWUP.OBD 1 Same query extended by crash type (reset reason & backtrace): crash_version c_type crash_count crash_resetreason crash_backtrace 3.2.008-101-ga8361db5 NL 66 Task watchdog 0x4008ae97 0x4008b12d 0x400e6fe4 0x40083d92 3.2.009-7-gc57fb4c7-dirty MI 2 Exception/panic 0x4008b013 0x4008b2a9 0x4010ea3b 0x4009792b 0x400840b9 0x40084721 0x4000bec7 0x401c8ad9 0x400f1f45 0x400f2b47 0x400f2b40 0x400f2b40 0x4015a971 0x4015981a 0x400f535d 0x400f5555 0x400f55bd 0x400f55cd 3.2.010-187-g54fa43ac NL 1 Exception/panic 0x400ff929 0x40161011 0x401616f5 0x4015e70a 0x4015ea45 0x4015f9be 0x4015fad5 3.2.011-4-g9de2ecd9-dirty KN 1 Task watchdog 0x4008b75f 0x4008b9f9 0x400e5a94 0x4008405a 3.2.013 TR2S 22 Task watchdog 0x4008b72f 0x4008b9c9 0x400e6abc 0x4008404a 3.2.013-223-g7c41d3a6 KN 2 Task watchdog 0x4008b03f 0x4008b2d5 0x400e70a4 0x40083d96 3.2.013-223-g7c41d3a6 RT 3 Task watchdog 0x4008b03f 0x4008b2d5 0x400e70a4 0x40083d96 3.2.013-223-g7c41d3a6 SE 1 Task watchdog 0x4008b03f 0x4008b2d5 0x400e70a4 0x40083d96 3.2.013-45-g10f2f729-dirty KN 1 Task watchdog 0x4008b75f 0x4008b9f9 0x400e58d0 0x4008405a 3.2.013-45-g10f2f729-dirty KN 1 Exception/panic 0x401087f3 0x40109f78 0x40134aa6 0x40134aec 0x40134751 0x400f5b09 0x400f5d22 0x400f5d99 0x400f5da9 3.2.014 KN 151 Task watchdog 0x4008e627 0x4008e8c1 0x400e6e34 0x40083f6a 3.2.014 KN 57 Unknown/unset 3.2.014 KN 34 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a 3.2.014 KN 13 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161 3.2.014 KN 12 Exception/panic 0x4008e627 0x4008e8c1 0x4008e8d8 0x40090920 0x40092270 0x40092226 3.2.014 KN 2 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x40131ff9 0x4013617e 0x40108302 0x40108572 0x401085e4 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161 3.2.014 KN 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f83fd 0x40160734 0x40293d7d 0x400f6711 0x400f676d 0x40293151 0x401678a0 0x40168484 0x40161d62 0x401622ce 0x4015f821 0x400f86ca 0x400f8865 0x400f894c 0x400f89ad 3.2.014 KN 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff 0x401064be 0x400f20c4 0x400f2161 3.2.014 MI 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a 3.2.014 MI 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5 0x4010a0c2 0x4010a3da 0x401064be 0x400f20c4 0x400f2161 3.2.014 NL 35 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151 0x40175f25 0x40176355 0x401625cd 0x40293d02 3.2.014 NL 21 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f83fd 0x4016054c 0x40293d7d 0x400f6711 0x400f676d 0x40293151 0x40175f25 0x401763c3 0x401625cd 0x40293d02 3.2.014 NL 4 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a 3.2.014 NL 3 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4 0x400f3145 0x40160684 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09 0x40176cb5 0x40177153 0x40162705 0x40293bc2 3.2.014 NL 2 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151 0x40175f25 0x40176379 0x401625cd 0x40293d02 3.2.014 NL 2 Exception/panic 0x4008e627 0x4008e8c1 0x400e090c 0x401c5a79 0x401c60c6 0x4012c361 0x4012c6fd 0x4012cca3 3.2.014 NL 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4 0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09 0x40176cb5 0x40177109 0x40162705 0x40293bc2 3.2.014 NL 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4 0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09 0x40176cb5 0x401770e5 0x40162705 0x40293bc2 3.2.014 NL 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4 0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1 0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed 3.2.014 RT 9 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a 3.2.014 RT 2 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x4013676b 0x401367ab 0x4013856e 0x4010b6c2 0x40108302 0x40109cb5 0x4010a10d 0x4010a3da 0x401064be 0x400f20c4 0x400f2161 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x4013676b 0x40136c43 0x40138006 0x40136302 0x400f86ca 0x400f8865 0x400f894c 0x400f89ad 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff 0x401064be 0x400f20c4 0x400f2161 3.2.014 RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x400f8675 0x400f86a9 0x4020dc91 3.2.014 SE 7 Task watchdog 0x4008e627 0x4008e8c1 0x400e612c 0x40083f6a 3.2.014 SE 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161 3.2.014 TR2N 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f7e98 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161 3.2.014 TR2S 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f2be4 0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1 0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed 3.2.014-136-g06dbaf15-dirty SE 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f40c4 0x400f46c9 0x40132107 0x4013684d 0x40108aba 0x40108d2a 0x40108dad 0x4010a63a 0x4010ab8a 0x40106c7a 0x400fb4ef 0x400fb59d 3.2.014-136-g06dbaf15-dirty SE 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e72d8 0x40083f6a 3.2.014-42-g6c884ce6 TR1N 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f85b8 0x400f8bbd 0x40132183 0x40136913 0x40108a1e 0x40108b90 0x4010ab13 0x40106bde 0x400f27d0 0x400f286d 3.2.014-84-g07f25e35 NL 7 Task watchdog 0x4008e627 0x4008e8c1 0x400e61e4 0x40083f6a 3.2.014-84-g07f25e35 NL 1 Exception/panic 0x4008e627 0x4008e8c1 0x400f8608 0x400f8b6d 0x4016132a 0x40295749 0x400f6e81 0x400f6edd 0x40294b15 0x401770cd 0x40177759 0x40163561 0x402956c6 3.2.015 KN 33 Unknown/unset 3.2.015 KN 5 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a 3.2.015 KN 2 Task watchdog 0x4008e627 0x4008e8c1 0x400e6f08 0x40083f6a 3.2.015 NL 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e6f08 0x40083f6a 3.2.015 NONE 2 Task watchdog 0x4008e627 0x4008e8c1 0x400e6f08 0x40083f6a 3.2.015 RZ 2 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a 3.2.015 SE 11 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a 3.2.015-11-g5f5d9c8b-dirty MG 1 Exception/panic 0x4017314a 0x4017324b 0x4017333a 0x4017334e 0x401644df 0x40164510 0x401645e4 0x400f541f 0x400f502e 0x400f4a16 0x400f4bb1 0x400f4c98 0x400f4cf9 3.2.015-25-g3397409b KN 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a 3.2.015-25-g3397409b NL 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e6200 0x40083f6a 3.2.015-28-g13c10bcf-dirty VWUP.OBD 2 Task watchdog 0x4008e627 0x4008e8c1 0x400e7050 0x40083f6a 3.2.015-30-gffd0cace-dirty RT 1 Exception/panic 0x4008e627 0x4008e8c1 0x400847e3 0x40084909 0x4011b3e9 0x4010d241 0x4010cd3d 0x4009389f 0x4008427d 0x400846e5 0x4000bec7 0x400ef696 0x400ef6a1 0x400eff52 0x400f0064 0x4014c7fd 0x4014c821 0x4014ced2 0x4014d2b7 0x4014db0c 0x40153c29 0x40108c1e 0x40109c4d 0x40108c1e 0x40108fe7 0x401090de 0x4010aa67 0x4010acee 0x40106dde 0x400f29c7 0x400f2a75 3.2.015-39-g0b2819b9 SE 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e6290 0x40083f6a 3.2.015-39-g0b2819b9 SQ 1 Exception/panic 0x400fec54 0x4015d851 0x4015de61 0x4015b169 0x4015b49b 0x4015c3d6 0x4015c4e9 3.2.015-46-ge93058df MI 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e7490 0x40083f6a 3.2.015-81-ga90d3a4c-dirty VWUP.OBD 1 Task watchdog 0x4008e627 0x4008e8c1 0x400e700c 0x40083f6a Car count by type: c_type c_count DBC 1 KN 28 KS 6 MB 1 MG 3 MI 6 NL 38 NONE 2 O2 2 RT 91 RZ 6 SE 27 SQ 2 TR 1 TR1N 1 TR2N 7 TR2S 9 TS 1 VAA12 1 VAA13 1 VAA14 1 VWUP 2 VWUP.OBD 1 XX 2 Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26