[Ovmsdev] Crash report server statistics (dexters-web)

Michael Balzer dexter at expeedo.de
Tue Sep 8 15:24:29 HKT 2020


Mark,

in case you'd like to do the same on the main server, this is my query:

SELECT
  LEFT(h_data, INSTR(h_data,'/')-1) AS firmware_version,
  SUBSTRING_INDEX(SUBSTRING_INDEX(m_msg, ',', 5), ',', -1) AS car_type,
  COUNT(*) AS crashtype_count,
  CONVERT_TZ(MAX(h_timestamp),'+00:00','SYSTEM') AS last_time,
  IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 12,
    SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',13),',',-1), '') AS
resetreason_name,
  SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',8),',',-1) AS crashtype,
  SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',9),',',-1) AS crashcore,
  IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
    SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',14),',',-1), '') AS
curr_event_name,
  IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
    SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',15),',',-1), '') AS
curr_event_handler,
  IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
    SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',16),',',-1), '') AS
curr_event_runtime,
  IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
    SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',17),',',-1), '') AS
wdt_tasknames,
  SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',11),',',-1) AS backtrace,
  SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',10),',',-1) AS registers
FROM ovms_historicalmessages h
JOIN ovms_carmessages c ON ( c.vehicleid = h.vehicleid AND m_code = 'F' )
WHERE
  h_recordtype = '*-OVM-DebugCrash' AND h_data LIKE '%build idf%'
  AND h_timestamp >= NOW() - INTERVAL 10 DAY
GROUP BY firmware_version, car_type, resetreason_name, backtrace,
curr_event_name, wdt_tasknames
ORDER BY firmware_version, car_type, last_time


Regards,
Michael


Am 08.09.20 um 09:18 schrieb Michael Balzer:
> Everyone,
>
> to enable all developers to retrieve the crash reports from my server,
> I've added a cron job to create a CSV report.
>
> The CSV report can be downloaded from here:
>
>   * https://ovms.dexters-web.de/firmware/developer/
>   * …direct URL:
>     https://ovms.dexters-web.de/firmware/developer/crashreport.csv
>
> The report covers all crash reports of the last 10 days and is updated
> once per hour.
>
> As explained before, to analyse the backtraces you will need a copy of
> the build's ELF file (build/ovms.elf) to use with the "a2l" script.
>
> The ELF files of my OTA builds can be downloaded from the respective
> OTA release directory:
>
>   * https://ovms.dexters-web.de/firmware/ota/
>
> Regards,
> Michael
>
>
> Am 07.09.20 um 20:42 schrieb Michael Balzer:
>> Soko,
>>
>> I know, but cannot tell you who is running that VWUP.OBD module
>> without permission of that user. I can tell him you'd like to get in
>> contact if you want me to do that.
>>
>> Crashes show in the boot status and are transmitted to the server (if
>> running the v2/v3 client). Download the records "*-OVM-DebugCrash" to
>> check your history, these records are kept on the server for 30 days.
>>
>> The reset counter in the boot status is for reboots of any kind.
>>
>> You can help by inspecting "your" backtraces using the .elf files and
>> a2l. You probably can't do that on the "3.2.015-81-ga90d3a4c-dirty"
>> build used by the VWUP.OBD user, that seems to be a local build of
>> that user. You can generally ask your beta testers to send their
>> crash reports to you. Elf files of the public releases are available
>> in the corresponding download directory in bzip2 compressed form,
>> example:
>>
>> https://ovms.dexters-web.de/firmware/ota/v3.1/3.2.015-47-g2d9d1215/
>>
>> Regards,
>> Michael
>>
>>
>> Am 07.09.20 um 18:06 schrieb Soko:
>>>
>>> Hi Michael,
>>>
>>> Interesting... especially the crash of VWUP.OBD do you know who's
>>> running this? I'm not quite sure how such crashes show but as far as
>>> I interpret my current status I haven't had any crash since 7 days:
>>> ------------------------
>>> Last boot was 533173 second(s) ago Time at boot: 2020-09-01 13:58:04
>>> CEST This is reset #7 since last power cycle Detected boot reason:
>>> SoftReset (12/12) Reset reason: Unknown/unset (0) Crash counters: 0
>>> total, 0 early
>>>
>>> ------------------------------------------------------------------------
>>>
>>> Firmware: 3.2.014-126-gf4c489b5-dirty/ota_0/main (build idf
>>> v3.3.2-881-g22d636b7b Sep 1 2020 12:48:35) Running partition: ota_0
>>> Boot partition: ota_0 Factory image: 3.2.013-65-gb8088aa9-dirty
>>> OTA_O image: 3.2.014-126-gf4c489b5-dirty OTA_1 image:
>>> 3.2.014-126-gf4c489b5-dirty ------------------------
>>>
>>> The #7 reset is from flashing the new firmware via OTA, correct?
>>>
>>> Anything I can activate/look-out-for to help here?
>>>
>>> Soko
>>>
>>> On 07.09.2020 11:33, Michael Balzer wrote:
>>>> Everyone,
>>>>
>>>> TL;DR: we've got a huge improvement with 3.2.015, but still quite a
>>>> lot crashes, with some focus on Kia Niro & Smart ED.
>>>>
>>>> Main crash reason now is the watchdog triggering, meaning some task
>>>> running more or less continuously for over 60 seconds, so the low
>>>> prio tasks don't get any CPU share. That crash type seems to happen
>>>> on all car types.
>>>>
>>>> Total crash report counts per version and car type for cars
>>>> registered at dexters-web.de, based on DebugCrash records of the
>>>> last 3 days:
>>>>
>>>> crash_version 	c_type 	crash_count
>>>> 3.2.008-101-ga8361db5 	NL 	66
>>>> 3.2.009-7-gc57fb4c7-dirty 	MI 	2
>>>> 3.2.010-187-g54fa43ac 	NL 	1
>>>> 3.2.011-4-g9de2ecd9-dirty 	KN 	1
>>>> 3.2.013 	TR2S 	22
>>>> 3.2.013-223-g7c41d3a6 	KN 	2
>>>> 3.2.013-223-g7c41d3a6 	RT 	3
>>>> 3.2.013-223-g7c41d3a6 	SE 	1
>>>> 3.2.013-45-g10f2f729-dirty 	KN 	2
>>>> 3.2.014 	KN 	272
>>>> 3.2.014 	MI 	1
>>>> 3.2.014 	NL 	70
>>>> 3.2.014 	RT 	15
>>>> 3.2.014 	SE 	8
>>>> 3.2.014 	TR2N 	1
>>>> 3.2.014 	TR2S 	1
>>>> 3.2.014-136-g06dbaf15-dirty 	SE 	2
>>>> 3.2.014-42-g6c884ce6 	TR1N 	1
>>>> 3.2.014-84-g07f25e35 	NL 	8
>>>> 3.2.015 	KN 	40
>>>> 3.2.015 	NL 	1
>>>> 3.2.015 	NONE 	2
>>>> 3.2.015 	RZ 	2
>>>> 3.2.015 	SE 	11
>>>> 3.2.015-11-g5f5d9c8b-dirty 	MG 	1
>>>> 3.2.015-25-g3397409b 	KN 	1
>>>> 3.2.015-25-g3397409b 	NL 	1
>>>> 3.2.015-28-g13c10bcf-dirty 	VWUP.OBD 	2
>>>> 3.2.015-30-gffd0cace-dirty 	RT 	1
>>>> 3.2.015-39-g0b2819b9 	SE 	1
>>>> 3.2.015-39-g0b2819b9 	SQ 	1
>>>> 3.2.015-46-ge93058df 	MI 	1
>>>> 3.2.015-81-ga90d3a4c-dirty 	VWUP.OBD 	1
>>>>
>>>>
>>>>
>>>> Same query extended by crash type (reset reason & backtrace):
>>>>
>>>> crash_version 	c_type 	crash_count 	crash_resetreason 	crash_backtrace
>>>> 3.2.008-101-ga8361db5 	NL 	66 	Task watchdog 	0x4008ae97 0x4008b12d
>>>> 0x400e6fe4 0x40083d92
>>>> 3.2.009-7-gc57fb4c7-dirty 	MI 	2 	Exception/panic 	0x4008b013
>>>> 0x4008b2a9 0x4010ea3b 0x4009792b 0x400840b9 0x40084721 0x4000bec7
>>>> 0x401c8ad9 0x400f1f45 0x400f2b47 0x400f2b40 0x400f2b40 0x4015a971
>>>> 0x4015981a 0x400f535d 0x400f5555 0x400f55bd 0x400f55cd
>>>> 3.2.010-187-g54fa43ac 	NL 	1 	Exception/panic 	0x400ff929
>>>> 0x40161011 0x401616f5 0x4015e70a 0x4015ea45 0x4015f9be 0x4015fad5
>>>> 3.2.011-4-g9de2ecd9-dirty 	KN 	1 	Task watchdog 	0x4008b75f
>>>> 0x4008b9f9 0x400e5a94 0x4008405a
>>>> 3.2.013 	TR2S 	22 	Task watchdog 	0x4008b72f 0x4008b9c9 0x400e6abc
>>>> 0x4008404a
>>>> 3.2.013-223-g7c41d3a6 	KN 	2 	Task watchdog 	0x4008b03f 0x4008b2d5
>>>> 0x400e70a4 0x40083d96
>>>> 3.2.013-223-g7c41d3a6 	RT 	3 	Task watchdog 	0x4008b03f 0x4008b2d5
>>>> 0x400e70a4 0x40083d96
>>>> 3.2.013-223-g7c41d3a6 	SE 	1 	Task watchdog 	0x4008b03f 0x4008b2d5
>>>> 0x400e70a4 0x40083d96
>>>> 3.2.013-45-g10f2f729-dirty 	KN 	1 	Task watchdog 	0x4008b75f
>>>> 0x4008b9f9 0x400e58d0 0x4008405a
>>>> 3.2.013-45-g10f2f729-dirty 	KN 	1 	Exception/panic 	0x401087f3
>>>> 0x40109f78 0x40134aa6 0x40134aec 0x40134751 0x400f5b09 0x400f5d22
>>>> 0x400f5d99 0x400f5da9
>>>> 3.2.014 	KN 	151 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6e34
>>>> 0x40083f6a
>>>> 3.2.014 	KN 	57 	Unknown/unset 	
>>>> 3.2.014 	KN 	34 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c
>>>> 0x40083f6a
>>>> 3.2.014 	KN 	13 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	KN 	12 	Exception/panic 	0x4008e627 0x4008e8c1 0x4008e8d8
>>>> 0x40090920 0x40092270 0x40092226
>>>> 3.2.014 	KN 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x40131ff9 0x4013617e 0x40108302 0x40108572
>>>> 0x401085e4 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	KN 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f83fd 0x40160734 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>>> 0x401678a0 0x40168484 0x40161d62 0x401622ce 0x4015f821 0x400f86ca
>>>> 0x400f8865 0x400f894c 0x400f89ad
>>>> 3.2.014 	KN 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff
>>>> 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	MI 	1 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c
>>>> 0x40083f6a
>>>> 3.2.014 	MI 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>>> 0x4010a0c2 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	NL 	35 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>>> 0x40175f25 0x40176355 0x401625cd 0x40293d02
>>>> 3.2.014 	NL 	21 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f83fd 0x4016054c 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>>> 0x40175f25 0x401763c3 0x401625cd 0x40293d02
>>>> 3.2.014 	NL 	4 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c
>>>> 0x40083f6a
>>>> 3.2.014 	NL 	3 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
>>>> 0x400f3145 0x40160684 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
>>>> 0x40176cb5 0x40177153 0x40162705 0x40293bc2
>>>> 3.2.014 	NL 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f83fd 0x4016059a 0x40293d7d 0x400f6711 0x400f676d 0x40293151
>>>> 0x40175f25 0x40176379 0x401625cd 0x40293d02
>>>> 3.2.014 	NL 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400e090c
>>>> 0x401c5a79 0x401c60c6 0x4012c361 0x4012c6fd 0x4012cca3
>>>> 3.2.014 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
>>>> 0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
>>>> 0x40176cb5 0x40177109 0x40162705 0x40293bc2
>>>> 3.2.014 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
>>>> 0x400f3145 0x401606d2 0x40293c3d 0x400ed511 0x400ed56d 0x40292d09
>>>> 0x40176cb5 0x401770e5 0x40162705 0x40293bc2
>>>> 3.2.014 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
>>>> 0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1
>>>> 0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed
>>>> 3.2.014 	RT 	9 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c
>>>> 0x40083f6a
>>>> 3.2.014 	RT 	2 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x4013676b 0x401367ab 0x4013856e 0x4010b6c2 0x40108302
>>>> 0x40109cb5 0x4010a10d 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x4013676b 0x40136c43 0x40138006 0x40136302 0x400f86ca
>>>> 0x400f8865 0x400f894c 0x400f89ad
>>>> 3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x401361f3 0x40108302 0x40108474 0x4010a3ff
>>>> 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	RT 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x400f8675 0x400f86a9 0x4020dc91
>>>> 3.2.014 	SE 	7 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e612c
>>>> 0x40083f6a
>>>> 3.2.014 	SE 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	TR2N 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f7e98
>>>> 0x400f849d 0x40131a63 0x401361a9 0x40108302 0x40108572 0x401085f5
>>>> 0x40109e82 0x4010a3da 0x401064be 0x400f20c4 0x400f2161
>>>> 3.2.014 	TR2S 	1 	Exception/panic 	0x4008e627 0x4008e8c1 0x400f2be4
>>>> 0x400f31e5 0x40131ac3 0x40136209 0x401082be 0x4010852e 0x401085b1
>>>> 0x40109e3e 0x4010a396 0x4010647a 0x400fb63f 0x400fb6ed
>>>> 3.2.014-136-g06dbaf15-dirty 	SE 	1 	Exception/panic 	0x4008e627
>>>> 0x4008e8c1 0x400f40c4 0x400f46c9 0x40132107 0x4013684d 0x40108aba
>>>> 0x40108d2a 0x40108dad 0x4010a63a 0x4010ab8a 0x40106c7a 0x400fb4ef
>>>> 0x400fb59d
>>>> 3.2.014-136-g06dbaf15-dirty 	SE 	1 	Task watchdog 	0x4008e627
>>>> 0x4008e8c1 0x400e72d8 0x40083f6a
>>>> 3.2.014-42-g6c884ce6 	TR1N 	1 	Exception/panic 	0x4008e627
>>>> 0x4008e8c1 0x400f85b8 0x400f8bbd 0x40132183 0x40136913 0x40108a1e
>>>> 0x40108b90 0x4010ab13 0x40106bde 0x400f27d0 0x400f286d
>>>> 3.2.014-84-g07f25e35 	NL 	7 	Task watchdog 	0x4008e627 0x4008e8c1
>>>> 0x400e61e4 0x40083f6a
>>>> 3.2.014-84-g07f25e35 	NL 	1 	Exception/panic 	0x4008e627 0x4008e8c1
>>>> 0x400f8608 0x400f8b6d 0x4016132a 0x40295749 0x400f6e81 0x400f6edd
>>>> 0x40294b15 0x401770cd 0x40177759 0x40163561 0x402956c6
>>>> 3.2.015 	KN 	33 	Unknown/unset 	
>>>> 3.2.015 	KN 	5 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6200
>>>> 0x40083f6a
>>>> 3.2.015 	KN 	2 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6f08
>>>> 0x40083f6a
>>>> 3.2.015 	NL 	1 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6f08
>>>> 0x40083f6a
>>>> 3.2.015 	NONE 	2 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6f08
>>>> 0x40083f6a
>>>> 3.2.015 	RZ 	2 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6200
>>>> 0x40083f6a
>>>> 3.2.015 	SE 	11 	Task watchdog 	0x4008e627 0x4008e8c1 0x400e6200
>>>> 0x40083f6a
>>>> 3.2.015-11-g5f5d9c8b-dirty 	MG 	1 	Exception/panic 	0x4017314a
>>>> 0x4017324b 0x4017333a 0x4017334e 0x401644df 0x40164510 0x401645e4
>>>> 0x400f541f 0x400f502e 0x400f4a16 0x400f4bb1 0x400f4c98 0x400f4cf9
>>>> 3.2.015-25-g3397409b 	KN 	1 	Task watchdog 	0x4008e627 0x4008e8c1
>>>> 0x400e6200 0x40083f6a
>>>> 3.2.015-25-g3397409b 	NL 	1 	Task watchdog 	0x4008e627 0x4008e8c1
>>>> 0x400e6200 0x40083f6a
>>>> 3.2.015-28-g13c10bcf-dirty 	VWUP.OBD 	2 	Task watchdog 	0x4008e627
>>>> 0x4008e8c1 0x400e7050 0x40083f6a
>>>> 3.2.015-30-gffd0cace-dirty 	RT 	1 	Exception/panic 	0x4008e627
>>>> 0x4008e8c1 0x400847e3 0x40084909 0x4011b3e9 0x4010d241 0x4010cd3d
>>>> 0x4009389f 0x4008427d 0x400846e5 0x4000bec7 0x400ef696 0x400ef6a1
>>>> 0x400eff52 0x400f0064 0x4014c7fd 0x4014c821 0x4014ced2 0x4014d2b7
>>>> 0x4014db0c 0x40153c29 0x40108c1e 0x40109c4d 0x40108c1e 0x40108fe7
>>>> 0x401090de 0x4010aa67 0x4010acee 0x40106dde 0x400f29c7 0x400f2a75
>>>> 3.2.015-39-g0b2819b9 	SE 	1 	Task watchdog 	0x4008e627 0x4008e8c1
>>>> 0x400e6290 0x40083f6a
>>>> 3.2.015-39-g0b2819b9 	SQ 	1 	Exception/panic 	0x400fec54 0x4015d851
>>>> 0x4015de61 0x4015b169 0x4015b49b 0x4015c3d6 0x4015c4e9
>>>> 3.2.015-46-ge93058df 	MI 	1 	Task watchdog 	0x4008e627 0x4008e8c1
>>>> 0x400e7490 0x40083f6a
>>>> 3.2.015-81-ga90d3a4c-dirty 	VWUP.OBD 	1 	Task watchdog 	0x4008e627
>>>> 0x4008e8c1 0x400e700c 0x40083f6a
>>>>
>>>>
>>>>
>>>> Car count by type:
>>>>
>>>> c_type 	c_count
>>>> DBC 	1
>>>> KN 	28
>>>> KS 	6
>>>> MB 	1
>>>> MG 	3
>>>> MI 	6
>>>> NL 	38
>>>> NONE 	2
>>>> O2 	2
>>>> RT 	91
>>>> RZ 	6
>>>> SE 	27
>>>> SQ 	2
>>>> TR 	1
>>>> TR1N 	1
>>>> TR2N 	7
>>>> TR2S 	9
>>>> TS 	1
>>>> VAA12 	1
>>>> VAA13 	1
>>>> VAA14 	1
>>>> VWUP 	2
>>>> VWUP.OBD 	1
>>>> XX 	2
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>> -- 
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>
>> -- 
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
> -- 
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20200908/ceceab31/attachment-0001.html>


More information about the OvmsDev mailing list