[Ovmsdev] Transition to new esp-idf

Michael Balzer dexter at expeedo.de
Fri Jan 19 22:08:35 HKT 2018


Tried & kept :)

With vehicle module running, connectivity _and_ SD card mounted, I now have

    OVMS > module memory
    Free 8-bit 46200/285768, 32-bit 17948/45120, SPIRAM 0/0

Great find. I hope that won't affect SD write performance too badly.


While testing I found a bug: after a manual "sd unmount" I get lots of

    E (121106) sdmmc_req: handle_idle_state_events unhandled: 00000008 00000002

…and new mounts afterwards won't work.

Regards,
Michael


Am 19.01.2018 um 01:39 schrieb Mark Webb-Johnson:
> Michael: Can you try menuconfig Component / FAT Filesystem support / Use separate cache for each file -> OFF. Documentation shows that controlling the FS_TINY
> flag (which sounds like something we want).
>
> With wifi, DEMO vehicle module, sd mounted, and server v2 running, I get:
>
>     Free 8-bit 70136/282488, 32-bit 17660/44832, SPIRAM 0/0
>
>     OVMS > sd unmount
>     OVMS > module memory
>     Free 8-bit 77604/282488, 32-bit 17660/44832, SPIRAM 0/0
>     --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>     tiT                 356   1088      0      0        +16    -20     +0     +0
>     Tmr Svc              20    528      0      0       -860  -6444     +0     +0
>
>
> Using “separate cache for each file”, I get:
>
>     Free 8-bit 29156/282496, 32-bit 17660/44832, SPIRAM 0/0
>
>     OVMS > sd unmount
>     OVMS > module memory
>     Free 8-bit 57096/282496, 32-bit 17660/44832, SPIRAM 0/0
>     --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>     Tmr Svc               0    552      0      0         -4 -27780     +0     +0
>
>
> Seems to be 27KB vs 7KB per filesystem mount, and we have two (/sd and /store). Overall win with FS_TINY for FAT is close to 41KB.
>
> Regards, Mark.
>
>> On 17 Jan 2018, at 9:23 AM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>
>> Note also that the memory debugging option adds a bit of overhead.
>>
>> With debugging enabled (heap: light impact):
>>
>>     OVMS > module memory
>>     Free 8-bit 79868/282496, 32-bit 17660/44832, SPIRAM 0/0
>>     --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>     no task            5312      0      0      0      +5312     +0     +0     +0
>>     esp_timer         49568      0    644      0     +49568     +0   +644     +0
>>     main              39444      0      0      0     +39444     +0     +0     +0
>>     ipc0              11096      0      0      0     +11096     +0     +0     +0
>>     Housekeeping      24740  19956      0      0     +24740 +19956     +0     +0
>>     tiT                 132      0      0      0       +132     +0     +0     +0
>>     Tmr Svc               4  27800      0      0         +4 +27800     +0     +0
>>     ipc1                 12      0      0      0        +12     +0     +0     +0
>>     AsyncConsole          0     36  26404      0         +0    +36 +26404     +0
>>
>>     OVMS > metrics list m.freeram
>>     m.freeram                                79868
>>
>>
>> With debugging enabled (heap: comprehensive):
>>
>>     OVMS > module memory
>>     Free 8-bit 79728/282320, 32-bit 17968/45140, SPIRAM 0/0
>>     --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>     no task            5312      0      0      0      +5312     +0     +0     +0
>>     esp_timer         49568      0    644      0     +49568     +0   +644     +0
>>     main              39444      0      0      0     +39444     +0     +0     +0
>>     ipc0              11096      0      0      0     +11096     +0     +0     +0
>>     Housekeeping      24644  20056      0      0     +24644 +20056     +0     +0
>>     tiT                 132      0      0      0       +132     +0     +0     +0
>>     Tmr Svc               4  27800      0      0         +4 +27800     +0     +0
>>     ipc1                 12      0      0      0        +12     +0     +0     +0
>>     AsyncConsole          0     20  26404      0         +0    +20 +26404     +0
>>     I (15614) simcom: State timeout, transition to 13
>>     I (15614) simcom: State: Enter PoweredOff state
>>     OVMS > metrics list m.freeram
>>     m.freeram                                79728
>>
>>
>> With debugging disabled:
>>
>>     OVMS > module memory
>>     To use these debugging tools, must set CONFIG_HEAP_TASK_TRACKING=y
>>     and have updated openvehicles/esp-idf
>>     OVMS > metrics list m.freeram
>>     m.freeram                                99956
>>
>>
>> I think we’ve still got some stuff we can look at to lighten the impact on internal memory. Once we can get the 3.1 hardware out, I can focus on that. The
>> memory tools Steve has built are really complete and usable now. They let us see exactly what is going on.
>>
>> If we had known going in that the 520KB SRAM of the ESP32 would be 80% used by the system, and only 20% for us, perhaps things would have been done
>> differently. Anyway, we are where we are, and the WROVER module should be able to move a bunch of stuff off to PSRAM.
>>
>> Regards, Mark.
>>
>>> On 17 Jan 2018, at 12:59 AM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>
>>> Bluetooth is disabled. Has been for a while, I don't see a chance to enable it with the 3.0 module.
>>>
>>> I've also disabled Javascript support (duktape), Wolf, the web server and MDNS and the OBD2ECU. Telnet is enabled as I sometimes use that parallel to the
>>> async console.
>>>
>>> That config gives me…
>>> OVMS > module memory 
>>> Free 8-bit 117364/286352, 32-bit 17972/45144, SPIRAM 0/0
>>> …after reboot (without SD card), and with Wifi, modem, Twizy & server v2 running & connected…
>>>
>>> OVMS > module memory
>>> Free 8-bit 23436/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>
>>> …when idle, will drop further with transmissions pending etc.
>>>
>>> Opening a telnet session then results in:
>>>
>>> OVMS > module memory
>>> Free 8-bit 18372/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>
>>> …and still stable.
>>>
>>> Regards,
>>> Michael
>>>
>>>
>>> Am 16.01.2018 um 01:11 schrieb Mark Webb-Johnson:
>>>> I’m seeing this:
>>>>
>>>> OVMS > module memory
>>>> Free 8-bit 79812/282268, 32-bit 17816/44988, SPIRAM 0/0
>>>>
>>>> OVMS > sd unmount
>>>> Unmounted SD CARD
>>>>
>>>> OVMS > module memory
>>>> Free 8-bit 107768/282268, 32-bit 17816/44988, SPIRAM 0/0
>>>> --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>>> Tmr Svc               0     20      0      0         -4 -27780     +0     +0
>>>>
>>>> I guess it is FAT fs stuff, but we use that for /store as well so not sure why so much. I will look into it.
>>>>
>>>> With wifi client, ovms server v2, tesla roadster vehicle module, I get this:
>>>>
>>>> OVMS > module memory
>>>> Free 8-bit 54224/282268, 32-bit 17816/44988, SPIRAM 0/0
>>>>
>>>> With the OBDII HUD, that drops to 45KB.
>>>>
>>>> Do you have bluetooth enabled (in menuconfig)?
>>>>
>>>> Regards, Mark.
>>>>
>>>>> On 16 Jan 2018, at 3:35 AM, Michael Balzer <dexter at expeedo.de> wrote:
>>>>>
>>>>> I've found the missing RAM. It's allocated for the SD card:
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 117364/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>>
>>>>> I (34235) sdcard: SD CARD has been inserted. Auto-mounting...
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 89348/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>> --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>>>> Tmr Svc               4  27804      0      0         +4 +27804     +0     +0
>>>>>
>>>>> I (68065) sdcard: SD CARD has been removed.
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 117332/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>> --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>>>> Tmr Svc               0     12      0      0         -4 -27792     +0     +0
>>>>>
>>>>> …again…
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 117332/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>> --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>>>> Tmr Svc               0     12      0      0         -4 -27792     +0     +0
>>>>>
>>>>> I (114985) sdcard: SD CARD has been inserted. Auto-mounting...
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 89348/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>> --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>>>> Tmr Svc               4  27804      0      0         +4 +27792     +0     +0
>>>>>
>>>>> E (141205) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x107
>>>>> E (141205) diskio_sdmmc: sdmmc_read_blocks failed (263)
>>>>> I (141515) sdcard: SD CARD has been removed.
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 117332/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>> --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>>>> Tmr Svc               0     12      0      0         -4 -27792     +0     +0
>>>>>
>>>>>
>>>>> Btw, I think the RAM usage of the telnet server is also higher than before:
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 117332/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>>
>>>>> OVMS > wifi mode client 
>>>>> Starting WIFI as a client for any defined SSID
>>>>> I (380175) wifi: wifi firmware version: 403db1d
>>>>> I (380175) wifi: config NVS flash: enabled
>>>>> I (380175) wifi: config nano formating: disabled
>>>>> I (380185) wifi: Init dynamic tx buffer num: 16
>>>>> I (380185) wifi: Init data frame dynamic rx buffer num: 16
>>>>> I (380185) wifi: Init management frame dynamic rx buffer num: 16
>>>>> I (380185) wifi: wifi driver task: 3ffe3db4, prio:23, stack:4096
>>>>> I (380185) wifi: Init static rx buffer num: 4
>>>>> I (380185) wifi: Init dynamic rx buffer num: 16
>>>>> I (380195) wifi: wifi power manager task: 0x3ffe846c prio: 21 stack: 2560
>>>>> I (380195) wifi: mode : sta (30:ae:a4:37:25:88)
>>>>> W (390205) wifi: incorrect scan type: 1073534024
>>>>> I (392615) esp32wifi: Found SSID devolo-f4068d73a03e - trying to connect
>>>>> I (393945) wifi: n:11 2, o:1 0, ap:255 255, sta:11 2, prof:1
>>>>> I (394595) wifi: state: init -> auth (b0)
>>>>> I (394605) wifi: state: auth -> assoc (0)
>>>>> I (394605) wifi: state: assoc -> run (10)
>>>>> I (394625) wifi: connected with devolo-f4068d73a03e, channel 11
>>>>> I (394665) esp32wifi: WiFi UP with SSID: devolo-f4068d73a03e, MAC: 30:ae:a4:37:25:88, IP: 192.168.2.101, mask: 255.255.255.0, gw: 192.168.2.1
>>>>> I (394665) telnet: Launching Telnet Server
>>>>> I (397615) wifi: pm start, type:0
>>>>>
>>>>> OVMS > module memory 
>>>>> Free 8-bit 89700/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>> --Task--     Total DRAM D/IRAM   IRAM SPIRAM   +/- DRAM D/IRAM   IRAM SPIRAM
>>>>> tiT                 128    716      0      0         +0   +484     +0     +0
>>>>> AsyncConsole          0  16552  26404      0         +0 +16356     +0     +0
>>>>> NetManTask            8    488      0      0         +8   +488     +0     +0
>>>>> wifi                  0   1600      0      0         +0  +1600     +0     +0
>>>>> eventTask             0   7564      0      0         +0  +7564     +0     +0
>>>>>
>>>>>
>>>>> …but I'm not sure about that, haven't tested that in a while.
>>>>>
>>>>>
>>>>> So, SD cards now work, and I'll not be able to use them due to the RAM usage… :(
>>>>>
>>>>>
>>>>>
>>>>> Am 15.01.2018 um 19:21 schrieb Michael Balzer:
>>>>>> Transition went fine. "make flash" didn't even erase "/store", I had expected that to happen, good to know.
>>>>>>
>>>>>> And…
>>>>>>
>>>>>> OVMS > sd status 
>>>>>> SD CARD is inserted
>>>>>> Name: SL32G
>>>>>> Type: SDHC/SDXC
>>>>>> Speed: default speed
>>>>>> Size: 30436MB
>>>>>> CSD: ver=1, sector_size=512, capacity=62333952 read_bl_len=9
>>>>>> SCR: sd_spec=2, bus_width=5
>>>>>>
>>>>>> → me too (the minion) :)
>>>>>>
>>>>>>
>>>>>> BUT… after reconfiguration to the same components as before, I'm now missing about 30-40 KB of free RAM.
>>>>>>
>>>>>> After loading just the Twizy module & the v2 server, I'm now again down to…
>>>>>>
>>>>>> OVMS > module memory 
>>>>>> Free 8-bit 2632/286352, 32-bit 17972/45144, SPIRAM 0/0
>>>>>>
>>>>>> …so it crashes as soon as it needs to actually do anything, like sending a notification.
>>>>>>
>>>>>> This is without any of the new CAN logging stuff yet, that's stashed away for the transition.
>>>>>>
>>>>>> I'll now check the config again for any more options to save RAM…
>>>>>>
>>>>>> Regards,
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> Am 15.01.2018 um 06:48 schrieb Stephen Casner:
>>>>>>> In order for the the "module memory" and "module tasks" commands to
>>>>>>> work, the configuration needs to be changed from the sdk.default.hw30
>>>>>>> that Mark just provided. CONFIG_FREERTOS_USE_TRACE_FACILITY needs to
>>>>>>> be enabled, CONFIG_HEAP_POISONING must be enabled (LIGHT or
>>>>>>> COMPREHENSIVE), and CONFIG_HEAP_TASK_TRACKING must be enabled.  I have
>>>>>>> attached a diff for the sdkconfig.  That diff also includes a section
>>>>>>> for enabling telnet, which some of you may be using.
>>>>>>>
>>>>>>> Please also note that in this revised implementation of "module
>>>>>>> memory" the block address is now the address that is returned from
>>>>>>> malloc, not the address of the header that precedes it, and the length
>>>>>>> of the block is now exclusive of the debugging overhead.
>>>>>>>
>>>>>>>                                                         -- Steve
>>>>>>>
>>>>>>> On Mon, 15 Jan 2018, Mark Webb-Johnson wrote:
>>>>>>>
>>>>>>>> All committed and pushed now.
>>>>>>>>
>>>>>>>> Developers MUST now update their ESP IDF and XTENSA build tools to use this new build. They should also compare sdkconfig and sdkconfig.defaults.hw30 to verify differences. I also suggest you do a ‘make flash’ (rather than ‘make app-flash’) at least once, to update the bootloader to the latest (as bootloaders before v2.1 are no longer supported).
>>>>>>>>
>>>>>>>> Check
>>>>>>>> https://esp-idf.readthedocs.io/en/latest/get-started/index.html#setup-toolchain <https://esp-idf.readthedocs.io/en/latest/get-started/index.html#setup-toolchain> <https://esp-idf.readthedocs.io/en/latest/get-started/index.html#setup-toolchain> <https://esp-idf.readthedocs.io/en/latest/get-started/index.html#setup-toolchain>
>>>>>>>> to see the version of XTENSA toolchain required for latest Espressif IDF ‘master’ builds. As of this writing, it is 1.22.0-80-g6c4433a-5.2.0.
>>>>>>>>
>>>>>>>> I hope this goes smoothly for you.
>>>>>>>>
>>>>>>>> Regards, Mark.
>>>>>>>>
>>>>>>>> commit dfcc658557fc480bcccb09b23bc24399bb6e7497 (HEAD -> master, origin/master, origin/HEAD)
>>>>>>>> Author: Mark Webb-Johnson <mark at webb-johnson.net> <mailto:mark at webb-johnson.net>
>>>>>>>> Date:   Mon Jan 15 07:56:45 2018 +0800
>>>>>>>>
>>>>>>>>     Provide a sdkconfig.default.hw31 for OVMS v3.1 hardware
>>>>>>>>
>>>>>>>> commit 003592d553c881f735fb228b08836a13c306cec3
>>>>>>>> Merge: e58c5b6 b28db5e
>>>>>>>> Author: Mark Webb-Johnson <mark at webb-johnson.net> <mailto:mark at webb-johnson.net>
>>>>>>>> Date:   Mon Jan 15 07:49:13 2018 +0800
>>>>>>>>
>>>>>>>>     Merge branch 'for-master'. This requires ESP-IDF v3.0 support:
>>>>>>>>
>>>>>>>>     OVMS developers should now:
>>>>>>>>
>>>>>>>>     1] Pull the latest OpenVehicles IDF, and checkout MASTER branch.
>>>>>>>>
>>>>>>>>     2] Update XTENSA tools to match version required by Espressif for MASTER branch.
>>>>>>>>
>>>>>>>>     3] For ovms hardware v3.0, a sdkconfig default file sdkconfig.default.hw30 has
>>>>>>>>        been provided. That can be merged/copied to sdkconfig as appropriate.
>>>>>>>>
>>>>>>>>     4] For ovms hardware v3.1, a sdkconfig default file sdkconfig.default.hw31 will
>>>>>>>>        been provided in the next update.
>>>>>>>>
>>>>>>>>     5] It is recommended that developers perform at least one full 'make flash'
>>>>>>>>        with this version, to update the bootloader to latest. Note that the
>>>>>>>>        sdkconfig.default.* files are now set to require bootloader at least v2.1
>>>>>>>>        (or later), and bootloader v2.0 is no longer supported.
>>>>>>>>
>>>>>>>>     Enjoy.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 15 Jan 2018, at 7:25 AM, Mark Webb-Johnson <mark at webb-johnson.net> <mailto:mark at webb-johnson.net> wrote:
>>>>>>>>>
>>>>>>>>> Merge is ok, but triple checking a couple of things. Should be committed and pushed within the next half hour.
>>>>>>>>>
>>>>>>>>> Regards, Mark
>>>>>>>>>
>>>>>>>>>> On 15 Jan 2018, at 1:09 AM, Stephen Casner <casner at acm.org> <mailto:casner at acm.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Mark,
>>>>>>>>>>
>>>>>>>>>> It looks like you were not able to do the sdkconfig.default and merge
>>>>>>>>>> as you planned?
>>>>>>>>>>
>>>>>>>>>>                                                       -- Steve
>>>>>>>>>>
>>>>>>>>>>> On Fri, 12 Jan 2018, Stephen Casner wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Sat, 13 Jan 2018, Mark Webb-Johnson wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> @Steve can you update our master clone to latest from Espressif and
>>>>>>>>>>>> make sure your stuff is still ok?
>>>>>>>>>>> I have just now done this.  I had already done a rebase a few days ago
>>>>>>>>>>> before committing the improved version of the OS changes so I could
>>>>>>>>>>> issue a pull request.  There were a few commits since then, which I
>>>>>>>>>>> have now merged.  My code still runs correctly.
>>>>>>>>>>>
>>>>>>>>>>>> I guess steps for developers will be:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) pull the openvehicles IDF and switch to master branch. Sub module update.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) download and install xtensa build chain to match.
>>>>>>>>>>>>
>>>>>>>>>>>> 3) update OVMS master, make clean, check menu config, then build and play.
>>>>>>>>>>> Sounds right.
>>>>>>>>>>>
>>>>>>>>>>>> I can get my part done by Sunday 14th night (HKT), so suggest to do
>>>>>>>>>>>> the Merge of the OVMS firmware master branch then.
>>>>>>>>>>> I'll be sleeping then, so you can go ahead and do the merge.
>>>>>>>>>>>
>>>>>>>>>>>                                                       -- Steve
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>>>>>>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev <http://lists.teslaclub.hk/mailman/listinfo/ovmsdev>
>>>>>>>>> _______________________________________________
>>>>>>>>> OvmsDev mailing list
>>>>>>>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>>>>>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev <http://lists.teslaclub.hk/mailman/listinfo/ovmsdev>
>>>>>>> _______________________________________________
>>>>>>> OvmsDev mailing list
>>>>>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>>>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev <http://lists.teslaclub.hk/mailman/listinfo/ovmsdev>
>>>>>> -- 
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev <http://lists.teslaclub.hk/mailman/listinfo/ovmsdev>
>>>>> -- 
>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.teslaclub.hk
>>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>>>>
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.teslaclub.hk
>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>>>
>>> -- 
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.teslaclub.hk
> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20180119/b89ecf51/attachment.htm>


More information about the OvmsDev mailing list