[Ovmsdev] v3 hardware disconnecting from v2 server
Mark Webb-Johnson
mark at webb-johnson.net
Tue Jan 30 08:31:56 HKT 2018
Strange. I don’t have the debugging enabled, but this is what I get:
OVMS > wifi mode client
Starting WIFI as a client for any defined SSID
I (108251) wifi: wifi firmware version: 403db1d
I (108251) wifi: config NVS flash: enabled
…
I (114861) wifi: connected with STUBBY, channel 11
I (116331) event: sta ip: 10.8.8.80, mask: 255.255.255.0, gw: 10.8.8.62
I (116331) ovms-mdns: Launching MDNS service
I (116331) esp32wifi: WiFi UP with SSID: STUBBY, MAC: 30:ae:a4:80:8d:b0, IP: 10.8.8.80, mask: 255.255.255.0, gw: 10.8.8.62
I (116331) webserver: Launching Web Server
I (116351) ssh: Launching SSH Server
I (117841) wifi: pm start, type:0
I (118881) webserver: HTTP GET /
I (120201) webserver: HTTP GET /
I (120991) webserver: HTTP GET /
OVMS > wifi mode off
Stopping wifi station...
I (123821) wifi: state: run -> init (0)
I (123821) wifi: pm stop, total sleep time: 0/5979789
I (123821) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
I (123841) wifi: flush txq
I (123841) wifi: stop sw txq
I (123841) wifi: lmac stop hw txq
I (123841) wifi: Deinit lldesc rx mblock:4
I (124121) webserver: Stopping Web Server
I (124121) ssh: Stopping SSH Server
OVMS > wifi mode client
Starting WIFI as a client for any defined SSID
I (126991) wifi: wifi firmware version: 403db1d
...
I (132851) esp32wifi: Found SSID STUBBY - trying to connect
I (134181) wifi: n:11 0, o:1 0, ap:255 255, sta:11 0, prof:1
I (134831) wifi: state: init -> auth (b0)
I (134831) wifi: state: auth -> assoc (0)
I (134841) wifi: state: assoc -> run (10)
I (134861) wifi: connected with STUBBY, channel 11
I (135571) event: sta ip: 10.8.8.80, mask: 255.255.255.0, gw: 10.8.8.62
I (135571) ovms-mdns: Launching MDNS service
I (135571) esp32wifi: WiFi UP with SSID: STUBBY, MAC: 30:ae:a4:80:8d:b0, IP: 10.8.8.80, mask: 255.255.255.0, gw: 10.8.8.62
I (135571) webserver: Launching Web Server
I (135591) ssh: Launching SSH Server
I (137071) webserver: HTTP GET /
I (137841) wifi: pm start, type:0
I (140001) webserver: HTTP GET /
Tried it 10 times to be certain.
Are you on the master branch of https://github.com/openvehicles/esp-idf.git <https://github.com/openvehicles/esp-idf.git> and up to date?
There are a bunch of menuconfig options in Components/LWIP. This is what I have:
│ │ [ ] Enable copy between Layer2 and Layer3 packets │ │
│ │ (10) Max number of open sockets │ │
│ │ [ ] Enable SO_REUSEADDR option │ │
│ │ [ ] Enable SO_RCVBUF option │ │
│ │ (1) Maximum number of NTP servers │ │
│ │ [ ] Enable fragment outgoing IP packets │ │
│ │ [ ] Enable reassembly incoming fragmented IP packets │ │
│ │ [ ] Enable LWIP statistics │ │
│ │ [*] Enable LWIP ARP trust │ │
│ │ (32) TCPIP task receive mail box size │ │
│ │ [ ] DHCP: Perform ARP check on any offered address │ │
│ │ DHCP server ---> │ │
│ │ [ ] Enable IPV4 Link-Local Addressing (AUTOIP) ---- │ │
│ │ [ ] Support per-interface loopback ---- │ │
│ │ TCP ---> │ │
│ │ UDP ---> │ │
│ │ (2560) TCP/IP Task Stack Size │ │
│ │ [*] Enable PPP support (new/experimental) ---> │ │
│ │ ICMP ---> │ │
│ │ LWIP RAW API --->
The SO_REUSEADDR option is relevant, but not enabled in my config. It does seem like mongoose is trying to use it. I wonder what LWIP default behaviour is in that case.
Regards, Mark.
> On 30 Jan 2018, at 5:51 AM, Michael Balzer <dexter at expeedo.de> wrote:
>
> I've added an option to send the mongoose debug log output to the ESP log framework and add a debug log to the if_destroy function.
>
> There was some talk about not changing the original mongoose files / maybe doing an OVMS fork, so I won't check the patch in. Diff attached.
>
> If you apply the patch: you'll need log level verbose for full debug details like these:
>
>
> I (14915) wifi: connected with devolo-f4068d73a03e, channel 11
> I (14915) events: Signal(system.wifi.sta.connected)
> I (14945) event: sta ip: 192.168.2.101, mask: 255.255.255.0, gw: 192.168.2.1
> I (14945) events: Signal(system.wifi.sta.gotip)
> I (14945) events: Signal(network.wifi.up)
> I (14945) ovms-mdns: Launching MDNS service
> I (14955) events: Signal(network.up)
> I (14955) esp32wifi: WiFi UP with SSID: devolo-f4068d73a03e, MAC: 30:ae:a4:37:25:88, IP: 192.168.2.101, mask: 255.255.255.0, gw: 192.168.2.1
> V (14965) mongoose: mg_socket_if_init 0x3ffb7348 using select()
> V (14965) mongoose: mg_mgr_init_opt ==================================
> V (14965) mongoose: mg_mgr_init_opt init mgr=0x3ffb7348
> I (14965) events: Signal(network.mgr.init)
> I (14965) webserver: Launching Web Server
> V (14995) mongoose: mg_socket_if_sock_se 0x3ffe068c 8192
> V (14995) mongoose: mg_add_conn 0x3ffb7348 0x3ffe068c
> I (14995) telnet: Launching Telnet Server
> V (15025) mongoose: mg_socket_if_sock_se 0x3ffe58f0 8193
> V (15025) mongoose: mg_add_conn 0x3ffb7348 0x3ffe58f0
> V (15495) simcom: tx: 41 54 0d 0a | AT..
> D (15495) simcom: tx scmd ch=0 len=4 : AT||
> I (15495) simcom: State timeout, transition to 13
> I (15495) simcom: State: Enter PoweredOff state
> I (17905) wifi: pm start, type:0
>
>
> now doing a HTTP GET /home from the browser:
>
> V (35485) mongoose: mg_mgr_handle_conn 0x3ffe068c fd=8192 fd_flags=1 nc_flags=1 rmbl=0 smbl=0
> V (35485) mongoose: mg_add_conn 0x3ffb7348 0x3ffe97e0
> V (35495) mongoose: mg_if_accept_new_con 0x3ffe068c 0x3ffe97e0 -1 0
> V (35495) mongoose: mg_accept_conn 0x3ffe97e0 conn from 192.168.2.103:33574
> V (35505) mongoose: mg_socket_if_sock_se 0x3ffe97e0 8194
> V (35505) mongoose: mg_call 0x3ffe97e0 proto ev=1 ev_data=0x3ffe97f8 flags=0 rmbl=0 smbl=0
> V (35505) mongoose: mg_call 0x3ffe97e0 user ev=1 ev_data=0x3ffe97f8 flags=0 rmbl=0 smbl=0
> D (35505) webserver: EventHandler: conn=0x3ffe97e0 ev=1 p=0x3ffe97f8
> V (35505) mongoose: mg_call 0x3ffe97e0 after user flags=0 rmbl=0 smbl=0
> V (35505) mongoose: mg_call 0x3ffe97e0 after proto flags=0 rmbl=0 smbl=0
> V (35505) mongoose: mg_mgr_handle_conn 0x3ffe068c after fd=8192 nc_flags=1 rmbl=0 smbl=0
> V (35505) mongoose: mg_mgr_handle_conn 0x3ffe97e0 fd=8194 fd_flags=1 nc_flags=0 rmbl=0 smbl=0
> V (35515) mongoose: mg_handle_tcp_read 0x3ffe97e0 350 bytes (PLAIN) <- 8194
> V (35515) mongoose: mg_recv_common 0x3ffe97e0 350 0
> V (35515) mongoose: mg_call 0x3ffe97e0 proto ev=3 ev_data=0x3ffe95d0 flags=0 rmbl=350 smbl=0
> V (35515) mongoose: mg_call 0x3ffe97e0 user ev=3 ev_data=0x3ffe95d0 flags=0 rmbl=350 smbl=0
> D (35515) webserver: EventHandler: conn=0x3ffe97e0 ev=3 p=0x3ffe95d0
> V (35515) mongoose: mg_call 0x3ffe97e0 after user flags=0 rmbl=350 smbl=0
> V (35515) mongoose: mg_http_handler2 0x3ffe97e0 192.168.2.103:33574 GET /home
> V (35515) mongoose: mg_call 0x3ffe97e0 user ev=102 ev_data=0x3ffe9400 flags=0 rmbl=350 smbl=0
> D (35515) webserver: EventHandler: conn=0x3ffe97e0 ev=102 p=0x3ffe9400
> V (35515) mongoose: mg_call 0x3ffe97e0 after user flags=0 rmbl=350 smbl=0
> V (35515) mongoose: mg_call 0x3ffe97e0 user ev=100 ev_data=0x3ffe9400 flags=0 rmbl=350 smbl=0
> D (35515) webserver: EventHandler: conn=0x3ffe97e0 ev=100 p=0x3ffe9400
> I (35515) webserver: HTTP GET /home
> V (35525) mongoose: mg_call 0x3ffe97e0 after user flags=0 rmbl=350 smbl=740
> V (35525) mongoose: mg_call 0x3ffe97e0 after proto flags=0 rmbl=0 smbl=740
> V (35525) mongoose: mg_mgr_handle_conn 0x3ffe97e0 after fd=8194 nc_flags=0 rmbl=0 smbl=740
> V (35525) mongoose: mg_mgr_handle_conn 0x3ffe97e0 fd=8194 fd_flags=2 nc_flags=0 rmbl=0 smbl=740
> V (35525) mongoose: mg_write_to_socket 0x3ffe97e0 740 bytes -> 8194
> V (35525) mongoose: mg_if_sent_cb 0x3ffe97e0 740
> V (35525) mongoose: mg_call 0x3ffe97e0 proto ev=4 ev_data=0x3ffe95e0 flags=0 rmbl=0 smbl=0
> V (35525) mongoose: mg_call 0x3ffe97e0 user ev=4 ev_data=0x3ffe95e0 flags=0 rmbl=0 smbl=0
> D (35525) webserver: EventHandler: conn=0x3ffe97e0 ev=4 p=0x3ffe95e0
> V (35525) mongoose: mg_call 0x3ffe97e0 after user flags=0 rmbl=0 smbl=0
> V (35525) mongoose: mg_call 0x3ffe97e0 after proto flags=0 rmbl=0 smbl=0
> V (35525) mongoose: mg_mgr_handle_conn 0x3ffe97e0 after fd=8194 nc_flags=0 rmbl=0 smbl=0
>
>
>
> OVMS > wifi mode off
> Stopping wifi station...
> I (41145) events: Signal(system.wifi.down)
> I (41145) events: Signal(network.wifi.down)
> I (41145) events: Signal(network.reconfigured)
> I (41145) events: Signal(network.down)
> I (41155) wifi: state: run -> init (0)
> I (41155) wifi: pm stop, total sleep time: 0/23262330
> I (41155) wifi: n:11 0, o:11 2, ap:255 255, sta:11 2, prof:1
> I (41165) wifi: flush txq
> I (41165) wifi: stop sw txq
> I (41165) wifi: lmac stop hw txq
> I (41165) wifi: Deinit lldesc rx mblock:4
> I (41165) events: Signal(system.wifi.sta.disconnected)
> I (41175) events: Signal(system.wifi.sta.stop)
> I (41525) events: Signal(network.mgr.stop)
> I (41525) webserver: Stopping Web Server
> I (41525) telnet: Stopping Telnet Server
> V (41535) mongoose: mg_mgr_free 0x3ffb7348
> V (41535) mongoose: mg_close_conn 0x3ffe97e0 0 8194
> V (41535) mongoose: mg_socket_if_destroy nc=0x3ffe97e0 sock=8194 flags=0
> V (41545) mongoose: mg_call 0x3ffe97e0 proto ev=5 ev_data=0x0 flags=0 rmbl=0 smbl=0
> V (41545) mongoose: mg_call 0x3ffe97e0 user ev=5 ev_data=0x0 flags=0 rmbl=0 smbl=0
> D (41545) webserver: EventHandler: conn=0x3ffe97e0 ev=5 p=0x0
> V (41545) mongoose: mg_call 0x3ffe97e0 after user flags=0 rmbl=0 smbl=0
> V (41545) mongoose: mg_call 0x3ffe97e0 after proto flags=0 rmbl=0 smbl=0
> V (41545) mongoose: mg_close_conn 0x3ffe58f0 1 8193
> V (41545) mongoose: mg_socket_if_destroy nc=0x3ffe58f0 sock=8193 flags=1
> V (41555) mongoose: mg_call 0x3ffe58f0 user ev=5 ev_data=0x0 flags=1 rmbl=0 smbl=0
> V (41555) mongoose: mg_call 0x3ffe58f0 after user flags=1 rmbl=0 smbl=0
> V (41555) mongoose: mg_close_conn 0x3ffe068c 1 8192
> V (41555) mongoose: mg_socket_if_destroy nc=0x3ffe068c sock=8192 flags=1
> V (41565) mongoose: mg_call 0x3ffe068c proto ev=5 ev_data=0x0 flags=1 rmbl=0 smbl=0
> V (41565) mongoose: mg_call 0x3ffe068c user ev=5 ev_data=0x0 flags=1 rmbl=0 smbl=0
> D (41565) webserver: EventHandler: conn=0x3ffe068c ev=5 p=0x0
> V (41565) mongoose: mg_call 0x3ffe068c after user flags=1 rmbl=0 smbl=0
> V (41565) mongoose: mg_call 0x3ffe068c after proto flags=1 rmbl=0 smbl=0
>
>
> … doesn't look wrong so far, socket get closed correctly. mg_socket_if_destroy calls closesocket = lwip_close_r, so everything _should_ be ok… but…
>
>
> OVMS > wifi mode client
> Starting WIFI as a client for any defined SSID
> I (47225) wifi: wifi firmware version: 403db1d
> I (47225) wifi: config NVS flash: enabled
> I (47225) wifi: config nano formating: disabled
> I (47225) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
> I (47225) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
> I (47245) wifi: Init dynamic tx buffer num: 16
> I (47245) wifi: Init data frame dynamic rx buffer num: 16
> I (47245) wifi: Init management frame dynamic rx buffer num: 16
> I (47245) wifi: wifi driver task: 3ffe251c, prio:23, stack:4096
> I (47245) wifi: Init static rx buffer num: 4
> I (47245) wifi: Init dynamic rx buffer num: 16
> I (47245) wifi: wifi power manager task: 0x3ffe5a9c prio: 21 stack: 2560
> I (47255) wifi: mode : sta (30:ae:a4:37:25:88)
> I (47255) events: Signal(system.wifi.sta.start)
> W (50495) wifi: incorrect scan type: 1073541416
> I (52905) events: Signal(system.wifi.scan.done)
> I (52905) esp32wifi: Found SSID devolo-f4068d73a03e - trying to connect
> I (54235) wifi: n:11 2, o:1 0, ap:255 255, sta:11 2, prof:1
> I (54885) wifi: state: init -> auth (b0)
> I (54895) wifi: state: auth -> assoc (0)
> I (54905) wifi: state: assoc -> run (10)
> I (54915) wifi: connected with devolo-f4068d73a03e, channel 11
> I (54915) events: Signal(system.wifi.sta.connected)
> I (54945) event: sta ip: 192.168.2.101, mask: 255.255.255.0, gw: 192.168.2.1
> I (54955) events: Signal(system.wifi.sta.gotip)
> I (54955) events: Signal(network.wifi.up)
> I (54955) ovms-mdns: Launching MDNS service
> I (54955) events: Signal(network.up)
> I (54965) esp32wifi: WiFi UP with SSID: devolo-f4068d73a03e, MAC: 30:ae:a4:37:25:88, IP: 192.168.2.101, mask: 255.255.255.0, gw: 192.168.2.1
> V (54965) mongoose: mg_socket_if_init 0x3ffb7348 using select()
> V (54965) mongoose: mg_mgr_init_opt ==================================
> V (54965) mongoose: mg_mgr_init_opt init mgr=0x3ffb7348
> I (54965) events: Signal(network.mgr.init)
> I (54965) webserver: Launching Web Server
> V (54995) mongoose: mg_bind_opt Failed to open listener: 112
> V (54995) mongoose: mg_socket_if_destroy nc=0x3ffe996c sock=-1 flags=1
> E (54995) webserver: Cannot bind to port 80: failed to open listener
> I (54995) telnet: Launching Telnet Server
> V (55025) mongoose: mg_socket_if_sock_se 0x3ffe98c8 8196
> V (55025) mongoose: mg_add_conn 0x3ffb7348 0x3ffe98c8
> I (57905) wifi: pm start, type:0
>
>
> … so it's indeed 112 = EADDRINUSE as assumed.
>
> As mongoose does everything right, this seems to be an LWIP issue? Mark, Steve?
>
>
> Am 29.01.2018 um 14:17 schrieb Mark Webb-Johnson:
>> The bind has to occur in the network.mgr.init handler. That is the point between mg_mgr_init and mg_mgr_poll.
>>
>> Perhaps in the handler we can check m_connected_wifi and only bind if wifi is up? But would also need to pickup network.reconfigured and check m_connected_wifi there as well.
>
> The webserver NetManInit function is already triggered on network.mgr.init and checks m_connected_wifi.
>
> Regards,
> Michael
>
>
>>
>> Regards, Mark.
>>
>>> On 29 Jan 2018, at 6:11 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>
>>> mg_bind() fails with "failed to open listener". I haven't traced it further yet, assumed it's the bind() that fails.
>>>
>>> Mongoose (now?) uses the standard socket API, so does not need to handle LWIP details, or does it?
>>>
>>> I can do some more debugging to get the exact point of failure.
>>>
>>> Regards,
>>> Michael
>>>
>>>
>>> Am 29.01.2018 um 02:28 schrieb Stephen Casner:
>>>> Mark,
>>>>
>>>> Back in November we discussed this problem of the client's half of the
>>>> connection being left open. Then as now, you commented that mongoose
>>>> sends a close event (MG_EV_CLOSE) for each open connection when the
>>>> interface went down.
>>>>
>>>> It does send a close event, but not until after the wifi is already
>>>> shut down so closing the socket at that point does not send any packet
>>>> to the client. I decided to punt on that issue, though, because
>>>> manually shutting down wifi is not an important use case. The more
>>>> likely case is that wifi connectivity is lost due to motion or other
>>>> causes and in that case no close packet can be delivered anyway.
>>>>
>>>> I interpreted Michael's message to be referring to a problem with
>>>> ports on the server (OVMS) end. That is, to say that mongoose didn't
>>>> clean up LWIP state properly. Michael, can you explain a bit more?
>>>>
>>>> -- Steve
>>>>
>>>> On Mon, 29 Jan 2018, Mark Webb-Johnson wrote:
>>>>
>>>>> I suspect that this is timing related:
>>>>>
>>>>> OVMS > wifi mode client XXX
>>>>> Starting WIFI as a client to XXX…
>>>>> …
>>>>> I (337100) ssh: Launching SSH Server
>>>>> V (338830) ssh: Event MG_EV_ACCEPT conn 0x3ffee8b4, data 0x3ffee8cc
>>>>> V (338840) ssh: Event MG_EV_RECV conn 0x3ffee8b4, data received 21
>>>>> V (338840) ssh: Event MG_EV_SEND conn 0x3ffee8b4, data 0x3ffe5d90
>>>>> V (338840) ssh: Event MG_EV_RECV conn 0x3ffee8b4, data received 1024
>>>>> V (338850) ssh: Event MG_EV_RECV conn 0x3ffee8b4, data received 408
>>>>> …
>>>>> OVMS > wifi mode off
>>>>> Stopping wifi station...
>>>>> I (350580) ovms-mdns: Stopping MDNS service
>>>>> I (350580) wifi: state: run -> init (0)
>>>>> I (350590) wifi: pm stop, total sleep time: 0/11930449
>>>>> I (350590) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
>>>>> I (350600) wifi: flush txq
>>>>> I (350600) wifi: stop sw txq
>>>>> I (350600) wifi: lmac stop hw txq
>>>>> I (350600) wifi: Deinit lldesc rx mblock:4
>>>>> I (351340) webserver: Stopping Web Server
>>>>> I (351340) ssh: Stopping SSH Server
>>>>> V (351540) ssh: Event MG_EV_CLOSE conn 0x3ffee8b4, data 0x0
>>>>> V (351540) ssh: Event MG_EV_CLOSE conn 0x3ffeb468, data 0x0
>>>>> I (357570) event: station ip lost
>>>>>
>>>>> But ssh client connection is still up on my workstation. Looks like
>>>>> the MG_EV_CLOSE events came in after the SSH server was stopped.
>>>>>
>>>>> I repeated the test, but with event logging on:
>>>>>
>>>>> OVMS > wifi mode off
>>>>> Stopping wifi station...
>>>>> I (34171) events: Signal(system.wifi.down)
>>>>> I (34171) events: Signal(network.wifi.down)
>>>>> I (34171) ovms-mdns: Stopping MDNS service
>>>>> I (34171) events: Signal(network.reconfigured)
>>>>> I (34171) events: Signal(network.down)
>>>>> I (34171) wifi: state: run -> init (0)
>>>>> I (34181) wifi: pm stop, total sleep time: 0/9603569
>>>>> I (34181) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
>>>>> I (34191) wifi: flush txq
>>>>> I (34191) wifi: stop sw txq
>>>>> I (34191) wifi: lmac stop hw txq
>>>>> I (34191) wifi: Deinit lldesc rx mblock:4
>>>>> I (34191) events: Signal(system.wifi.sta.disconnected)
>>>>> I (34191) events: Signal(system.wifi.sta.stop)
>>>>> I (35131) events: Signal(network.mgr.stop)
>>>>> I (35131) webserver: Stopping Web Server
>>>>> I (35131) ssh: Stopping SSH Server
>>>>> V (35331) ssh: Event MG_EV_CLOSE conn 0x3ffec138, data 0x0
>>>>> V (35331) ssh: Event MG_EV_CLOSE conn 0x3ffeb768, data 0x0
>>>>>
>>>>> MDNS exits quicker because it responds to the network.wifi.down message, while ssh uses the network.mgr.stop event. In a clean shutdown there is a difference.
>>>>>
>>>>> Perhaps we should issue network.mgr.stop earlier? However, even if we close the connections from our end, I’m not sure if the packet would make it through to the other end before the wifi disconnects.
>>>>>
>>>>> Regards, Mark.
>>>>>
>>>>>> On 29 Jan 2018, at 2:13 AM, Stephen Casner <casner at acm.org> <mailto:casner at acm.org> wrote:
>>>>>>
>>>>>> I handled wifi shutdowns cleanly when I first implemented telnet and
>>>>>> ssh as their own tasks. Now that they are under Mongoose, it is out
>>>>>> of my control. The socket is owned by the Mongoose code.
>>>>>>
>>>>>> -- Steve
>>>>>>
>>>>>> On Sun, 28 Jan 2018, Michael Balzer wrote:
>>>>>>
>>>>>>> I've begun working on the webserver and noticed something that may
>>>>>>> be correlated to this: sockets don't get closed when losing the
>>>>>>> connection. The effect is visible on both web and telnet server (ssh
>>>>>>> not tested). To reproduce, switch the Wifi network with an open
>>>>>>> connection, the port will not be available until timeout.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Michael
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OvmsDev mailing list
>>>>>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>>>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev <http://lists.teslaclub.hk/mailman/listinfo/ovmsdev>
>>>
>>>
>>> --
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev <http://lists.teslaclub.hk/mailman/listinfo/ovmsdev>
>>
>>
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev <http://lists.teslaclub.hk/mailman/listinfo/ovmsdev>
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> <mongoose-esp-log.diff>_______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.teslaclub.hk
> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20180130/c2d62f7f/attachment.htm>
More information about the OvmsDev
mailing list