<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Reviving this old topic still impacting us…<div class=""><br class=""></div><div class="">Comparing the master vs for-v3.3 branches, the only thing that changed related to ppp is that in 3.3 the ppp object is dynamically created and destroyed, while in master it is statically allocated at boot. The ppp code is so simple, and I can’t see how that could be a problem. Perhaps related to position in memory, and some other memory corruption?</div><div class=""><br class=""></div><div class="">Anyway, I changed it to not destroy the ppp object when the gsm connection is lost, but merely shutdown the ppp (which is what master branch does). While it is still dynamically allocated, it is no longer as dynamic (being created just once at startup of the cellular system). I’ve never managed to reliably repeat this problem in my environment, but I think this should help.It has been running on my desktop test unit for the past four days without issue.</div><div class=""><br class=""></div><div class="">That code is committed now. I would appreciate it if others who saw this problem could try again with this latest build of the for-v3.3 branch.<br class=""><div class=""><br class=""></div><div class="">Regards, Mark.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 24 Mar 2021, at 3:53 PM, Mark Webb-Johnson <<a href="mailto:mark@webb-johnson.net" class="">mark@webb-johnson.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="content-isolator__container"><div class="protected-part"><div class="protected-title">Signed PGP part</div><div class="protected-content"><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Good grief, this is not so easy. Now we have:<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><br class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class=""><br class=""></font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">Core 1 register dump:</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">PC : 0x40008044 PS : 0x00060f30 A0 : 0x800fe2cc A1 : 0x3ffcaa90</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">A2 : 0x3f413acc A3 : 0x00000046 A4 : 0x00e6807e A5 : 0x00000000</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">A6 : 0x00000000 A7 : 0x00000000 A8 : 0x00000010 A9 : 0x00e6807e</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">A10 : 0x00000078 A11 : 0x00000009 A12 : 0x3ffcaa3f A13 : 0x00000032</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">A14 : 0x00000000 A15 : 0x3ffcaa48 SAR : 0x00000004 EXCCAUSE: 0x0000001c</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">EXCVADDR: 0x00e6807e LBEG : 0x4008bdad LEND : 0x4008bdd1 LCOUNT : 0x800f93f4</font></blockquote></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class=""><br class="">ELF file SHA256: 74bb0a75eeb4578b<br class=""><br class="">Backtrace: 0x40008044:0x3ffcaa90 0x400fe2c9:0x3ffcab20 0x400fe412:0x3ffcabb0 0x402937b5:0x3ffcabd0</font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class=""><br class=""></font></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><font face="Andale Mono" style="font-style: normal; font-size: 12px;" class="">0x400fe2c9 is in OvmsNetManager::DoSafePrioritiseAndIndicate() (/Users/hq.mark.johnson/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_netmanager.cpp:723).<br class="">718<span class="Apple-tab-span" style="white-space: pre;"> </span> }<br class="">719<br class="">720<span class="Apple-tab-span" style="white-space: pre;"> </span> for (struct netif *pri = netif_list; pri != NULL; pri=pri->next)<br class="">721<span class="Apple-tab-span" style="white-space: pre;"> </span> {<br class="">722<span class="Apple-tab-span" style="white-space: pre;"> </span> ESP_EARLY_LOGI(TAG,"DoSafePrioritiseAndIndicate: interface %p",pri);<br class="">723<span class="Apple-tab-span" style="white-space: pre;"> </span> ESP_EARLY_LOGI(TAG,"DoSafePrioritiseAndIndicate: name %s",pri->name);<br class="">724<span class="Apple-tab-span" style="white-space: pre;"> </span> if ((pri->name[0]==search[0])&&<br class="">725<span class="Apple-tab-span" style="white-space: pre;"> </span> (pri->name[1]==search[1]))<br class="">726<span class="Apple-tab-span" style="white-space: pre;"> </span> {<br class="">727<span class="Apple-tab-span" style="white-space: pre;"> </span> if (search[0] != m_previous_name[0] || search[1] != m_previous_name[1])<br class="">0x400fe412 is in SafePrioritiseAndIndicate(void*) (/Users/hq.mark.johnson/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_netmanager.cpp:676).<br class="">671<span class="Apple-tab-span" style="white-space: pre;"> </span> }<br class="">672<span class="Apple-tab-span" style="white-space: pre;"> </span> }<br class="">673<br class="">674<span class="Apple-tab-span" style="white-space: pre;"> </span>void SafePrioritiseAndIndicate(void* ctx)<br class="">675<span class="Apple-tab-span" style="white-space: pre;"> </span> {<br class="">676<span class="Apple-tab-span" style="white-space: pre;"> </span> MyNetManager.DoSafePrioritiseAndIndicate();<br class="">677<span class="Apple-tab-span" style="white-space: pre;"> </span> }<br class="">678<br class="">679<span class="Apple-tab-span" style="white-space: pre;"> </span>void OvmsNetManager::PrioritiseAndIndicate()<br class="">680<span class="Apple-tab-span" style="white-space: pre;"> </span> {<br class="">0x402937b5 is in tcpip_thread (/Users/hq.mark.johnson/esp/esp-idf/components/lwip/lwip/src/api/tcpip.c:158).<br class="">153<span class="Apple-tab-span" style="white-space: pre;"> </span> break;<br class="">154<span class="Apple-tab-span" style="white-space: pre;"> </span>#endif /* LWIP_TCPIP_TIMEOUT && LWIP_TIMERS */<br class="">155<br class="">156<span class="Apple-tab-span" style="white-space: pre;"> </span> case TCPIP_MSG_CALLBACK:<br class="">157<span class="Apple-tab-span" style="white-space: pre;"> </span> LWIP_DEBUGF(TCPIP_DEBUG, ("tcpip_thread: CALLBACK %p\n", (void *)msg));<br class="">158<span class="Apple-tab-span" style="white-space: pre;"> </span> msg->msg.cb.function(msg->msg.cb.ctx);<br class="">159<span class="Apple-tab-span" style="white-space: pre;"> </span> memp_free(MEMP_TCPIP_MSG_API, msg);<br class="">160<span class="Apple-tab-span" style="white-space: pre;"> </span> break;<br class="">161<br class="">162<span class="Apple-tab-span" style="white-space: pre;"> </span> case TCPIP_MSG_CALLBACK_STATIC:</font></blockquote><div class=""><br class=""></div><div class="">So the issue is most likely corruption of the network interface structure, not thread safe traversal.</div><div class=""><br class=""></div><div class="">I had added some ESP_EARLY_LOGI statements, so can see a little more of what is going on:</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class=""><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (103202) gsm-ppp: Initialising...</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (103212) events: Signal(system.modem.netmode)</span></font></div></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105902) netmanager: DoSafePrioritiseAndIndicate: start</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105902) netmanager: DoSafePrioritiseAndIndicate: connected wifi</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105912) netmanager: DoSafePrioritiseAndIndicate: interface 0x3ffed6a0</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105912) netmanager: DoSafePrioritiseAndIndicate: name pp</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105922) netmanager: DoSafePrioritiseAndIndicate: interface 0x3ffde854</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105932) netmanager: DoSafePrioritiseAndIndicate: name ap</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105932) netmanager: DoSafePrioritiseAndIndicate: interface 0x3ffde640</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105942) netmanager: DoSafePrioritiseAndIndicate: name st</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105952) netmanager: DoSafePrioritiseAndIndicate: end</span></font></div></div><div class=""><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105902) gsm-ppp: StatusCallBack: None</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105902) gsm-ppp: status_cb: Connected</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (105902) gsm-ppp: our_ipaddr = 10.52.40.80</span></font></div></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">…</span></font></div><div class=""><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3708442) cellular: PPP Connection disconnected</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3708442) cellular: PPP Connection disconnected</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709212) netmanager: DoSafePrioritiseAndIndicate: start</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709212) netmanager: DoSafePrioritiseAndIndicate: connected wifi</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709212) netmanager: DoSafePrioritiseAndIndicate: interface 0x3ffed6a0</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709222) netmanager: DoSafePrioritiseAndIndicate: name pp</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709222) netmanager: DoSafePrioritiseAndIndicate: interface 0x30323930</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709232) netmanager: DoSafePrioritiseAndIndicate: name f</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709242) netmanager: DoSafePrioritiseAndIndicate: interface 0x667fc000</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">I (3709252) netmanager: DoSafePrioritiseAndIndicate: name</span></font></div><div class=""><font face="Andale Mono" class=""><span style="font-style: normal; font-size: 12px;" class="">Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1)</span></font></div></div></blockquote><div class=""><div class=""><br class=""></div><div class="">Doesn’t help much, apart from confirm the corruption. Took about an hour to recreate the problem.</div><div class=""><br class=""></div><div class="">I’ll keep looking.</div><div class=""><br class=""></div><div class="">Regards, Mark.</div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 23 Mar 2021, at 4:05 PM, Mark Webb-Johnson <<a href="mailto:mark@webb-johnson.net" class="">mark@webb-johnson.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="content-isolator__container"><div class="protected-part"><div class="protected-title">Signed PGP part</div><div class="protected-content"><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">My attempt didn’t work (still crashes), so I’m now trying your suggestion of wrapping PrioritiseAndIndicate() in a tcpip_callback_with_block. <div class=""><br class=""></div><div class=""> 🤞🏻</div><div class=""><br class=""></div><div class="">Regards, Mark.<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 23 Mar 2021, at 3:02 PM, Michael Balzer <<a href="mailto:dexter@expeedo.de" class="">dexter@expeedo.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="content-isolator__container"><div class="protected-part"><div class="protected-title">Signed PGP part</div><div class="protected-content">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div class="">
Mark,<br class="">
<br class="">
regarding point 2: I've had the same issue with jobs that need to
iterate over the mongoose connection list, I introduced the
netmanager job queue for this to delegate these to the mongoose
context.<br class="">
<br class="">
I remember seeing LwIP has a similar API while browsing the source…
yes, found it: the "tcpip_callback…" functions, e.g.:<br class="">
<br class="">
<font face="monospace" class="">/**<br class="">
* Call a specific function in the thread context of<br class="">
* tcpip_thread for easy access synchronization.<br class="">
* A function called in that way may access lwIP core code<br class="">
* without fearing concurrent access.<br class="">
*<br class="">
* @param function the function to call<br class="">
* @param ctx parameter passed to f<br class="">
* @param block 1 to block until the request is posted, 0 to
non-blocking mode<br class="">
* @return ERR_OK if the function was called, another err_t if not<br class="">
*/<br class="">
err_t<br class="">
tcpip_callback_with_block(tcpip_callback_fn function, void *ctx,
u8_t block)</font><br class="">
<br class="">
So we probably can use this to execute PrioritiseAndIndicate() in
the LwIP context.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 23.03.21 um 06:47 schrieb Mark
Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:CB78F43A-5706-4504-A0B2-F287DC10B40C@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
OK, some progress…
<div class=""><br class="">
</div>
<div class="">
<ol class="MailOutline">
<li class="">I’ve added a check in the OvmsSyncHttpClient code
to refuse to block while running as the netman (mongoose)
task. This will now simply fail the http connection, and log
an error. Not perfect, and not a solution to the core
problem, but at least it avoids a known problem.<br class="">
<br class="">
I’m not sure of the best permanent solution to this. It
seems that we need a callback interface to run commands
asynchronously, and use that in mongoose event handlers.
Adding another mongoose event loop, or using a separate
networking socket with select(), just minimise the problem -
they don’t solve it. The core issue here is blocking during
a mongoose event delivery. That is going to pause all high
level networking.<br class="">
<br class="">
</li>
<li class="">I found a race condition in ovms_netmanager that
seems nasty. The new cellular code could raise duplicate
modem.down signals, picked up and handled in
ovms_netmanager. As part of that it calls
a PrioritiseAndIndicate() function that iterates over the
network interface list (maintained by LWIP). If that network
interface list is modified (eg; removing an interface) while
it is being traversed, nasty crashes can happen. The ‘fix’
I’ve done is again just a workaround to try to reduce the
duplicate signals and hence reduce the likelyhood of the
problem happening, but it won’t fix the core problem (that
is in both master and for-v3.3).<br class="">
<br class="">
There is a netif_find function in LWIP, but (a) that
requires an interface number that we don’t have, and (b)
doesn’t seem to lock the list either.<br class="">
<br class="">
Can’t think of an elegant solution to this, other than
modifications to lwip. We could add our own mutex and use
that whenever we talk to lwip, but even that would miss out
on some modifications to the network interface list, I
guess. </li>
</ol>
<div class=""><br class="">
</div>
<div class="">These two changes are in ‘pre’ now, and I am trying them in
my car.</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 22 Mar 2021, at 6:06 PM, Michael Balzer
<<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="content-isolator__container">
<div class="protected-part">
<div class="protected-title">Signed PGP part</div>
<div class="protected-content">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div class=""> In master, running commands via ssh
or server-v2 block, because these are running
synchronously in the mongoose context.<br class="">
<br class="">
Running commands via web doesn't block, as the
webcommand class starts a separate task for each
execution.<br class="">
<br class="">
The firmware config page does a synchronous call
to MyOTA.GetStatus(), so that call is executed in
the mongoose context. It still works in master,
just needs a second or two to fetch the version
file.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 22.03.21 um 10:38
schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:1ACB6945-97E0-45E7-AA07-4EB5AC42B96F@webb-johnson.net" class="">
<meta http-equiv="content-type" content="text/html; charset=UTF-8" class="">
<div dir="ltr" class="">In master branch, at the
moment, if a command is run from the web shell
(or server v2), surely the mongoose task will
block as the web server / server v2 blocks
waiting for the command to run to completion?</div>
<div dir="ltr" class=""><br class="">
</div>
<div dir="ltr" class="">Doesn’t necessarily need
to be a networking command. Something long
running like the string speed tests.</div>
<div dir="ltr" class=""><br class="">
</div>
<div dir="ltr" class="">In v3.3 I can easily
detect the task wait being requested in the
http library (by seeing if current task id ==
mongoose task), and fail (which I should do
anyway). But I am more concerned with the
general case now (which I think may be wrong
in both master and for-v3.3).</div>
<div dir="ltr" class=""><br class="">
</div>
<div dir="ltr" class="">Regards, Mark</div>
<div dir="ltr" class=""><br class="">
</div>
<div dir="ltr" class="">
<blockquote type="cite" class="">On 22 Mar
2021, at 5:22 PM, Michael Balzer <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><dexter@expeedo.de></a>
wrote:<br class="">
<br class="">
</blockquote>
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
I think we must avoid blocking the Mongoose
task, as that's the central network
dispatcher.<br class="">
<br class="">
Chris had implemented a workaround in one of
his PRs that could allow that to be done
temporarily by running a local Mongoose main
loop during a synchronous operation, but I
still see potential issues from that, as it
wasn't the standard handling as done by the
task, and as it may need to recurse.<br class="">
<br class="">
Maybe the old OvmsHttpClient using socket
I/O is the right way for synchronous network
operations?<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 22.03.21 um
07:15 schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:F88B060F-2CB7-4F77-99AD-08ACD03E95B9@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
Not sure how to resolve this.
<div class=""><br class="">
</div>
<div class="">
<div style="caret-color: rgb(0, 0, 0);" class="">OvmsSyncHttpClient is
currently used in commands from
ovms_plugins and ovms_ota.</div>
</div>
<div class=""><br class="">
</div>
<div class="">I could bring back
the OvmsHttpClient blocking
(non-mongoose) implementation, but I
don’t think that would address the core
problem here:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px;
border: none; padding: 0px;" class="">
<div class="">Inside a mongoose callback
(inside the mongoose networking task),
we are making blocking calls (and in
particular calls that could block for
several tens of seconds).</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">But fundamentally is it ok
to block the mongoose networking task
for extended periods during a mongoose
event callback?</div>
<div class=""><br class="">
</div>
<div class="">Mark</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 21 Mar 2021, at
9:57 PM, Michael Balzer <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="content-isolator__container">
<div class="protected-part">
<div class="protected-title">Signed
PGP part</div>
<div class="protected-content">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div class=""> I've found
opening the web UI
firmware page or calling
"ota status" via ssh to
consistently deadlock the
network on my module.<br class="">
<br class="">
<font class="" face="monospace">I
(130531) webserver: HTTP
GET /cfg/firmware<br class="">
D (130531) http:
OvmsSyncHttpClient:
Connect to <a href="http://ovms.dexters-web.de/" class="" moz-do-not-send="true">ovms.dexters-web.de:80</a><br class="">
D (130541) http:
OvmsSyncHttpClient:
waiting for completion</font><br class="">
<br class="">
After that log message,
the network is dead, and
the netmanager also
doesn't respond:<br class="">
<br class="">
<font class="" face="monospace">OVMS#
network list <br class="">
ERROR: job failed<br class="">
D (183241) netmanager:
send cmd 1 from
0x3ffe7054<br class="">
W (193241) netmanager:
ExecuteJob: cmd 1:
timeout</font><br class="">
<br class="">
The interfaces seem to be
registered and online, but
nothing gets in or out:<br class="">
<br class="">
<font class="" face="monospace">OVMS#
network status <br class="">
Interface#3: pp3 (ifup=1
linkup=1)<br class="">
IPv4:
10.170.195.13/255.255.255.255
gateway 10.64.64.64<br class="">
<br class="">
Interface#2: ap2 (ifup=1
linkup=1)<br class="">
IPv4:
192.168.4.1/255.255.255.0
gateway 192.168.4.1<br class="">
<br class="">
Interface#1: st1 (ifup=1
linkup=1)<br class="">
IPv4:
192.168.2.106/255.255.255.0
gateway 192.168.2.1<br class="">
<br class="">
DNS: 192.168.2.1<br class="">
<br class="">
Default Interface: st1
(192.168.2.106/255.255.255.0
gateway 192.168.2.1)</font><br class="">
<br class="">
<br class="">
A couple of minutes later,
server-v2 recognizes the
stale connection and
issues a network restart,
which fails resulting in
the same behaviour as
shown below with finally
forced reboot by loss of
an important event.<br class="">
<br class="">
Doing "ota status" from
USB works normally, so
this looks like
OvmsSyncHttpClient not
being able to run from
within a mongoose client.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am
18.03.21 um 08:14
schrieb Mark
Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:1FD94B8C-1060-44BF-8E3A-6D3534352100@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
Tried to repeat this,
but not having much
success. Here is my car
module, with network
still up:
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<div class="">
<div class="">OVMS#
boot status</div>
<div class="">Last
boot was 262355
second(s) ago</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">I did
manage to catch one
network related
crash after
repeatedly
disconnecting and
reconnecting the
cellular antenna.
This was:</div>
<div class=""><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-style:
normal;
font-size:
12px;" class="">I
(3717989)
cellular: PPP
Connection
disconnected</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-style:
normal;
font-size:
12px;" class="">Guru
Meditation
Error: Core 1
panic'ed
(LoadProhibited).
Exception was
unhandled.</span></font></div>
</div>
<div class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class=""><br class="">
</span></font></div>
<div class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">0x400fe082 is in
OvmsNetManager::PrioritiseAndIndicate()
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_netmanager.cpp:707).</span></font></div>
</blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-style:
normal;
font-size:
12px;" class="">707
if
((pri->name[0]==search[0])&&</span></font></div>
</blockquote>
<font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x400ed360
is in
OvmsMetricString::SetValue(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_metrics.cpp:1358).</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">1357 void
OvmsMetricString::SetValue(std::string
value)</span></font></blockquote>
<font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x4008bdad
is at
../../../../.././newlib/libc/machine/xtensa/strcmp.S:586.</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x4008bdd1
is at
../../../../.././newlib/libc/machine/xtensa/strcmp.S:604.</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x400fe886
is in
OvmsNetManager::ModemDown(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, void*)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_netmanager.cpp:522).</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">522
PrioritiseAndIndicate();</span></font></blockquote>
</blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x400fd752
is in
std::_Function_handler<void
(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char>
>, void*),
std::_Bind<std::_Mem_fn<void
(OvmsNetManager::*)(std::__cxx11::basic_string<char,
std::char_traits<char>,
std::allocator<char> >, void*)> (OvmsNetManager*,
std::_Placeholder<1>,
std::_Placeholder<2>)> >::_M_invoke(std::_Any_data
const&,
std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&&,
void*&&)
(/home/openvehicles/build/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/functional:600).</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">600 {
return
(__object->*_M_pmf)(std::forward<_Args>(__args)...); }</span></font></blockquote>
</blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x400f512e
is in
std::function<void
(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char>
>,
void*)>::operator()(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, void*)
const
(/home/openvehicles/build/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/functional:2271).</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">2271
return
_M_invoker(_M_functor,
std::forward<_ArgTypes>(__args)...);</span></font></blockquote>
</blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x400f52f1
is in
OvmsEvents::HandleQueueSignalEvent(event_queue_t*)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_events.cpp:283).</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">283
m_current_callback->m_callback(m_current_event,
msg->body.signal.data);</span></font></blockquote>
</blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x400f53d8
is in
OvmsEvents::EventTask()
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_events.cpp:237).</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">237
HandleQueueSignalEvent(&msg);</span></font></blockquote>
</blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class=""><br class="">
</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style:
normal; font-size:
12px;" class="">0x400f545d
is in
EventLaunchTask(void*)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_events.cpp:80).</span></font></blockquote>
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class="">
<blockquote style="margin: 0 0 0
40px; border: none;
padding: 0px;" class=""><font class="" face="Andale Mono"><span style="font-style: normal; font-size: 12px;" class="">80
me->EventTask();</span></font></blockquote>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">My
for_v3.3 branch does
include the
preliminary changes
to support the wifi
at 20MHz bandwidth,
and perhaps those
could be affecting
things. I do notice
that if I ‘power
wifi off’, then
‘wifi mode client’,
it can connect to
the station, but not
get an IP address.
I’ve just tried to
merge in the latest
fixes to that, and
rebuilt a release. I
will continue to
test with that.</div>
<div class=""><br class="">
</div>
<div class="">Regards,
Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On
12 Mar 2021, at
10:32 PM,
Michael Balzer
<<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<div class="content-isolator__container">
<div class="protected-part">
<div class="protected-title">Signed
PGP part</div>
<div class="protected-content">I
just tried
switching to
for-v3.3 in my
car module
after tests on
my desk module
were OK, and
I've run into
the very same
problem with
for-v3.3. So
the issue
isn't related
to esp-idf.<br class="">
<br class="">
The network
only
occasionally
starts
normally, but
even then all
connectivity
is lost after
a couple of
minutes.<br class="">
<br class="">
The stale
connection
watchdog in
server-v2
triggers a
network
restart, but
that also
doesn't seem
to succeed:<br class="">
<br class="">
2021-03-12
14:53:01.802
CET W (981652)
ovms-server-v2: Detected stale connection (issue #241), restarting
network<br class="">
2021-03-12
14:53:01.802
CET I (981652)
esp32wifi:
Restart<br class="">
2021-03-12
14:53:01.802
CET I (981652)
esp32wifi:
Stopping WIFI
station<br class="">
2021-03-12
14:53:01.812
CET I (981662)
wifi:state:
run -> init
(0)<br class="">
2021-03-12
14:53:01.812
CET I (981662)
wifi:pm stop,
total sleep
time:
831205045 us /
975329961 us<br class="">
<br class="">
2021-03-12
14:53:01.812
CET I (981662)
wifi:new:<1,0>, old:<1,1>, ap:<1,1>, sta:<1,1>,
prof:1<br class="">
2021-03-12
14:53:01.832
CET I (981682)
wifi:flush txq<br class="">
2021-03-12
14:53:01.842
CET I (981692)
wifi:stop sw
txq<br class="">
2021-03-12
14:53:01.842
CET I (981692)
wifi:lmac stop
hw txq<br class="">
2021-03-12
14:53:01.852
CET I (981702)
esp32wifi:
Powering down
WIFI driver<br class="">
2021-03-12
14:53:01.852
CET I (981702)
wifi:Deinit
lldesc rx
mblock:16<br class="">
2021-03-12
14:53:01.862
CET I (981712)
esp32wifi:
Powering up
WIFI driver<br class="">
2021-03-12
14:53:01.862
CET I (981712)
wifi:nvs_log_init, erase log key successfully, reinit nvs log<br class="">
2021-03-12
14:53:01.882
CET I (981732)
wifi:wifi
driver task:
3ffd4d84,
prio:23,
stack:3584,
core=0<br class="">
2021-03-12
14:53:01.882
CET I (981732)
system_api:
Base MAC
address is not
set, read
default base
MAC address
from BLK0 of
EFUSE<br class="">
2021-03-12
14:53:01.882
CET I (981732)
system_api:
Base MAC
address is not
set, read
default base
MAC address
from BLK0 of
EFUSE<br class="">
2021-03-12
14:53:01.902
CET I (981752)
wifi:wifi
firmware
version:
30f9e79<br class="">
2021-03-12
14:53:01.912
CET I (981762)
wifi:config
NVS flash:
enabled<br class="">
2021-03-12
14:53:01.912
CET I (981762)
wifi:config
nano
formating:
disabled<br class="">
2021-03-12
14:53:01.912
CET I (981762)
wifi:Init data
frame dynamic
rx buffer num:
16<br class="">
2021-03-12
14:53:01.912
CET I (981762)
wifi:Init
management
frame dynamic
rx buffer num:
16<br class="">
2021-03-12
14:53:01.922
CET I (981772)
wifi:Init
management
short buffer
num: 32<br class="">
2021-03-12
14:53:01.922
CET I (981772)
wifi:Init
dynamic tx
buffer num: 16<br class="">
2021-03-12
14:53:01.922
CET I (981772)
wifi:Init
static rx
buffer size:
2212<br class="">
2021-03-12
14:53:01.922
CET I (981772)
wifi:Init
static rx
buffer num: 16<br class="">
2021-03-12
14:53:01.922
CET I (981772)
wifi:Init
dynamic rx
buffer num: 16<br class="">
2021-03-12
14:53:02.642
CET I (982492)
wifi:mode :
sta
(30:ae:a4:5f:e7:ec)
+ softAP
(30:ae:a4:5f:e7:ed)<br class="">
2021-03-12
14:53:02.652
CET I (982502)
wifi:Total
power save
buffer number:
8<br class="">
2021-03-12
14:53:02.652
CET I (982502)
cellular-modem-auto: Restart<br class="">
2021-03-12
14:53:02.662
CET I (982512)
cellular:
State: Enter
PowerOffOn
state<br class="">
2021-03-12
14:53:02.662
CET I (982512)
gsm-ppp:
Shutting down
(hard)...<br class="">
2021-03-12
14:53:02.662
CET I (982512)
gsm-ppp:
StatusCallBack:
User Interrupt<br class="">
2021-03-12
14:53:02.662
CET I (982512)
gsm-ppp: PPP
connection has
been closed<br class="">
2021-03-12
14:53:02.662
CET I (982512)
gsm-ppp: PPP
is shutdown<br class="">
2021-03-12
14:53:02.662
CET I (982512)
gsm-ppp:
Shutting down
(hard)...<br class="">
2021-03-12
14:53:02.672
CET I (982522)
gsm-ppp:
StatusCallBack:
User Interrupt<br class="">
2021-03-12
14:53:02.672
CET I (982522)
gsm-ppp: PPP
connection has
been closed<br class="">
2021-03-12
14:53:02.672
CET I (982522)
gsm-ppp: PPP
is shutdown<br class="">
2021-03-12
14:53:02.672
CET I (982522)
gsm-nmea:
Shutdown
(direct)<br class="">
2021-03-12
14:53:02.672
CET I (982522)
cellular-modem-auto: Power Cycle<br class="">
2021-03-12
14:53:04.682
CET D (984532)
events:
Signal(system.wifi.down)<br class="">
2021-03-12
14:53:04.682
CET I (984532)
netmanager:
WIFI client
stop<br class="">
2021-03-12
14:53:04.682
CET E (984532)
netmanager:
Inconsistent
state: no
interface of
type 'pp'
found<br class="">
2021-03-12
14:53:04.682
CET I (984532)
netmanager:
WIFI client
down (with
MODEM up):
reconfigured
for MODEM
priority<br class="">
2021-03-12
14:53:04.692
CET D (984542)
events:
Signal(system.event)<br class="">
2021-03-12
14:53:04.692
CET D (984542)
events:
Signal(system.wifi.sta.disconnected)<br class="">
2021-03-12
14:53:04.692
CET E (984542)
netmanager:
Inconsistent
state: no
interface of
type 'pp'
found<br class="">
2021-03-12
14:53:04.692
CET I (984542)
esp32wifi: STA
disconnected
with reason 8
= ASSOC_LEAVE<br class="">
2021-03-12
14:53:04.702
CET D (984552)
events:
Signal(system.event)<br class="">
2021-03-12
14:53:04.702
CET D (984552)
events:
Signal(system.wifi.sta.stop)<br class="">
2021-03-12
14:53:04.702
CET E (984552)
netmanager:
Inconsistent
state: no
interface of
type 'pp'
found<br class="">
2021-03-12
14:53:04.712
CET D (984562)
events:
Signal(system.event)<br class="">
2021-03-12
14:53:04.712
CET D (984562)
events:
Signal(system.wifi.ap.stop)<br class="">
2021-03-12
14:53:04.712
CET E (984562)
netmanager:
Inconsistent
state: no
interface of
type 'pp'
found<br class="">
2021-03-12
14:53:04.712
CET I (984562)
netmanager:
WIFI access
point is down<br class="">
2021-03-12
14:53:04.712
CET I (984562)
esp32wifi: AP
stopped<br class="">
2021-03-12
14:53:04.722
CET D (984572)
events:
Signal(network.wifi.sta.bad)<br class="">
2021-03-12
14:53:04.722
CET D (984572)
events:
Signal(system.event)<br class="">
2021-03-12
14:53:04.722
CET D (984572)
events:
Signal(system.wifi.sta.start)<br class="">
2021-03-12
14:53:04.732
CET D (984582)
events:
Signal(system.modem.down)<br class="">
2021-03-12
14:53:04.742
CET I (984592)
netmanager:
MODEM down
(with WIFI
client down):
network
connectivity
has been lost<br class="">
2021-03-12
14:53:04.742
CET D (984592)
events:
Signal(system.modem.down)<br class="">
2021-03-12
14:53:04.752
CET D (984602)
events:
Signal(system.event)<br class="">
2021-03-12
14:53:04.752
CET D (984602)
events:
Signal(system.wifi.ap.start)<br class="">
2021-03-12
14:53:04.752
CET I (984602)
netmanager:
WIFI access
point is up<br class="">
2021-03-12
14:53:26.802
CET E
(1006652)
events:
SignalEvent:
queue overflow
(running
system.wifi.ap.start->netmanager
for 23 sec),
event
'ticker.1'
dropped<br class="">
2021-03-12
14:53:27.802
CET E
(1007652)
events:
SignalEvent:
queue overflow
(running
system.wifi.ap.start->netmanager
for 24 sec),
event
'ticker.1'
dropped<br class="">
2021-03-12
14:53:28.802
CET E
(1008652)
events:
SignalEvent:
queue overflow
(running
system.wifi.ap.start->netmanager
for 25 sec),
event
'ticker.1'
dropped<br class="">
…and so on
until<br class="">
2021-03-12
14:54:01.802
CET E
(1041652)
events:
SignalEvent:
lost important
event =>
aborting<br class="">
<br class="">
<br class="">
I need my car
now, so will
switch back to
master for
now.<br class="">
<br class="">
Mark, if
you've got
specific debug
logs I should
fetch on the
next try, tell
me.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 12.03.21 um
05:47 schrieb
Craig Leres:<br class="">
<blockquote type="cite" class="">I
just updated
to
3.2.016-68-g8e10c6b7
and still get
the network
hang
immediately
after booting
and logging
into the web
gui.<br class="">
<br class="">
But I see now
my problem is
likely that
I'm not using
the right
esp-idf (duh).
Is there a way
I can have
master build
using
~/esp/esp-idf
and have
for-v3.3 use a
different
path?)<br class="">
<br class="">
Craig<br class="">
_______________________________________________<br class="">
OvmsDev
mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
<a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</blockquote>
<br class="">
-- <br class="">
Michael Balzer
* Helkenberger
Weg 9 *
D-58256
Ennepetal<br class="">
Fon 02333 /
833 5735 *
Handy 0176 /
206 989 26<br class="">
<br class="">
<br class="">
</div>
</div>
<br class="">
<iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div>
</div>
</div>
<br class="">
<iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
<span class="">_______________________________________________</span><br class="">
<span class="">OvmsDev mailing list</span><br class="">
<span class=""><a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a></span><br class="">
<span class=""><a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a></span><br class="">
</div>
</blockquote>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div>
</div>
</div>
<br class="">
<iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div>
</div></div><br class=""><iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" width="200" height="10" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg=="></iframe></div></div></blockquote></div><br class=""></div></div></div></div><br class=""><iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" width="200" height="10" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg=="></iframe></div></div></blockquote></div><br class=""></div></div></div></div><br class=""><iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" width="200" height="10" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg=="></iframe></div></div></blockquote></div><br class=""></div></div></body></html>