<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thanks for the feedback.<div class=""><br class=""></div><div class="">I should have a final sample of v3.3 hardware with rev3 esp32 in my hands towards the end of this month (this is the same sample set that goes to be destroyed by the certification labs). Is there any way of triggering the bug earlier, for replication? Like a stress test or something? Or just have to wait.</div><div class=""><br class=""></div><div class="">Regards, Mark.<br class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 17 Sep 2021, at 3:24 PM, Michael Balzer <<a href="mailto:dexter@expeedo.de" class="">dexter@expeedo.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="content-isolator__container"><div class="protected-part"><div class="protected-title">Signed PGP part</div><div class="protected-content">
  
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
  
  <div class="">
    Mark,<br class="">
    <br class="">
    I've been running the new for-v3.3 version this week on both of my
    modules without ppp issues.<br class="">
    <br class="">
    Duktape still occasionally runs into the null/undefined issue with
    for…in:<br class="">
    <br class="">
<a class="moz-txt-link-freetext" href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/474#issuecomment-744005044">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/474#issuecomment-744005044</a><br class="">
    <br class="">
    for…in normally doesn't throw an error even if you run over null or
    undefined.<br class="">
    <br class="">
    I think both could still be the SPIRAM bug, now probably only
    occurring with very specific conditions. We build with LWIP using
    SPIRAM as well, so the PPP instance is allocated from SPIRAM also.
    Reallocating the instance on each new connect implies a higher
    chance of triggering the problem if it's address specific. The
    Duktape stack object addresses vary continuously with the running
    event handlers and user interactions, so that also has a high chance
    of occasionally triggering an address specific bug.<br class="">
    <br class="">
    We need to test the revision 3 ESP32 on this.<br class="">
    <br class="">
    Regards,<br class="">
    Michael<br class="">
    <br class="">
    <br class="">
    <div class="moz-cite-prefix">Am 09.09.21 um 02:31 schrieb Mark
      Webb-Johnson:<br class="">
    </div>
    <blockquote type="cite" cite="mid:20735B00-13FF-4F16-B319-2CA6CFE8E879@webb-johnson.net" class="">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
      Reviving this old topic still impacting us…
      <div class=""><br class="">
      </div>
      <div class="">Comparing the master vs for-v3.3 branches, the only
        thing that changed related to ppp is that in 3.3 the ppp object
        is dynamically created and destroyed, while in master it is
        statically allocated at boot. The ppp code is so simple, and I
        can’t see how that could be a problem. Perhaps related to
        position in memory, and some other memory corruption?</div>
      <div class=""><br class="">
      </div>
      <div class="">Anyway, I changed it to not destroy the ppp object
        when the gsm connection is lost, but merely shutdown the ppp
        (which is what master branch does). While it is still
        dynamically allocated, it is no longer as dynamic (being created
        just once at startup of the cellular system). I’ve never managed
        to reliably repeat this problem in my environment, but I think
        this should help.It has been running on my desktop test unit for
        the past four days without issue.</div>
      <div class=""><br class="">
      </div>
      <div class="">That code is committed now. I would appreciate it if
        others who saw this problem could try again with this latest
        build of the for-v3.3 branch.<br class="">
        <div class=""><br class="">
        </div>
        <div class="">Regards, Mark.<br class="">
          <div class=""><br class="">
            <blockquote type="cite" class="">
              <div class="">On 24 Mar 2021, at 3:53 PM, Mark
                Webb-Johnson <<a href="mailto:mark@webb-johnson.net" class="" moz-do-not-send="true">mark@webb-johnson.net</a>>
                wrote:</div>
              <br class="Apple-interchange-newline">
              <div class="">
                <div class="content-isolator__container">
                  <div class="protected-part">
                    <div class="protected-title">Signed PGP part</div>
                    <div class="protected-content">
                      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                      <div style="word-wrap: break-word;
                        -webkit-nbsp-mode: space; line-break:
                        after-white-space;" class="">Good grief, this is
                        not so easy. Now we have:
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class=""><br class="">
                          <font style="font-style: normal; font-size:
                            12px;" class="" face="Andale Mono">Guru
                            Meditation Error: Core  1 panic'ed
                            (LoadProhibited). Exception was unhandled.</font></blockquote>
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class=""><font style="font-style: normal; font-size: 12px;" class="" face="Andale Mono"><br class="">
                          </font></blockquote>
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class=""><font style="font-style: normal; font-size: 12px;" class="" face="Andale Mono">Core 1 register
                            dump:</font></blockquote>
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class="">
                          <blockquote style="margin: 0 0 0 40px; border:
                            none; padding: 0px;" class=""><font style="font-style: normal; font-size:
                              12px;" class="" face="Andale Mono">PC    
                               : 0x40008044  PS      : 0x00060f30  A0  
                                 : 0x800fe2cc  A1      : 0x3ffcaa90</font></blockquote>
                          <blockquote style="margin: 0 0 0 40px; border:
                            none; padding: 0px;" class=""><font style="font-style: normal; font-size:
                              12px;" class="" face="Andale Mono">A2    
                               : 0x3f413acc  A3      : 0x00000046  A4  
                                 : 0x00e6807e  A5      : 0x00000000</font></blockquote>
                          <blockquote style="margin: 0 0 0 40px; border:
                            none; padding: 0px;" class=""><font style="font-style: normal; font-size:
                              12px;" class="" face="Andale Mono">A6    
                               : 0x00000000  A7      : 0x00000000  A8  
                                 : 0x00000010  A9      : 0x00e6807e</font></blockquote>
                          <blockquote style="margin: 0 0 0 40px; border:
                            none; padding: 0px;" class=""><font style="font-style: normal; font-size:
                              12px;" class="" face="Andale Mono">A10    
                              : 0x00000078  A11     : 0x00000009  A12  
                                : 0x3ffcaa3f  A13     : 0x00000032</font></blockquote>
                          <blockquote style="margin: 0 0 0 40px; border:
                            none; padding: 0px;" class=""><font style="font-style: normal; font-size:
                              12px;" class="" face="Andale Mono">A14    
                              : 0x00000000  A15     : 0x3ffcaa48  SAR  
                                : 0x00000004  EXCCAUSE: 0x0000001c</font></blockquote>
                          <blockquote style="margin: 0 0 0 40px; border:
                            none; padding: 0px;" class=""><font style="font-style: normal; font-size:
                              12px;" class="" face="Andale Mono">EXCVADDR:
                              0x00e6807e  LBEG    : 0x4008bdad  LEND  
                               : 0x4008bdd1  LCOUNT  : 0x800f93f4</font></blockquote>
                        </blockquote>
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class=""><font style="font-style: normal; font-size: 12px;" class="" face="Andale Mono"><br class="">
                            ELF file SHA256: 74bb0a75eeb4578b<br class="">
                            <br class="">
                            Backtrace: 0x40008044:0x3ffcaa90
                            0x400fe2c9:0x3ffcab20 0x400fe412:0x3ffcabb0
                            0x402937b5:0x3ffcabd0</font></blockquote>
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class=""><font style="font-style: normal; font-size: 12px;" class="" face="Andale Mono"><br class="">
                          </font></blockquote>
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class=""><font style="font-style: normal; font-size: 12px;" class="" face="Andale Mono">0x400fe2c9 is in
OvmsNetManager::DoSafePrioritiseAndIndicate()
(/Users/hq.mark.johnson/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_netmanager.cpp:723).<br class="">
                            718<span class="Apple-tab-span" style="white-space: pre;">        </span>
                               }<br class="">
                            719<br class="">
                            720<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             for (struct netif *pri = netif_list; pri !=
                            NULL; pri=pri->next)<br class="">
                            721<span class="Apple-tab-span" style="white-space: pre;">        </span>
                               {<br class="">
                            722<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             
                             ESP_EARLY_LOGI(TAG,"DoSafePrioritiseAndIndicate:
                            interface %p",pri);<br class="">
                            723<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             
                             ESP_EARLY_LOGI(TAG,"DoSafePrioritiseAndIndicate:
                              name    %s",pri->name);<br class="">
                            724<span class="Apple-tab-span" style="white-space: pre;">        </span>
                               if
                            ((pri->name[0]==search[0])&&<br class="">
                            725<span class="Apple-tab-span" style="white-space: pre;">        </span>
                                   (pri->name[1]==search[1]))<br class="">
                            726<span class="Apple-tab-span" style="white-space: pre;">        </span>
                                 {<br class="">
                            727<span class="Apple-tab-span" style="white-space: pre;">        </span>
                                 if (search[0] != m_previous_name[0] ||
                            search[1] != m_previous_name[1])<br class="">
                            0x400fe412 is in
                            SafePrioritiseAndIndicate(void*)
(/Users/hq.mark.johnson/Documents/ovms/Open-Vehicle-Monitoring-System-3/vehicle/OVMS.V3/main/ovms_netmanager.cpp:676).<br class="">
                            671<span class="Apple-tab-span" style="white-space: pre;">        </span>
                               }<br class="">
                            672<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             }<br class="">
                            673<br class="">
                            674<span class="Apple-tab-span" style="white-space: pre;">        </span>void
                            SafePrioritiseAndIndicate(void* ctx)<br class="">
                            675<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             {<br class="">
                            676<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             MyNetManager.DoSafePrioritiseAndIndicate();<br class="">
                            677<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             }<br class="">
                            678<br class="">
                            679<span class="Apple-tab-span" style="white-space: pre;">        </span>void
                            OvmsNetManager::PrioritiseAndIndicate()<br class="">
                            680<span class="Apple-tab-span" style="white-space: pre;">        </span>
                             {<br class="">
                            0x402937b5 is in tcpip_thread
(/Users/hq.mark.johnson/esp/esp-idf/components/lwip/lwip/src/api/tcpip.c:158).<br class="">
                            153<span class="Apple-tab-span" style="white-space: pre;">        </span>
                                 break;<br class="">
                            154<span class="Apple-tab-span" style="white-space: pre;">        </span>#endif
                            /* LWIP_TCPIP_TIMEOUT && LWIP_TIMERS
                            */<br class="">
                            155<br class="">
                            156<span class="Apple-tab-span" style="white-space: pre;">        </span>
                               case TCPIP_MSG_CALLBACK:<br class="">
                            157<span class="Apple-tab-span" style="white-space: pre;">        </span>
                                 LWIP_DEBUGF(TCPIP_DEBUG,
                            ("tcpip_thread: CALLBACK %p\n", (void
                            *)msg));<br class="">
                            158<span class="Apple-tab-span" style="white-space: pre;">        </span>
                               
                             msg->msg.cb.function(msg->msg.cb.ctx);<br class="">
                            159<span class="Apple-tab-span" style="white-space: pre;">        </span>
                                 memp_free(MEMP_TCPIP_MSG_API, msg);<br class="">
                            160<span class="Apple-tab-span" style="white-space: pre;">        </span>
                                 break;<br class="">
                            161<br class="">
                            162<span class="Apple-tab-span" style="white-space: pre;">        </span>
                               case TCPIP_MSG_CALLBACK_STATIC:</font></blockquote>
                        <div class=""><br class="">
                        </div>
                        <div class="">So the issue is most likely
                          corruption of the network interface structure,
                          not thread safe traversal.</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">I had added some ESP_EARLY_LOGI
                          statements, so can see a little more of what
                          is going on:</div>
                        <div class=""><br class="">
                        </div>
                        <blockquote style="margin: 0 0 0 40px; border:
                          none; padding: 0px;" class="">
                          <div class="">
                            <div class="">
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 12px;" class="">I
                                    (103202) gsm-ppp: Initialising...</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 12px;" class="">I
                                    (103212) events:
                                    Signal(system.modem.netmode)</span></font></div>
                            </div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105902)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: start</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105902)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: connected
                                  wifi</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105912)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: interface
                                  0x3ffed6a0</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105912)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate:   name  
                                   pp</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105922)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: interface
                                  0x3ffde854</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105932)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate:   name  
                                   ap</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105932)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: interface
                                  0x3ffde640</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105942)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate:   name  
                                   st</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105952)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: end</span></font></div>
                          </div>
                          <div class="">
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105902)
                                  gsm-ppp: StatusCallBack: None</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105902)
                                  gsm-ppp: status_cb: Connected</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (105902)
                                  gsm-ppp:    our_ipaddr  = 10.52.40.80</span></font></div>
                          </div>
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-style: normal;
                                font-size: 12px;" class="">…</span></font></div>
                          <div class="">
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3708442)
                                  cellular: PPP Connection disconnected</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3708442)
                                  cellular: PPP Connection disconnected</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709212)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: start</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709212)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: connected
                                  wifi</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709212)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: interface
                                  0x3ffed6a0</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709222)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate:   name  
                                   pp</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709222)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: interface
                                  0x30323930</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709232)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate:   name  
                                   f</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709242)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate: interface
                                  0x667fc000</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">I (3709252)
                                  netmanager:
                                  DoSafePrioritiseAndIndicate:   name</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 12px;" class="">Guru
                                  Meditation Error: Core  1 panic'ed
                                  (Interrupt wdt timeout on CPU1)</span></font></div>
                          </div>
                        </blockquote>
                        <div class="">
                          <div class=""><br class="">
                          </div>
                          <div class="">Doesn’t help much, apart from
                            confirm the corruption. Took about an hour
                            to recreate the problem.</div>
                          <div class=""><br class="">
                          </div>
                          <div class="">I’ll keep looking.</div>
                          <div class=""><br class="">
                          </div>
                          <div class="">Regards, Mark.</div>
                          <div class=""><br class="">
                            <blockquote type="cite" class="">
                              <div class="">On 23 Mar 2021, at 4:05 PM,
                                Mark Webb-Johnson <<a href="mailto:mark@webb-johnson.net" class="" moz-do-not-send="true">mark@webb-johnson.net</a>>
                                wrote:</div>
                              <br class="Apple-interchange-newline">
                              <div class="">
                                <div class="content-isolator__container">
                                  <div class="protected-part">
                                    <div class="protected-title">Signed
                                      PGP part</div>
                                    <div class="protected-content">
                                      <meta http-equiv="Content-Type" content="text/html;
                                        charset=UTF-8" class="">
                                      <div style="word-wrap: break-word;
                                        -webkit-nbsp-mode: space;
                                        line-break: after-white-space;" class="">My attempt didn’t work
                                        (still crashes), so I’m now
                                        trying your suggestion of
                                        wrapping PrioritiseAndIndicate()
                                        in a tcpip_callback_with_block. 
                                        <div class=""><br class="">
                                        </div>
                                        <div class=""> 🤞🏻</div>
                                        <div class=""><br class="">
                                        </div>
                                        <div class="">Regards, Mark.<br class="">
                                          <div class=""><br class="">
                                            <blockquote type="cite" class="">
                                              <div class="">On 23 Mar
                                                2021, at 3:02 PM,
                                                Michael Balzer <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
                                                wrote:</div>
                                              <br class="Apple-interchange-newline">
                                              <div class="">
                                                <div class="content-isolator__container">
                                                  <div class="protected-part">
                                                    <div class="protected-title">Signed
                                                      PGP part</div>
                                                    <div class="protected-content">
                                                      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                                                      <div class="">
                                                        Mark,<br class="">
                                                        <br class="">
                                                        regarding point
                                                        2: I've had the
                                                        same issue with
                                                        jobs that need
                                                        to iterate over
                                                        the mongoose
                                                        connection list,
                                                        I introduced the
                                                        netmanager job
                                                        queue for this
                                                        to delegate
                                                        these to the
                                                        mongoose
                                                        context.<br class="">
                                                        <br class="">
                                                        I remember
                                                        seeing LwIP has
                                                        a similar API
                                                        while browsing
                                                        the source… yes,
                                                        found it: the
                                                        "tcpip_callback…"
                                                        functions, e.g.:<br class="">
                                                        <br class="">
                                                        <font class="" face="monospace">/**<br class="">
                                                           * Call a
                                                          specific
                                                          function in
                                                          the thread
                                                          context of<br class="">
                                                           *
                                                          tcpip_thread
                                                          for easy
                                                          access
                                                          synchronization.<br class="">
                                                           * A function
                                                          called in that
                                                          way may access
                                                          lwIP core code<br class="">
                                                           * without
                                                          fearing
                                                          concurrent
                                                          access.<br class="">
                                                           *<br class="">
                                                           * @param
                                                          function the
                                                          function to
                                                          call<br class="">
                                                           * @param ctx
                                                          parameter
                                                          passed to f<br class="">
                                                           * @param
                                                          block 1 to
                                                          block until
                                                          the request is
                                                          posted, 0 to
                                                          non-blocking
                                                          mode<br class="">
                                                           * @return
                                                          ERR_OK if the
                                                          function was
                                                          called,
                                                          another err_t
                                                          if not<br class="">
                                                           */<br class="">
                                                          err_t<br class="">
tcpip_callback_with_block(tcpip_callback_fn function, void *ctx, u8_t
                                                          block)</font><br class="">
                                                        <br class="">
                                                        So we probably
                                                        can use this to
                                                        execute
                                                        PrioritiseAndIndicate()
                                                        in the LwIP
                                                        context.<br class="">
                                                        <br class="">
                                                        Regards,<br class="">
                                                        Michael<br class="">
                                                        <br class="">
                                                        <br class="">
                                                        <div class="moz-cite-prefix">Am
                                                          23.03.21 um
                                                          06:47 schrieb
                                                          Mark
                                                          Webb-Johnson:<br class="">
                                                        </div>
                                                        <blockquote type="cite" cite="mid:CB78F43A-5706-4504-A0B2-F287DC10B40C@webb-johnson.net" class="">
                                                          <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                                                          OK, some
                                                          progress…
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">
                                                          <ol class="MailOutline">
                                                          <li class="">I’ve
                                                          added a check
                                                          in
                                                          the OvmsSyncHttpClient
                                                          code to refuse
                                                          to block while
                                                          running as the
                                                          netman
                                                          (mongoose)
                                                          task. This
                                                          will now
                                                          simply fail
                                                          the http
                                                          connection,
                                                          and log an
                                                          error. Not
                                                          perfect, and
                                                          not a solution
                                                          to the core
                                                          problem, but
                                                          at least it
                                                          avoids a known
                                                          problem.<br class="">
                                                          <br class="">
                                                          I’m not sure
                                                          of the best
                                                          permanent
                                                          solution to
                                                          this. It seems
                                                          that we need a
                                                          callback
                                                          interface to
                                                          run commands
                                                          asynchronously,
                                                          and use that
                                                          in mongoose
                                                          event
                                                          handlers.
                                                          Adding another
                                                          mongoose event
                                                          loop, or using
                                                          a separate
                                                          networking
                                                          socket with
                                                          select(), just
                                                          minimise the
                                                          problem - they
                                                          don’t solve
                                                          it. The core
                                                          issue here is
                                                          blocking
                                                          during a
                                                          mongoose event
                                                          delivery. That
                                                          is going to
                                                          pause all high
                                                          level
                                                          networking.<br class="">
                                                          <br class="">
                                                          </li>
                                                          <li class="">I
                                                          found a race
                                                          condition in
                                                          ovms_netmanager
                                                          that seems
                                                          nasty. The new
                                                          cellular code
                                                          could raise
                                                          duplicate
                                                          modem.down
                                                          signals,
                                                          picked up and
                                                          handled in
                                                          ovms_netmanager.
                                                          As part of
                                                          that it calls
a PrioritiseAndIndicate() function that iterates over the network
                                                          interface list
                                                          (maintained by
                                                          LWIP). If that
                                                          network
                                                          interface list
                                                          is modified
                                                          (eg; removing
                                                          an interface)
                                                          while it is
                                                          being
                                                          traversed,
                                                          nasty crashes
                                                          can happen.
                                                          The ‘fix’ I’ve
                                                          done is again
                                                          just a
                                                          workaround to
                                                          try to reduce
                                                          the duplicate
                                                          signals and
                                                          hence reduce
                                                          the likelyhood
                                                          of the problem
                                                          happening, but
                                                          it won’t fix
                                                          the core
                                                          problem (that
                                                          is in both
                                                          master and
                                                          for-v3.3).<br class="">
                                                          <br class="">
                                                          There is a
                                                          netif_find
                                                          function in
                                                          LWIP, but (a)
                                                          that requires
                                                          an interface
                                                          number that we
                                                          don’t have,
                                                          and (b)
                                                          doesn’t seem
                                                          to lock the
                                                          list either.<br class="">
                                                          <br class="">
                                                          Can’t think of
                                                          an elegant
                                                          solution to
                                                          this, other
                                                          than
                                                          modifications
                                                          to lwip. We
                                                          could add our
                                                          own mutex and
                                                          use that
                                                          whenever we
                                                          talk to lwip,
                                                          but even that
                                                          would miss out
                                                          on some
                                                          modifications
                                                          to the network
                                                          interface
                                                          list, I
                                                          guess. </li>
                                                          </ol>
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">These
                                                          two changes
                                                          are in ‘pre’
                                                          now, and I am
                                                          trying them in
                                                          my car.</div>
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">Regards,
                                                          Mark.</div>
                                                          <div class=""><br class="">
                                                          <blockquote type="cite" class="">
                                                          <div class="">On
                                                          22 Mar 2021,
                                                          at 6:06 PM,
                                                          Michael Balzer
                                                          <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
                                                          wrote:</div>
                                                          <br class="Apple-interchange-newline">
                                                          <div class="">
                                                          <div class="content-isolator__container">
                                                          <div class="protected-part">
                                                          <div class="protected-title">Signed
                                                          PGP part</div>
                                                          <div class="protected-content">
                                                          <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                                                          <div class="">
                                                          In master,
                                                          running
                                                          commands via
                                                          ssh or
                                                          server-v2
                                                          block, because
                                                          these are
                                                          running
                                                          synchronously
                                                          in the
                                                          mongoose
                                                          context.<br class="">
                                                          <br class="">
                                                          Running
                                                          commands via
                                                          web doesn't
                                                          block, as the
                                                          webcommand
                                                          class starts a
                                                          separate task
                                                          for each
                                                          execution.<br class="">
                                                          <br class="">
                                                          The firmware
                                                          config page
                                                          does a
                                                          synchronous
                                                          call to
                                                          MyOTA.GetStatus(),
                                                          so that call
                                                          is executed in
                                                          the mongoose
                                                          context. It
                                                          still works in
                                                          master, just
                                                          needs a second
                                                          or two to
                                                          fetch the
                                                          version file.<br class="">
                                                          <br class="">
                                                          Regards,<br class="">
                                                          Michael<br class="">
                                                          <br class="">
                                                          <br class="">
                                                          <div class="moz-cite-prefix">Am
                                                          22.03.21 um
                                                          10:38 schrieb
                                                          Mark
                                                          Webb-Johnson:<br class="">
                                                          </div>
                                                          <blockquote type="cite" cite="mid:1ACB6945-97E0-45E7-AA07-4EB5AC42B96F@webb-johnson.net" class="">
                                                          <meta http-equiv="content-type" content="text/html; charset=UTF-8" class="">
                                                          <div dir="ltr" class="">In
                                                          master branch,
                                                          at the moment,
                                                          if a command
                                                          is run from
                                                          the web shell
                                                          (or server
                                                          v2), surely
                                                          the mongoose
                                                          task will
                                                          block as the
                                                          web server /
                                                          server v2
                                                          blocks waiting
                                                          for the
                                                          command to run
                                                          to completion?</div>
                                                          <div dir="ltr" class=""><br class="">
                                                          </div>
                                                          <div dir="ltr" class="">Doesn’t
                                                          necessarily
                                                          need to be a
                                                          networking
                                                          command.
                                                          Something long
                                                          running like
                                                          the string
                                                          speed tests.</div>
                                                          <div dir="ltr" class=""><br class="">
                                                          </div>
                                                          <div dir="ltr" class="">In
                                                          v3.3 I can
                                                          easily detect
                                                          the task wait
                                                          being
                                                          requested in
                                                          the http
                                                          library (by
                                                          seeing if
                                                          current task
                                                          id == mongoose
                                                          task), and
                                                          fail (which I
                                                          should do
                                                          anyway). But I
                                                          am more
                                                          concerned with
                                                          the general
                                                          case now
                                                          (which I think
                                                          may be wrong
                                                          in both master
                                                          and for-v3.3).</div>
                                                          <div dir="ltr" class=""><br class="">
                                                          </div>
                                                          <div dir="ltr" class="">Regards,
                                                          Mark</div>
                                                          <div dir="ltr" class=""><br class="">
                                                          </div>
                                                          <div dir="ltr" class="">
                                                          <blockquote type="cite" class="">On 22
                                                          Mar 2021, at
                                                          5:22 PM,
                                                          Michael Balzer
                                                          <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><dexter@expeedo.de></a>
                                                          wrote:<br class="">
                                                          <br class="">
                                                          </blockquote>
                                                          </div>
                                                          <blockquote type="cite" class="">
                                                          <div dir="ltr" class="">
                                                          <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                                                          I think we
                                                          must avoid
                                                          blocking the
                                                          Mongoose task,
                                                          as that's the
                                                          central
                                                          network
                                                          dispatcher.<br class="">
                                                          <br class="">
                                                          Chris had
                                                          implemented a
                                                          workaround in
                                                          one of his PRs
                                                          that could
                                                          allow that to
                                                          be done
                                                          temporarily by
                                                          running a
                                                          local Mongoose
                                                          main loop
                                                          during a
                                                          synchronous
                                                          operation, but
                                                          I still see
                                                          potential
                                                          issues from
                                                          that, as it
                                                          wasn't the
                                                          standard
                                                          handling as
                                                          done by the
                                                          task, and as
                                                          it may need to
                                                          recurse.<br class="">
                                                          <br class="">
                                                          Maybe the old
                                                          OvmsHttpClient
                                                          using socket
                                                          I/O is the
                                                          right way for
                                                          synchronous
                                                          network
                                                          operations?<br class="">
                                                          <br class="">
                                                          Regards,<br class="">
                                                          Michael<br class="">
                                                          <br class="">
                                                          <br class="">
                                                          <div class="moz-cite-prefix">Am
                                                          22.03.21 um
                                                          07:15 schrieb
                                                          Mark
                                                          Webb-Johnson:<br class="">
                                                          </div>
                                                          <blockquote type="cite" cite="mid:F88B060F-2CB7-4F77-99AD-08ACD03E95B9@webb-johnson.net" class="">
                                                          <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                                                          Not sure how
                                                          to resolve
                                                          this.
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">
                                                          <div style="caret-color:
                                                          rgb(0, 0, 0);" class="">OvmsSyncHttpClient
                                                          is currently
                                                          used in
                                                          commands from
                                                          ovms_plugins
                                                          and ovms_ota.</div>
                                                          </div>
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">I
                                                          could bring
                                                          back
                                                          the OvmsHttpClient
                                                          blocking
                                                          (non-mongoose)
implementation, but I don’t think that would address the core problem
                                                          here:</div>
                                                          <div class=""><br class="">
                                                          </div>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <div class="">Inside
                                                          a mongoose
                                                          callback
                                                          (inside the
                                                          mongoose
                                                          networking
                                                          task), we are
                                                          making
                                                          blocking calls
                                                          (and in
                                                          particular
                                                          calls that
                                                          could block
                                                          for several
                                                          tens of
                                                          seconds).</div>
                                                          </blockquote>
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">But
                                                          fundamentally
                                                          is it ok to
                                                          block the
                                                          mongoose
                                                          networking
                                                          task for
                                                          extended
                                                          periods during
                                                          a mongoose
                                                          event
                                                          callback?</div>
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">Mark</div>
                                                          <div class="">
                                                          <div class=""><br class="">
                                                          <blockquote type="cite" class="">
                                                          <div class="">On
                                                          21 Mar 2021,
                                                          at 9:57 PM,
                                                          Michael Balzer
                                                          <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
                                                          wrote:</div>
                                                          <br class="Apple-interchange-newline">
                                                          <div class="">
                                                          <div class="content-isolator__container">
                                                          <div class="protected-part">
                                                          <div class="protected-title">Signed
                                                          PGP part</div>
                                                          <div class="protected-content">
                                                          <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                                                          <div class="">
                                                          I've found
                                                          opening the
                                                          web UI
                                                          firmware page
                                                          or calling
                                                          "ota status"
                                                          via ssh to
                                                          consistently
                                                          deadlock the
                                                          network on my
                                                          module.<br class="">
                                                          <br class="">
                                                          <font class="" face="monospace">I (130531) webserver: HTTP GET /cfg/firmware<br class="">
                                                          D (130531)
                                                          http:
                                                          OvmsSyncHttpClient:
                                                          Connect to <a href="http://ovms.dexters-web.de/" class="" moz-do-not-send="true">ovms.dexters-web.de:80</a><br class="">
                                                          D (130541)
                                                          http:
                                                          OvmsSyncHttpClient:
                                                          waiting for
                                                          completion</font><br class="">
                                                          <br class="">
                                                          After that log
                                                          message, the
                                                          network is
                                                          dead, and the
                                                          netmanager
                                                          also doesn't
                                                          respond:<br class="">
                                                          <br class="">
                                                          <font class="" face="monospace">OVMS# network list <br class="">
                                                          ERROR: job
                                                          failed<br class="">
                                                          D (183241)
                                                          netmanager:
                                                          send cmd 1
                                                          from
                                                          0x3ffe7054<br class="">
                                                          W (193241)
                                                          netmanager:
                                                          ExecuteJob:
                                                          cmd 1: timeout</font><br class="">
                                                          <br class="">
                                                          The interfaces
                                                          seem to be
                                                          registered and
                                                          online, but
                                                          nothing gets
                                                          in or out:<br class="">
                                                          <br class="">
                                                          <font class="" face="monospace">OVMS# network status <br class="">
                                                          Interface#3:
                                                          pp3 (ifup=1
                                                          linkup=1)<br class="">
                                                            IPv4:
                                                          10.170.195.13/255.255.255.255
                                                          gateway
                                                          10.64.64.64<br class="">
                                                          <br class="">
                                                          Interface#2:
                                                          ap2 (ifup=1
                                                          linkup=1)<br class="">
                                                            IPv4:
                                                          192.168.4.1/255.255.255.0
                                                          gateway
                                                          192.168.4.1<br class="">
                                                          <br class="">
                                                          Interface#1:
                                                          st1 (ifup=1
                                                          linkup=1)<br class="">
                                                            IPv4:
                                                          192.168.2.106/255.255.255.0
                                                          gateway
                                                          192.168.2.1<br class="">
                                                          <br class="">
                                                          DNS:
                                                          192.168.2.1<br class="">
                                                          <br class="">
                                                          Default
                                                          Interface: st1
(192.168.2.106/255.255.255.0 gateway 192.168.2.1)</font><br class="">
                                                          <br class="">
                                                          <br class="">
                                                          A couple of
                                                          minutes later,
                                                          server-v2
                                                          recognizes the
                                                          stale
                                                          connection and
                                                          issues a
                                                          network
                                                          restart, which
                                                          fails
                                                          resulting in
                                                          the same
                                                          behaviour as
                                                          shown below
                                                          with finally
                                                          forced reboot
                                                          by loss of an
                                                          important
                                                          event.<br class="">
                                                          <br class="">
                                                          Doing "ota
                                                          status" from
                                                          USB works
                                                          normally, so
                                                          this looks
                                                          like
                                                          OvmsSyncHttpClient
                                                          not being able
                                                          to run from
                                                          within a
                                                          mongoose
                                                          client.<br class="">
                                                          <br class="">
                                                          Regards,<br class="">
                                                          Michael<br class="">
                                                          <br class="">
                                                          <br class="">
                                                          <div class="moz-cite-prefix">Am
                                                          18.03.21 um
                                                          08:14 schrieb
                                                          Mark
                                                          Webb-Johnson:<br class="">
                                                          </div>
                                                          <blockquote type="cite" cite="mid:1FD94B8C-1060-44BF-8E3A-6D3534352100@webb-johnson.net" class="">
                                                          <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
                                                          Tried to
                                                          repeat this,
                                                          but not having
                                                          much success.
                                                          Here is my car
                                                          module, with
                                                          network still
                                                          up:
                                                          <div class=""><br class="">
                                                          </div>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <div class="">
                                                          <div class="">OVMS#
                                                          boot status</div>
                                                          <div class="">Last
                                                          boot was
                                                          262355
                                                          second(s) ago</div>
                                                          </div>
                                                          </blockquote>
                                                          <div class="">
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">I
                                                          did manage to
                                                          catch one
                                                          network
                                                          related crash
                                                          after
                                                          repeatedly
                                                          disconnecting
                                                          and
                                                          reconnecting
                                                          the cellular
                                                          antenna. This
                                                          was:</div>
                                                          <div class=""><br class="">
                                                          </div>
                                                          </div>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <div class="">
                                                          <div class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">I
                                                          (3717989)
                                                          cellular: PPP
                                                          Connection
                                                          disconnected</span></font></div>
                                                          <div class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">Guru
                                                          Meditation
                                                          Error: Core  1
                                                          panic'ed
                                                          (LoadProhibited).
                                                          Exception was
                                                          unhandled.</span></font></div>
                                                          </div>
                                                          <div class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></div>
                                                          <div class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400fe082
                                                          is in
                                                          OvmsNetManager::PrioritiseAndIndicate()
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_netmanager.cpp:707).</span></font></div>
                                                          </blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <div class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">707  
                                                                if
                                                          ((pri->name[0]==search[0])&&</span></font></div>
                                                          </blockquote>
                                                          <font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400ed360
                                                          is in
                                                          OvmsMetricString::SetValue(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_metrics.cpp:1358).</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">1357
                                                             void
                                                          OvmsMetricString::SetValue(std::string
                                                          value)</span></font></blockquote>
                                                          <font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x4008bdad
                                                          is at
                                                          ../../../../.././newlib/libc/machine/xtensa/strcmp.S:586.</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x4008bdd1
                                                          is at
                                                          ../../../../.././newlib/libc/machine/xtensa/strcmp.S:604.</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400fe886
                                                          is in
                                                          OvmsNetManager::ModemDown(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, void*)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_netmanager.cpp:522).</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">522  
                                                               
                                                          PrioritiseAndIndicate();</span></font></blockquote>
                                                          </blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400fd752
                                                          is in
                                                          std::_Function_handler<void
(std::__cxx11::basic_string<char, std::char_traits<char>,
                                                          std::allocator<char>
                                                          >, void*),
std::_Bind<std::_Mem_fn<void
(OvmsNetManager::*)(std::__cxx11::basic_string<char,
                                                          std::char_traits<char>,
std::allocator<char> >, void*)> (OvmsNetManager*,
                                                          std::_Placeholder<1>,
std::_Placeholder<2>)> >::_M_invoke(std::_Any_data
                                                          const&,
                                                          std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&&,
void*&&)
(/home/openvehicles/build/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/functional:600).</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">600  
                                                                    {
                                                          return
(__object->*_M_pmf)(std::forward<_Args>(__args)...); }</span></font></blockquote>
                                                          </blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400f512e
                                                          is in
                                                          std::function<void
(std::__cxx11::basic_string<char, std::char_traits<char>,
                                                          std::allocator<char>
                                                          >,
                                                          void*)>::operator()(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, void*)
                                                          const
(/home/openvehicles/build/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/functional:2271).</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">2271
                                                                 
                                                           return
                                                          _M_invoker(_M_functor,
std::forward<_ArgTypes>(__args)...);</span></font></blockquote>
                                                          </blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400f52f1
                                                          is in
                                                          OvmsEvents::HandleQueueSignalEvent(event_queue_t*)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_events.cpp:283).</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">283  
                                                                   
                                                          m_current_callback->m_callback(m_current_event,
msg->body.signal.data);</span></font></blockquote>
                                                          </blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400f53d8
                                                          is in
                                                          OvmsEvents::EventTask()
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_events.cpp:237).</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">237  
                                                                     
HandleQueueSignalEvent(&msg);</span></font></blockquote>
                                                          </blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class=""><br class="">
                                                          </span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">0x400f545d
                                                          is in
                                                          EventLaunchTask(void*)
(/home/openvehicles/build/Open-Vehicle-Monitoring-System-pre/vehicle/OVMS.V3/main/ovms_events.cpp:80).</span></font></blockquote>
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class="">
                                                          <blockquote style="margin:
                                                          0 0 0 40px;
                                                          border: none;
                                                          padding: 0px;" class=""><font class="" face="Andale
                                                          Mono"><span style="font-style:
                                                          normal;
                                                          font-size:
                                                          12px;" class="">80  
                                                             
                                                           me->EventTask();</span></font></blockquote>
                                                          </blockquote>
                                                          <div class="">
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">My
                                                          for_v3.3
                                                          branch does
                                                          include the
                                                          preliminary
                                                          changes to
                                                          support the
                                                          wifi at 20MHz
                                                          bandwidth, and
                                                          perhaps those
                                                          could be
                                                          affecting
                                                          things. I do
                                                          notice that if
                                                          I ‘power wifi
                                                          off’, then
                                                          ‘wifi mode
                                                          client’, it
                                                          can connect to
                                                          the station,
                                                          but not get an
                                                          IP address.
                                                          I’ve just
                                                          tried to merge
                                                          in the latest
                                                          fixes to that,
                                                          and rebuilt a
                                                          release. I
                                                          will continue
                                                          to test with
                                                          that.</div>
                                                          <div class=""><br class="">
                                                          </div>
                                                          <div class="">Regards,
                                                          Mark.</div>
                                                          <div class=""><br class="">
                                                          <blockquote type="cite" class="">
                                                          <div class="">On
                                                          12 Mar 2021,
                                                          at 10:32 PM,
                                                          Michael Balzer
                                                          <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
                                                          wrote:</div>
                                                          <br class="Apple-interchange-newline">
                                                          <div class="">
                                                          <div class="">
                                                          <div class="content-isolator__container">
                                                          <div class="protected-part">
                                                          <div class="protected-title">Signed
                                                          PGP part</div>
                                                          <div class="protected-content">I
                                                          just tried
                                                          switching to
                                                          for-v3.3 in my
                                                          car module
                                                          after tests on
                                                          my desk module
                                                          were OK, and
                                                          I've run into
                                                          the very same
                                                          problem with
                                                          for-v3.3. So
                                                          the issue
                                                          isn't related
                                                          to esp-idf.<br class="">
                                                          <br class="">
                                                          The network
                                                          only
                                                          occasionally
                                                          starts
                                                          normally, but
                                                          even then all
                                                          connectivity
                                                          is lost after
                                                          a couple of
                                                          minutes.<br class="">
                                                          <br class="">
                                                          The stale
                                                          connection
                                                          watchdog in
                                                          server-v2
                                                          triggers a
                                                          network
                                                          restart, but
                                                          that also
                                                          doesn't seem
                                                          to succeed:<br class="">
                                                          <br class="">
                                                          2021-03-12
                                                          14:53:01.802
                                                          CET W (981652)
ovms-server-v2: Detected stale connection (issue #241), restarting
                                                          network<br class="">
                                                          2021-03-12
                                                          14:53:01.802
                                                          CET I (981652)
                                                          esp32wifi:
                                                          Restart<br class="">
                                                          2021-03-12
                                                          14:53:01.802
                                                          CET I (981652)
                                                          esp32wifi:
                                                          Stopping WIFI
                                                          station<br class="">
                                                          2021-03-12
                                                          14:53:01.812
                                                          CET I (981662)
                                                          wifi:state:
                                                          run -> init
                                                          (0)<br class="">
                                                          2021-03-12
                                                          14:53:01.812
                                                          CET I (981662)
                                                          wifi:pm stop,
                                                          total sleep
                                                          time:
                                                          831205045 us /
                                                          975329961 us<br class="">
                                                          <br class="">
                                                          2021-03-12
                                                          14:53:01.812
                                                          CET I (981662)
wifi:new:<1,0>, old:<1,1>, ap:<1,1>, sta:<1,1>,
                                                          prof:1<br class="">
                                                          2021-03-12
                                                          14:53:01.832
                                                          CET I (981682)
                                                          wifi:flush txq<br class="">
                                                          2021-03-12
                                                          14:53:01.842
                                                          CET I (981692)
                                                          wifi:stop sw
                                                          txq<br class="">
                                                          2021-03-12
                                                          14:53:01.842
                                                          CET I (981692)
                                                          wifi:lmac stop
                                                          hw txq<br class="">
                                                          2021-03-12
                                                          14:53:01.852
                                                          CET I (981702)
                                                          esp32wifi:
                                                          Powering down
                                                          WIFI driver<br class="">
                                                          2021-03-12
                                                          14:53:01.852
                                                          CET I (981702)
                                                          wifi:Deinit
                                                          lldesc rx
                                                          mblock:16<br class="">
                                                          2021-03-12
                                                          14:53:01.862
                                                          CET I (981712)
                                                          esp32wifi:
                                                          Powering up
                                                          WIFI driver<br class="">
                                                          2021-03-12
                                                          14:53:01.862
                                                          CET I (981712)
wifi:nvs_log_init, erase log key successfully, reinit nvs log<br class="">
                                                          2021-03-12
                                                          14:53:01.882
                                                          CET I (981732)
                                                          wifi:wifi
                                                          driver task:
                                                          3ffd4d84,
                                                          prio:23,
                                                          stack:3584,
                                                          core=0<br class="">
                                                          2021-03-12
                                                          14:53:01.882
                                                          CET I (981732)
                                                          system_api:
                                                          Base MAC
                                                          address is not
                                                          set, read
                                                          default base
                                                          MAC address
                                                          from BLK0 of
                                                          EFUSE<br class="">
                                                          2021-03-12
                                                          14:53:01.882
                                                          CET I (981732)
                                                          system_api:
                                                          Base MAC
                                                          address is not
                                                          set, read
                                                          default base
                                                          MAC address
                                                          from BLK0 of
                                                          EFUSE<br class="">
                                                          2021-03-12
                                                          14:53:01.902
                                                          CET I (981752)
                                                          wifi:wifi
                                                          firmware
                                                          version:
                                                          30f9e79<br class="">
                                                          2021-03-12
                                                          14:53:01.912
                                                          CET I (981762)
                                                          wifi:config
                                                          NVS flash:
                                                          enabled<br class="">
                                                          2021-03-12
                                                          14:53:01.912
                                                          CET I (981762)
                                                          wifi:config
                                                          nano
                                                          formating:
                                                          disabled<br class="">
                                                          2021-03-12
                                                          14:53:01.912
                                                          CET I (981762)
                                                          wifi:Init data
                                                          frame dynamic
                                                          rx buffer num:
                                                          16<br class="">
                                                          2021-03-12
                                                          14:53:01.912
                                                          CET I (981762)
                                                          wifi:Init
                                                          management
                                                          frame dynamic
                                                          rx buffer num:
                                                          16<br class="">
                                                          2021-03-12
                                                          14:53:01.922
                                                          CET I (981772)
                                                          wifi:Init
                                                          management
                                                          short buffer
                                                          num: 32<br class="">
                                                          2021-03-12
                                                          14:53:01.922
                                                          CET I (981772)
                                                          wifi:Init
                                                          dynamic tx
                                                          buffer num: 16<br class="">
                                                          2021-03-12
                                                          14:53:01.922
                                                          CET I (981772)
                                                          wifi:Init
                                                          static rx
                                                          buffer size:
                                                          2212<br class="">
                                                          2021-03-12
                                                          14:53:01.922
                                                          CET I (981772)
                                                          wifi:Init
                                                          static rx
                                                          buffer num: 16<br class="">
                                                          2021-03-12
                                                          14:53:01.922
                                                          CET I (981772)
                                                          wifi:Init
                                                          dynamic rx
                                                          buffer num: 16<br class="">
                                                          2021-03-12
                                                          14:53:02.642
                                                          CET I (982492)
                                                          wifi:mode :
                                                          sta
                                                          (30:ae:a4:5f:e7:ec)
                                                          + softAP
                                                          (30:ae:a4:5f:e7:ed)<br class="">
                                                          2021-03-12
                                                          14:53:02.652
                                                          CET I (982502)
                                                          wifi:Total
                                                          power save
                                                          buffer number:
                                                          8<br class="">
                                                          2021-03-12
                                                          14:53:02.652
                                                          CET I (982502)
cellular-modem-auto: Restart<br class="">
                                                          2021-03-12
                                                          14:53:02.662
                                                          CET I (982512)
                                                          cellular:
                                                          State: Enter
                                                          PowerOffOn
                                                          state<br class="">
                                                          2021-03-12
                                                          14:53:02.662
                                                          CET I (982512)
                                                          gsm-ppp:
                                                          Shutting down
                                                          (hard)...<br class="">
                                                          2021-03-12
                                                          14:53:02.662
                                                          CET I (982512)
                                                          gsm-ppp:
                                                          StatusCallBack:
                                                          User Interrupt<br class="">
                                                          2021-03-12
                                                          14:53:02.662
                                                          CET I (982512)
                                                          gsm-ppp: PPP
                                                          connection has
                                                          been closed<br class="">
                                                          2021-03-12
                                                          14:53:02.662
                                                          CET I (982512)
                                                          gsm-ppp: PPP
                                                          is shutdown<br class="">
                                                          2021-03-12
                                                          14:53:02.662
                                                          CET I (982512)
                                                          gsm-ppp:
                                                          Shutting down
                                                          (hard)...<br class="">
                                                          2021-03-12
                                                          14:53:02.672
                                                          CET I (982522)
                                                          gsm-ppp:
                                                          StatusCallBack:
                                                          User Interrupt<br class="">
                                                          2021-03-12
                                                          14:53:02.672
                                                          CET I (982522)
                                                          gsm-ppp: PPP
                                                          connection has
                                                          been closed<br class="">
                                                          2021-03-12
                                                          14:53:02.672
                                                          CET I (982522)
                                                          gsm-ppp: PPP
                                                          is shutdown<br class="">
                                                          2021-03-12
                                                          14:53:02.672
                                                          CET I (982522)
                                                          gsm-nmea:
                                                          Shutdown
                                                          (direct)<br class="">
                                                          2021-03-12
                                                          14:53:02.672
                                                          CET I (982522)
cellular-modem-auto: Power Cycle<br class="">
                                                          2021-03-12
                                                          14:53:04.682
                                                          CET D (984532)
                                                          events:
                                                          Signal(system.wifi.down)<br class="">
                                                          2021-03-12
                                                          14:53:04.682
                                                          CET I (984532)
                                                          netmanager:
                                                          WIFI client
                                                          stop<br class="">
                                                          2021-03-12
                                                          14:53:04.682
                                                          CET E (984532)
                                                          netmanager:
                                                          Inconsistent
                                                          state: no
                                                          interface of
                                                          type 'pp'
                                                          found<br class="">
                                                          2021-03-12
                                                          14:53:04.682
                                                          CET I (984532)
                                                          netmanager:
                                                          WIFI client
                                                          down (with
                                                          MODEM up):
                                                          reconfigured
                                                          for MODEM
                                                          priority<br class="">
                                                          2021-03-12
                                                          14:53:04.692
                                                          CET D (984542)
                                                          events:
                                                          Signal(system.event)<br class="">
                                                          2021-03-12
                                                          14:53:04.692
                                                          CET D (984542)
                                                          events:
                                                          Signal(system.wifi.sta.disconnected)<br class="">
                                                          2021-03-12
                                                          14:53:04.692
                                                          CET E (984542)
                                                          netmanager:
                                                          Inconsistent
                                                          state: no
                                                          interface of
                                                          type 'pp'
                                                          found<br class="">
                                                          2021-03-12
                                                          14:53:04.692
                                                          CET I (984542)
                                                          esp32wifi: STA
                                                          disconnected
                                                          with reason 8
                                                          = ASSOC_LEAVE<br class="">
                                                          2021-03-12
                                                          14:53:04.702
                                                          CET D (984552)
                                                          events:
                                                          Signal(system.event)<br class="">
                                                          2021-03-12
                                                          14:53:04.702
                                                          CET D (984552)
                                                          events:
                                                          Signal(system.wifi.sta.stop)<br class="">
                                                          2021-03-12
                                                          14:53:04.702
                                                          CET E (984552)
                                                          netmanager:
                                                          Inconsistent
                                                          state: no
                                                          interface of
                                                          type 'pp'
                                                          found<br class="">
                                                          2021-03-12
                                                          14:53:04.712
                                                          CET D (984562)
                                                          events:
                                                          Signal(system.event)<br class="">
                                                          2021-03-12
                                                          14:53:04.712
                                                          CET D (984562)
                                                          events:
                                                          Signal(system.wifi.ap.stop)<br class="">
                                                          2021-03-12
                                                          14:53:04.712
                                                          CET E (984562)
                                                          netmanager:
                                                          Inconsistent
                                                          state: no
                                                          interface of
                                                          type 'pp'
                                                          found<br class="">
                                                          2021-03-12
                                                          14:53:04.712
                                                          CET I (984562)
                                                          netmanager:
                                                          WIFI access
                                                          point is down<br class="">
                                                          2021-03-12
                                                          14:53:04.712
                                                          CET I (984562)
                                                          esp32wifi: AP
                                                          stopped<br class="">
                                                          2021-03-12
                                                          14:53:04.722
                                                          CET D (984572)
                                                          events:
                                                          Signal(network.wifi.sta.bad)<br class="">
                                                          2021-03-12
                                                          14:53:04.722
                                                          CET D (984572)
                                                          events:
                                                          Signal(system.event)<br class="">
                                                          2021-03-12
                                                          14:53:04.722
                                                          CET D (984572)
                                                          events:
                                                          Signal(system.wifi.sta.start)<br class="">
                                                          2021-03-12
                                                          14:53:04.732
                                                          CET D (984582)
                                                          events:
                                                          Signal(system.modem.down)<br class="">
                                                          2021-03-12
                                                          14:53:04.742
                                                          CET I (984592)
                                                          netmanager:
                                                          MODEM down
                                                          (with WIFI
                                                          client down):
                                                          network
                                                          connectivity
                                                          has been lost<br class="">
                                                          2021-03-12
                                                          14:53:04.742
                                                          CET D (984592)
                                                          events:
                                                          Signal(system.modem.down)<br class="">
                                                          2021-03-12
                                                          14:53:04.752
                                                          CET D (984602)
                                                          events:
                                                          Signal(system.event)<br class="">
                                                          2021-03-12
                                                          14:53:04.752
                                                          CET D (984602)
                                                          events:
                                                          Signal(system.wifi.ap.start)<br class="">
                                                          2021-03-12
                                                          14:53:04.752
                                                          CET I (984602)
                                                          netmanager:
                                                          WIFI access
                                                          point is up<br class="">
                                                          2021-03-12
                                                          14:53:26.802
                                                          CET E
                                                          (1006652)
                                                          events:
                                                          SignalEvent:
                                                          queue overflow
                                                          (running
                                                          system.wifi.ap.start->netmanager
                                                          for 23 sec),
                                                          event
                                                          'ticker.1'
                                                          dropped<br class="">
                                                          2021-03-12
                                                          14:53:27.802
                                                          CET E
                                                          (1007652)
                                                          events:
                                                          SignalEvent:
                                                          queue overflow
                                                          (running
                                                          system.wifi.ap.start->netmanager
                                                          for 24 sec),
                                                          event
                                                          'ticker.1'
                                                          dropped<br class="">
                                                          2021-03-12
                                                          14:53:28.802
                                                          CET E
                                                          (1008652)
                                                          events:
                                                          SignalEvent:
                                                          queue overflow
                                                          (running
                                                          system.wifi.ap.start->netmanager
                                                          for 25 sec),
                                                          event
                                                          'ticker.1'
                                                          dropped<br class="">
                                                          …and so on
                                                          until<br class="">
                                                          2021-03-12
                                                          14:54:01.802
                                                          CET E
                                                          (1041652)
                                                          events:
                                                          SignalEvent:
                                                          lost important
                                                          event =>
                                                          aborting<br class="">
                                                          <br class="">
                                                          <br class="">
                                                          I need my car
                                                          now, so will
                                                          switch back to
                                                          master for
                                                          now.<br class="">
                                                          <br class="">
                                                          Mark, if
                                                          you've got
                                                          specific debug
                                                          logs I should
                                                          fetch on the
                                                          next try, tell
                                                          me.<br class="">
                                                          <br class="">
                                                          Regards,<br class="">
                                                          Michael<br class="">
                                                          <br class="">
                                                          <br class="">
                                                          Am 12.03.21 um
                                                          05:47 schrieb
                                                          Craig Leres:<br class="">
                                                          <blockquote type="cite" class="">I
                                                          just updated
                                                          to
                                                          3.2.016-68-g8e10c6b7
                                                          and still get
                                                          the network
                                                          hang
                                                          immediately
                                                          after booting
                                                          and logging
                                                          into the web
                                                          gui.<br class="">
                                                          <br class="">
                                                          But I see now
                                                          my problem is
                                                          likely that
                                                          I'm not using
                                                          the right
                                                          esp-idf (duh).
                                                          Is there a way
                                                          I can have
                                                          master build
                                                          using
                                                          ~/esp/esp-idf
                                                          and have
                                                          for-v3.3 use a
                                                          different
                                                          path?)<br class="">
                                                          <br class="">
                                                                  Craig<br class="">
_______________________________________________<br class="">
                                                          OvmsDev
                                                          mailing list<br class="">
                                                          <a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
                                                          <a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
                                                          </blockquote>
                                                          <br class="">
                                                          -- <br class="">
                                                          Michael Balzer
                                                          * Helkenberger
                                                          Weg 9 *
                                                          D-58256
                                                          Ennepetal<br class="">
                                                          Fon 02333 /
                                                          833 5735 *
                                                          Handy 0176 /
                                                          206 989 26<br class="">
                                                          <br class="">
                                                          <br class="">
                                                          </div>
                                                          </div>
                                                          <br class="">
                                                          <iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br class="">
                                                          </div>
                                                          <br class="">
                                                          <fieldset class="mimeAttachmentHeader"></fieldset>
                                                          <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                                          </blockquote>
                                                          <br class="">
                                                          <pre class="moz-signature" cols="72">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br class="">
                                                          <iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br class="">
                                                          </div>
                                                          <br class="">
                                                          <fieldset class="mimeAttachmentHeader"></fieldset>
                                                          <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                                          </blockquote>
                                                          <br class="">
                                                          <pre class="moz-signature" cols="72">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
                                                          <span class="">_______________________________________________</span><br class="">
                                                          <span class="">OvmsDev
                                                          mailing list</span><br class="">
                                                          <span class=""><a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a></span><br class="">
                                                          <span class=""><a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a></span><br class="">
                                                          </div>
                                                          </blockquote>
                                                          <br class="">
                                                          <fieldset class="mimeAttachmentHeader"></fieldset>
                                                          <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                                          </blockquote>
                                                          <br class="">
                                                          <pre class="moz-signature" cols="72">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br class="">
                                                          <iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br class="">
                                                          </div>
                                                          <br class="">
                                                          <fieldset class="mimeAttachmentHeader"></fieldset>
                                                          <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                                        </blockquote>
                                                        <br class="">
                                                        <pre class="moz-signature" cols="72">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
                                                      </div>
                                                    </div>
                                                  </div>
                                                  <br class="">
                                                  <iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br class="">
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                  <br class="">
                                  <iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
                              </div>
                            </blockquote>
                          </div>
                          <br class="">
                        </div>
                      </div>
                    </div>
                  </div>
                  <br class="">
                  <iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
              </div>
            </blockquote>
          </div>
          <br class="">
        </div>
      </div>
      <br class="">
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
    </blockquote>
    <br class="">
    <pre class="moz-signature" cols="72">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
  </div>

</div></div><br class=""><iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" width="200" height="10" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg=="></iframe></div></div></blockquote></div><br class=""></div></div></body></html>