<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Followed you on my server.<br>
    <br>
    Btw:<br>
    <tt>branch    COUNT(*)</tt><tt><br>
    </tt><tt>eap             14</tt><tt><br>
    </tt><tt>edge            42</tt><tt><br>
    </tt><tt>main           183</tt><br>
    <br>
    Edge has more users than eap, but most of the edge users don't
    enable auto update.<br>
    <br>
    Also:<br>
    <tt>version    cars    crashes    crashratio</tt><tt><br>
    </tt><tt>new          18         16        0.8889</tt><tt><br>
    </tt><tt>old         223       1315        5.8969</tt><br>
    <br>
    That's comparing crashes of my edge release (built with toolchain
    -98) to all previous releases still in use (which also covers some
    3.1.x releases) within the last 10 days, so crash ratio is very good
    with this release and toolchain -98.<br>
    <br>
    I also haven't seen the issue #241 situation again with this
    release.<br>
    <br>
    Regards,<br>
    Michael<br>
    <br>
    <br>
    <div class="moz-cite-prefix">Am 17.09.19 um 08:21 schrieb Mark
      Webb-Johnson:<br>
    </div>
    <blockquote type="cite"
      cite="mid:8A27D7CF-F4B6-4B90-B238-4BA6D2362247@webb-johnson.net">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      OK. I’ve released to EAP. Also included the basic Zoe code.
      <div class=""><br class="">
      </div>
      <div class="">Regards, Mark.<br class="">
        <div><br class="">
          <blockquote type="cite" class="">
            <div class="">On 13 Sep 2019, at 9:03 PM, Michael Balzer
              <<a href="mailto:dexter@expeedo.de" class=""
                moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
            <br class="Apple-interchange-newline">
            <div class="">
              <meta http-equiv="Content-Type" content="text/html;
                charset=UTF-8" class="">
              <div text="#000000" bgcolor="#FFFFFF" class=""> I'd say we
                should release this version now.<br class="">
                <br class="">
                No new issues so far, the #241 workaround works as
                designed, and crash ratio is low.<br class="">
                <br class="">
                Regards,<br class="">
                Michael<br class="">
                <br class="">
                <br class="">
                <div class="moz-cite-prefix">Am 09.09.19 um 17:18
                  schrieb Michael Balzer:<br class="">
                </div>
                <blockquote type="cite"
                  cite="mid:c107cc46-32e3-30c8-bb2e-777a751e01c2@expeedo.de"
                  class="">
                  <meta http-equiv="Content-Type" content="text/html;
                    charset=UTF-8" class="">
                  Mark,<br class="">
                  <br class="">
                  I did the same checks as you and came to the same
                  conclusion.<br class="">
                  <br class="">
                  For the webserver timer, that looks non-trivial but
                  really only pushes requests into the job queue and
                  takes care to never block. A bit of a potential issue
                  there is it does some logging in case of trouble (job
                  queue full), and that may be bad if logging to an SD
                  file is enabled. We still need to rework the file
                  logging to use a separate task as well (issue #107).<br
                    class="">
                  <br class="">
                  From my task data today, here are some first
                  impressions on our normal CPU load:<br class="">
                  <br class="">
                  a) Idle tasks:<br class="">
                  <br class="">
                  <span id="cid:part1.6F0619A2.5B8C50B2@expeedo.de"><ndclolldladjlbpf.png></span><br
                    class="">
                  <br class="">
                  b) Active tasks:<br class="">
                  <br class="">
                  <span id="cid:part2.15B6112D.B51716A0@expeedo.de"><chgkakeodpaocemg.png></span><br
                    class="">
                  <br class="">
                  <br class="">
                  10:00 - 10:30 was a drive from home (wifi) to office
                  (no wifi), then another drive 13:00 - 13:30 and back
                  home 14:00 - 14:15.<br class="">
                  <br class="">
                  Driving load on the NetMan is caused by the web
                  dashboard. The half hour peaks are from the 12V
                  topping from the main battery the Twizy does when
                  parked.<br class="">
                  <br class="">
                  So that's nowhere near "too much" in terms of CPU load
                  -- normally. I'm waiting to catch the issue situation
                  with this instrumentation.<br class="">
                  <br class="">
                  Regards,<br class="">
                  Michael<br class="">
                  <br class="">
                  <br class="">
                  <div class="moz-cite-prefix">Am 09.09.19 um 05:43
                    schrieb Mark Webb-Johnson:<br class="">
                  </div>
                  <blockquote type="cite"
                    cite="mid:B87B8287-6913-42F8-A27E-ED7E0C76C520@webb-johnson.net"
                    class="">
                    <meta http-equiv="Content-Type" content="text/html;
                      charset=UTF-8" class="">
                    <div class="">tldr; Perhaps we should be using
                      esp_timer and esp_timer_get_time() to update
                      monotonic time then dispatch the ticker.* events?</div>
                    <div class=""><br class="">
                    </div>
                    <div class="">Long story: Trying to see how
                      monotonic time could not be updating...</div>
                    <div class=""><br class="">
                    </div>
                    I never really worked out how this operates in ESP32
                    IDF freertos. We have three tasks involved in
                    timers:
                    <div class=""><br class="">
                    </div>
                    <div class="">
                      <ul class="MailOutline">
                        <li class="">esp_timer</li>
                        <li class="">Tmr Svc</li>
                        <li class="">eventTask</li>
                      </ul>
                      <div class=""><br class="">
                      </div>
                      <div class="">From my understanding, Tmr Svc
                        provides high level timer support for freertos
                        (xTimerCreate, etc). And esp_timer provides
                        esp32 specific timer support (<a
href="https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/esp_timer.html"
                          class="" moz-do-not-send="true">https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/esp_timer.html</a>).
                        We use Tmr Svc, and note the warnings provided
                        in the espressif documentation:</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">
                        <div class="">
                          <ul class="MailOutline">
                            <li class="">Maximum resolution is equal to
                              RTOS tick period</li>
                            <li class="">Timer callbacks are dispatched
                              from a low-priority task</li>
                          </ul>
                        </div>
                      </div>
                      <div class=""><br class="">
                      </div>
                      <div class="">Currently, our housekeeping
                        uses xTimerCreate to create a 1 second timer
                        (calling HousekeepingTicker1). So, in which task
                        is HousekeepingTicker1() called (esp_timer or
                        Tmr Svc) - I always suspected the latter (but
                        never checked). Then HousekeepingTicker1()
                        raises a signal (using ovms events) that pushes
                        it onto a queue (if the queue is full, it
                        discards). The eventTask will then read that
                        queue, and dispatch it to our ticker.1
                        listeners.</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">HousekeepingTicker1 is responsible
                        for updating monotonictime, and given it’s
                        simple calling of MyEvents.SignalEvent (which
                        just queues it and discards on overflow), I
                        don’t think that can block for any substantial
                        time.</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">I did a search for xTimerCreate in
                        our code base, and find these used:</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">
                        <ul class="MailOutline">
                          <li class="">components/ovms_webserver/src/ovms_webserver.cpp<br
                              class="">
                            m_update_ticker = xTimerCreate("Web client
                            update ticker", 250 / portTICK_PERIOD_MS,
                            pdTRUE, NULL, UpdateTicker);<br class="">
                            <br class="">
                            This seems to do quite a bit. In particular
                            queue handling and semaphores. All seem to
                            be non-blocking, but the flow is
                            non-trivial.<br class="">
                            <br class="">
                          </li>
                          <li class="">components/vehicle_nissanleaf/src/vehicle_nissanleaf.cpp<br
                              class="">
                            m_remoteCommandTimer = xTimerCreate("Nissan
                            Leaf Remote Command", 100 /
                            portTICK_PERIOD_MS, pdTRUE, this,
                            remoteCommandTimer);<br class="">
                            m_ccDisableTimer = xTimerCreate("Nissan Leaf
                            CC Disable", 1000 / portTICK_PERIOD_MS,
                            pdFALSE, this, ccDisableTimer);<br class="">
                            <br class="">
                            Seem ok, and non-blocking.<br class="">
                            <br class="">
                          </li>
                          <li class="">components/vehicle_renaulttwizy/src/rt_sevcon.cpp<br
                              class="">
                            m_kickdown_timer = xTimerCreate("RT
                            kickdown", pdMS_TO_TICKS(100), pdTRUE, NULL,
                            KickdownTimer);<br class="">
                            <br class="">
                            Seems ok.<br class="">
                            <br class="">
                          </li>
                          <li class="">components/vehicle_smarted/src/vehicle_smarted.cpp<br
                              class="">
                            m_locking_timer = xTimerCreate("Smart ED
                            Locking Timer", 500 / portTICK_PERIOD_MS,
                            pdTRUE, this, SmartEDLockingTimer);<br
                              class="">
                            <br class="">
                            This code looks a bit dodgy because
                            CommandLock and CommandUnlock both create
                            this timer, and start it - but neither check
                            if it is already created. That said, after
                            it fires the timer is deleted by the
                            handler.<br class="">
                            <br class="">
                          </li>
                          <li class="">components/vehicle_teslaroadster/src/vehicle_teslaroadster.cpp<br
                              class="">
                            m_speedo_timer = xTimerCreate("TR ticker",<br
                              class="">
                            m_homelink_timer = xTimerCreate("Tesla
                            Roadster Homelink Timer", durationms /
                            portTICK_PERIOD_MS, pdTRUE, this,
                            TeslaRoadsterHomelinkTimer);<br class="">
                            <br class="">
                            Similar lack of checking for duplicate
                            timers. But I don’t see any blocking.</li>
                        </ul>
                      </div>
                      <div class=""><br class="">
                      </div>
                      <div class="">So, I don’t really think _we_ are
                        starving the TmrSvc. Most likely something in
                        the core framework.</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">Regards, Mark.</div>
                      <div class=""><br class="">
                        <blockquote type="cite" class="">
                          <div class="">On 7 Sep 2019, at 4:55 PM,
                            Michael Balzer <<a
                              href="mailto:dexter@expeedo.de" class=""
                              moz-do-not-send="true">dexter@expeedo.de</a>>
                            wrote:</div>
                          <br class="Apple-interchange-newline">
                          <div class="">
                            <meta http-equiv="Content-Type"
                              content="text/html; charset=UTF-8"
                              class="">
                            <div text="#000000" bgcolor="#FFFFFF"
                              class=""> I think the RTOS timer service
                              task starves. It's running on core 0 with
                              priority 1.<br class="">
                              <br class="">
                              Taks on core 0 sorted by priority:<br
                                class="">
                              <br class="">
                              <tt class="">Number of Tasks = 20     
                                Stack:  Now   Max Total    Heap 32-bit
                                SPIRAM C# PRI</tt><tt class=""><br
                                  class="">
                              </tt><tt class=""><tt class="">3FFC84A8  6
                                  Blk ipc0              388   500 
                                  1024    7788      0      0  0  24</tt><tt
                                  class=""><br class="">
                                </tt></tt><tt class=""><tt class="">3FFC77F0 
                                  5 Blk OVMS CanRx        428   428 
                                  2048    3052      0  31844  0  23</tt><tt
                                  class=""><br class="">
                                </tt>3FFAFBF4  1 Blk esp_timer        
                                400   656  4096   35928    644  25804 
                                0  22</tt><tt class=""><br class="">
                              </tt><tt class=""><tt class="">3FFD3240 19
                                  Blk wifi              460  2716 
                                  3584   43720      0     20  0  22</tt><tt
                                  class=""><br class="">
                                </tt>3FFC03C4  2 Blk eventTask        
                                448  1984  4608     104      0      0 
                                0  20</tt><tt class=""><br class="">
                              </tt><tt class=""><tt class="">3FFC8F14 17
                                  Blk tiT               500  2308 
                                  3072    6552      0      0  *  18</tt><tt
                                  class=""><br class="">
                                </tt></tt><tt class=""><tt class="">3FFE14F0
                                  26 Blk OVMS COrx         456   456 
                                  4096       0      0      0  0   7</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">3FFE19D4 27 Blk OVMS
                                  COwrk        476   476  3072      
                                  0      0      0  0   7</tt><tt
                                  class=""><br class="">
                                </tt>3FFCBC34 12 Blk Tmr Svc          
                                352   928  3072      88      0      0 
                                0   1</tt><tt class=""><br class="">
                              </tt><tt class="">3FFE7708 23 Blk
                                mdns              468  1396  4096    
                                108      0      0  0   1</tt><tt
                                class=""><br class="">
                              </tt><br class="">
                              I don't think it's our CanRx, as that only
                              fetches and queues CAN frames, the actual
                              work is done by the listeners. The CO
                              tasks only run for CANopen jobs, which are
                              few for normal operation.<br class="">
                              <br class="">
                              That leaves the system tasks, with main
                              suspect -once again- the wifi blob.<br
                                class="">
                              <br class="">
                              We need to know how much CPU time the
                              tasks actually use now. I think I saw some
                              option for this in the FreeRTOS config.<br
                                class="">
                              <br class="">
                              Regards,<br class="">
                              Michael<br class="">
                              <br class="">
                              <br class="">
                              <div class="moz-cite-prefix">Am 06.09.19
                                um 23:15 schrieb Michael Balzer:<br
                                  class="">
                              </div>
                              <blockquote type="cite"
                                cite="mid:01014878-5a73-b611-4334-989b8aea76c5@expeedo.de"
                                class="">
                                <meta http-equiv="Content-Type"
                                  content="text/html; charset=UTF-8"
                                  class="">
                                The workaround is based on the
                                monotonictime being updated per second,
                                as do the history record offsets.<br
                                  class="">
                                <br class="">
                                Apparently, that mechanism doesn't work
                                reliably. That may be an indicator for
                                some bigger underlying issue.<br
                                  class="">
                                <br class="">
                                Example log excerpt:<br class="">
                                <br class="">
                                <tt class="">2019-09-06 22:07:48.126919
                                  +0200 info  main: #173 C MITPROHB rx
                                  msg h
                                  964,0,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:03.089031 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-10,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:05.041574 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-20,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:05.052644 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-30,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:05.063617 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-49,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:05.077527 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-59,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:05.193775 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-70,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:13.190645 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-80,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:22.077994 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-90,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:09:54.590300 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-109,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:11:10.127054 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-119,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:11:16.794200 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-130,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:11:22.455652 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-140,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:12:49.423412 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-150,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:12:49.442096 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-169,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:12:49.461941 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-179,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:14:39.828133 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-190,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:14:39.858144 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-200,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:14:52.020319 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-210,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:14:54.452637 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-229,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:15:12.613935 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-239,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:15:35.223845 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-250,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:16:09.255059 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-260,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:17:31.919754 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-270,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:19:23.366267 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-289,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:21:57.344609 +0200 info  main: #173
                                  C MITPROHB rx msg h
                                  964,-299,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:23:40.082406 +0200 info  main: #31
                                  C MITPROHB rx msg h
                                  964,-1027,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><tt class="">2019-09-06
                                  22:25:58.061883 +0200 info  main: #31
                                  C MITPROHB rx msg h
                                  964,-1040,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
                                  class=""><br class="">
                                </tt><br class="">
                                <br class="">
                                This shows the ticker was only run 299
                                times from 22:07:48 to 22:21:57.<br
                                  class="">
                                <br class="">
                                After 22:21:57 the workaround was
                                triggered and did a reconnect.
                                Apparently during that network
                                reinitialization of 103 seconds, the per
                                second ticker was run 628 times.<br
                                  class="">
                                <br class="">
                                That can't be catching up on the event
                                queue, as that queue has only 20 slots.
                                So something strange is going on here.<br
                                  class="">
                                <br class="">
                                Any ideas?<br class="">
                                <br class="">
                                Regards,<br class="">
                                Michael<br class="">
                                <br class="">
                                <br class="">
                                <div class="moz-cite-prefix">Am 06.09.19
                                  um 08:04 schrieb Michael Balzer:<br
                                    class="">
                                </div>
                                <blockquote type="cite"
                                  cite="mid:9ee4a7b5-59a2-a05d-046d-e932b65e3e34@expeedo.de"
                                  class="">
                                  <meta http-equiv="Content-Type"
                                    content="text/html; charset=UTF-8"
                                    class="">
                                  Mark & anyone else running a V2
                                  server,<br class="">
                                  <br class="">
                                  as most cars don't send history
                                  records, this also needs the change to
                                  the server I just pushed, i.e. server
                                  version 2.4.2.<br class="">
                                  <br class="">
                                  <a
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System/commits/master"
                                    moz-do-not-send="true" class="">https://github.com/openvehicles/Open-Vehicle-Monitoring-System/commits/master</a><br
                                    class="">
                                  <br class="">
                                  Regards,<br class="">
                                  Michael<br class="">
                                  <br class="">
                                  <br class="">
                                  <div class="moz-cite-prefix">Am
                                    05.09.19 um 19:55 schrieb Michael
                                    Balzer:<br class="">
                                  </div>
                                  <blockquote type="cite"
                                    cite="mid:91892018-3ecb-2dc9-b430-e79fe598cc4c@expeedo.de"
                                    class="">
                                    <meta http-equiv="Content-Type"
                                      content="text/html; charset=UTF-8"
                                      class="">
                                    I've pushed the nasty workaround:
                                    the v2 server checks for no RX over
                                    15 minutes, then restarts the
                                    network (wifi & modem) as
                                    configured for autostart.<br
                                      class="">
                                    <br class="">
                                    Rolled out on my server in edge as
                                    3.2.002-237-ge075f655.<br class="">
                                    <br class="">
                                    Please test.<br class="">
                                    <br class="">
                                    Regards,<br class="">
                                    Michael<br class="">
                                    <br class="">
                                    <br class="">
                                    <div class="moz-cite-prefix">Am
                                      05.09.19 um 01:58 schrieb Mark
                                      Webb-Johnson:<br class="">
                                    </div>
                                    <blockquote type="cite"
                                      cite="mid:D4C2658B-49FB-48CB-A687-9A7325117BC4@webb-johnson.net"
                                      class="">
                                      <meta http-equiv="Content-Type"
                                        content="text/html;
                                        charset=UTF-8" class="">
                                      <div class="">
                                        <blockquote type="cite" class="">
                                          <div text="#000000"
                                            bgcolor="#FFFFFF" class="">Mark,
                                            you can check your server
                                            logs for history messages
                                            with ridiculous time
                                            offsets:<br class="">
                                            <blockquote class=""><tt
                                                class="">[sddexter@ns27
                                                server]$ cat
                                                log-20190903 | egrep "rx
                                                msg h [0-9]+,-[0-9]{4}"
                                                | wc -l<br class="">
                                                455283</tt></blockquote>
                                          </div>
                                        </blockquote>
                                        <br class="">
                                      </div>
                                      I checked my logs and see 12
                                      vehicles showing this. But, 2 only
                                      show this for a debugcrash log
                                      (which is expected, I guess, if
                                      the time is not synced at report
                                      time). I’ve got 4 cars with the
                                      offset > 10,000.
                                      <div class=""><br class="">
                                      </div>
                                      <div class="">Regards, Mark.<br
                                          class="">
                                        <div class=""><br class="">
                                          <blockquote type="cite"
                                            class="">
                                            <div class="">On 4 Sep 2019,
                                              at 4:45 AM, Michael Balzer
                                              <<a
                                                href="mailto:dexter@expeedo.de"
                                                class=""
                                                moz-do-not-send="true">dexter@expeedo.de</a>>
                                              wrote:</div>
                                            <br
                                              class="Apple-interchange-newline">
                                            <div class="">
                                              <meta
                                                http-equiv="Content-Type"
                                                content="text/html;
                                                charset=UTF-8" class="">
                                              <div text="#000000"
                                                bgcolor="#FFFFFF"
                                                class=""> Everyone,<br
                                                  class="">
                                                <br class="">
                                                I've pushed a change
                                                that needs some testing.<br
                                                  class="">
                                                <br class="">
                                                I had the issue myself
                                                now parking at a certain
                                                distance from my garage
                                                wifi AP, i.e. on the
                                                edge of "in", after wifi
                                                had been disconnected
                                                for some hours, and with
                                                the module still
                                                connected via modem. The
                                                wifi blob had been
                                                trying to connect to the
                                                AP for about two hours.<br
                                                  class="">
                                                <br class="">
                                                As seen before, the
                                                module saw no error,
                                                just the server
                                                responses and commands
                                                stopped coming in. I
                                                noticed the default
                                                interface was still
                                                "st1" despite wifi
                                                having been disconnected
                                                and modem connected. The
                                                DNS was also still
                                                configured for my wifi
                                                network, and the
                                                interface seemed to have
                                                an IP address -- but
                                                wasn't pingable from the
                                                wifi network.<br
                                                  class="">
                                                <br class="">
                                                A power cycle of the
                                                modem solved the issue
                                                without reboot. So the
                                                cause may be in the
                                                modem/ppp subsystem, or
                                                it may be related (in
                                                some weird way) to the
                                                default interface / DNS
                                                setup.<br class="">
                                                <br class="">
                                                More tests showed the
                                                default interface
                                                again/still got set by
                                                the wifi blob itself at
                                                some point, overriding
                                                our modem
                                                prioritization. The
                                                events we didn't handle
                                                up to now were
                                                "sta.connected" and
                                                "sta.lostip", so I added
                                                these, and the bug
                                                didn't show up again
                                                since then. That doesn't
                                                mean anything, so we
                                                need to test this.<br
                                                  class="">
                                                <br class="">
                                                The default interface
                                                really shouldn't affect
                                                inbound packet routing
                                                of an established
                                                connection, but there
                                                always may be strange
                                                bugs lurking in those
                                                libs.<br class="">
                                                <br class="">
                                                The change also
                                                reimplements the wifi
                                                signal strength reading,
                                                as the tests also showed
                                                that still wasn't
                                                working well using the
                                                CSI callback. It now
                                                seems to be much more
                                                reliable.<br class="">
                                                <br class="">
                                                Please test &
                                                report. The single
                                                module will be hard to
                                                test, as the bug isn't
                                                reproducable easily, but
                                                you can still try if
                                                wifi / modem transitions
                                                work well.<br class="">
                                                <br class="">
                                                Mark, you can check your
                                                server logs for history
                                                messages with ridiculous
                                                time offsets:<br
                                                  class="">
                                                <blockquote class=""><tt
                                                    class="">[sddexter@ns27
                                                    server]$ cat
                                                    log-20190903 | egrep
                                                    "rx msg h
                                                    [0-9]+,-[0-9]{4}" |
                                                    wc -l<br class="">
                                                    455283</tt><br
                                                    class="">
                                                </blockquote>
                                                The bug now severely
                                                affects the V2 server
                                                performance, as the
                                                server is single
                                                threaded and doesn't
                                                scale very well to this
                                                kind of bulk data
                                                bursts, especially when
                                                coming from multiple
                                                modules in parallel. So
                                                we really need to solve
                                                this now. Slow reactions
                                                or connection drops from
                                                my server lately have
                                                been due to this bug. If
                                                this change doesn't
                                                solve it, we'll need to
                                                add some reboot trigger
                                                on "too many server v2
                                                notification
                                                retransmissions" -- or
                                                maybe a modem power
                                                cycle will do, that
                                                wouldn't discard the
                                                data.<br class="">
                                                <br class="">
                                                Thanks,<br class="">
                                                Michael<br class="">
                                                <br class="">
                                                <br class="">
                                                <div
                                                  class="moz-cite-prefix">Am
                                                  03.09.19 um 07:46
                                                  schrieb Mark
                                                  Webb-Johnson:<br
                                                    class="">
                                                </div>
                                                <blockquote type="cite"
cite="mid:7A81965B-AD97-4DFC-9768-9A03A74107BA@webb-johnson.net"
                                                  class="">
                                                  <pre class="moz-quote-pre" wrap="">No problem. We can hold. I won’t commit anything for the next few days (and agree to hold-off on Markos’s pull). Let me know when you are ready.

Regards, Mark.

</pre>
                                                  <blockquote
                                                    type="cite" class="">
                                                    <pre class="moz-quote-pre" wrap="">On 3 Sep 2019, at 1:58 AM, Michael Balzer <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><dexter@expeedo.de></a> wrote:

Mark, please wait.

I may just have found the cause for issue #241, or at least something I need to investigate before releasing.

I need to dig into my logs first, and try something.

Regards,
Michael


Am 02.09.19 um 12:23 schrieb Michael Balzer:
</pre>
                                                    <blockquote
                                                      type="cite"
                                                      class="">
                                                      <pre class="moz-quote-pre" wrap="">Nothing open from my side at the moment.

I haven't had the time to look in to Markos pull request, but from a first check also think that's going too deep to be included in this release.

Regards,
Michael


Am 02.09.19 um 04:15 schrieb Mark Webb-Johnson:
</pre>
                                                      <blockquote
                                                        type="cite"
                                                        class="">
                                                        <pre class="moz-quote-pre" wrap="">I think it is well past time for a 3.2.003 release. Things seems table in edge (although some things only partially implemented).

Anything people want to include at the last minute, or can we go ahead and build?

Regards, Mark.

_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                                      </blockquote>
                                                    </blockquote>
                                                    <pre class="moz-quote-pre" wrap="">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                                  </blockquote>
                                                  <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                                </blockquote>
                                                <br class="">
                                                <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                                              </div>
_______________________________________________<br class="">
                                              OvmsDev mailing list<br
                                                class="">
                                              <a
                                                href="mailto:OvmsDev@lists.openvehicles.com"
                                                class=""
                                                moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                                class="">
                                              <a
                                                class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                                moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                                class="">
                                            </div>
                                          </blockquote>
                                        </div>
                                        <br class="">
                                      </div>
                                      <br class="">
                                      <fieldset
                                        class="mimeAttachmentHeader"></fieldset>
                                      <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                    </blockquote>
                                    <br class="">
                                    <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                                    <br class="">
                                    <fieldset
                                      class="mimeAttachmentHeader"></fieldset>
                                    <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                  </blockquote>
                                  <br class="">
                                  <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                                  <br class="">
                                  <fieldset class="mimeAttachmentHeader"></fieldset>
                                  <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                </blockquote>
                                <br class="">
                                <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                                <br class="">
                                <fieldset class="mimeAttachmentHeader"></fieldset>
                                <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                              </blockquote>
                              <br class="">
                              <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                            </div>
_______________________________________________<br class="">
                            OvmsDev mailing list<br class="">
                            <a
                              href="mailto:OvmsDev@lists.openvehicles.com"
                              class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                              class="">
                            <a class="moz-txt-link-freetext"
                              href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                              moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                              class="">
                          </div>
                        </blockquote>
                      </div>
                      <br class="">
                    </div>
                    <br class="">
                    <fieldset class="mimeAttachmentHeader"></fieldset>
                    <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                  </blockquote>
                  <br class="">
                  <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                  <br class="">
                  <fieldset class="mimeAttachmentHeader"></fieldset>
                  <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                </blockquote>
                <br class="">
                <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
              </div>
              _______________________________________________<br
                class="">
              OvmsDev mailing list<br class="">
              <a href="mailto:OvmsDev@lists.openvehicles.com" class=""
                moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                class="">
              <a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                class="">
            </div>
          </blockquote>
        </div>
        <br class="">
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
  </body>
</html>