<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    I could test the modem now: modem & PPP is basically working,
    but it doesn't deliver the mobile network DNS. That is clearly an
    issue of the new esp-idf, not the Hologram SIM, as it works
    perfectly with the old esp-idf.<br>
    <br>
    The network manager gets all signals as expected and tries to save
    & restore the modem DNS, but it's just 0.0.0.0 on every connect.<br>
    <br>
    When setting a manual DNS, modem connectivity & switching
    wifi/modem works.<br>
    <br>
    I haven't found a matching issue on github yet.<br>
    <br>
    Regards,<br>
    Michael<br>
    <br>
    <br>
    <div class="moz-cite-prefix">Am 23.01.19 um 12:35 schrieb Mark
      Webb-Johnson:<br>
    </div>
    <blockquote type="cite"
      cite="mid:BE2C4B43-8249-4440-B57B-E73973D1D653@webb-johnson.net">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      Great news (the bug fix, not the new wifi bug).
      <div class=""><br class="">
      </div>
      <div class="">Looking forward to seeing this finally arrive. The
        last annoying framework bug for me is this wifi unable to
        reconnect issue.</div>
      <div class=""><br class="">
      </div>
      <div class="">Regards, Mark.<br class="">
        <div><br class="">
          <blockquote type="cite" class="">
            <div class="">On 23 Jan 2019, at 4:17 AM, Michael Balzer
              <<a href="mailto:dexter@expeedo.de" class=""
                moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
            <br class="Apple-interchange-newline">
            <div class="">
              <meta http-equiv="Content-Type" content="text/html;
                charset=UTF-8" class="">
              <div text="#000000" bgcolor="#FFFFFF" class=""> Wifi AP
                mode is broken. A fix should be available in some days:<br
                  class="">
                <a class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/issues/2915#issuecomment-452606571"
                  moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2915#issuecomment-452606571</a><br
                  class="">
                <br class="">
                I haven't been able to test the modem yet. It seems to
                work but my test SIM ran out of balance, so I need to
                wait for the payment process to finish.<br class="">
                <br class="">
                Apart from that it works pretty well so far.<br class="">
                <br class="">
                <br class="">
                <div class="moz-cite-prefix">Am 22.01.19 um 17:43
                  schrieb Michael Balzer:<br class="">
                </div>
                <blockquote type="cite"
                  cite="mid:8483fdbb-36cc-6bf6-0f2f-ab07600fa9cd@expeedo.de"
                  class="">
                  <meta http-equiv="Content-Type" content="text/html;
                    charset=UTF-8" class="">
                  Dmitry found out we had a bad timing with our merge.
                  The bug was fixed by him just five days later:<br
                    class="">
                  <br class="">
                  <a class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/commit/82eca97300410dd743de7f116836879cf3cdc6ba"
                    moz-do-not-send="true">https://github.com/espressif/esp-idf/commit/82eca97300410dd743de7f116836879cf3cdc6ba</a><br
                    class="">
                  <br class="">
                  I have merged the latest upstream HEAD locally now. It
                  builds correctly, with just one little issue in the
                  Kconfig for Smart ED, and seems to work correctly as
                  well. I'll do some testing now.<br class="">
                  <br class="">
                  Regards,<br class="">
                  Michael<br class="">
                  <br class="">
                  <br class="">
                  <div class="moz-cite-prefix">Am 27.12.18 um 11:03
                    schrieb Michael Balzer:<br class="">
                  </div>
                  <blockquote type="cite"
                    cite="mid:f5e8a4fa-e23d-9a5d-1459-fa8053d74fd9@expeedo.de"
                    class="">
                    <meta http-equiv="Content-Type" content="text/html;
                      charset=UTF-8" class="">
                    Update: I managed to reproduce the bug, and Dmitry's
                    assumption is correct.<br class="">
                    <br class="">
                    The wear leveling system attempts to do an update
                    when mounting a partition formatted with the old
                    version. That update fails and leads to the next
                    reboot failing to mount the partition.<br class="">
                    <br class="">
                    So the WL update process needs to be fixed.<br
                      class="">
                    <br class="">
                    Regards,<br class="">
                    Michael<br class="">
                    <br class="">
                    <br class="">
                    <div class="moz-cite-prefix">Am 27.11.18 um 22:35
                      schrieb Michael Balzer:<br class="">
                    </div>
                    <blockquote type="cite"
                      cite="mid:b31dd300-d7b3-0850-295d-19f7e34eb736@expeedo.de"
                      class="">
                      <meta http-equiv="Content-Type"
                        content="text/html; charset=UTF-8" class="">
                      @all: we need some help on this.<br class="">
                      <br class="">
                      Dmitry has asked us to verify an assumption about
                      the source of the problem. Unfortunately, my
                      module has decided to not show the bug anymore.<br
                        class="">
                      <br class="">
                      Can anyone a) reproduce the bug and b) check
                      Dmitry's code change? The code change is very
                      simple:<br class="">
                      <blockquote class=""><a
                          class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779"
                          moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779</a><br
                          class="">
                      </blockquote>
                      …that's in file <tt class="">esp-idf/components/wear_levelling/WL_Flash.cpp</tt>
                      line 183.<br class="">
                      <br class="">
                      Chances are the bug is more frequently present on
                      V3.1 first batch modules.<br class="">
                      <br class="">
                      <u class="">To test a build including the bug, do</u>:<br
                        class="">
                      <blockquote class=""><tt class="">OVMS# config
                          backup /sd/cfgbackup.zip</tt><br class="">
                        <br class="">
                        <tt class="">~/…/esp-idf> git checkout
                          5afc1b0cbbcd5903744cd2948ee235b7016eeede</tt><br
                          class="">
                        <tt class=""> ~/…/esp-idf> git merge
                          --no-commit
                          9d609af54c63e7f949a4fbc43d4f1c13b57f49d8</tt><br
                          class="">
                        <tt class=""> ~/…/esp-idf> git submodule
                          update --recursive</tt><br class="">
                        <tt class=""> </tt><br class="">
                        <tt class="">~/…/OVMS.V3> git checkout
                          c3c908c00a09c4a841d1fec115be785c8f486270</tt><br
                          class="">
                        <tt class="">~/…/OVMS.V3> make app-flash</tt><br
                          class="">
                      </blockquote>
                      Then do some reboots of the module.<br class="">
                      <br class="">
                      If the bug shows up, usually the second reboot
                      already will result in a lost configuration.
                      You'll see some warnings in the boot process
                      (mount failure), and the prompt will directly be
                      "#" as no module password is in place.<br class="">
                      <br class="">
                      If that happens, rebuild after applying the change
                      as described by Dmitry, restore your config (see
                      below), check if the bug is gone. If it isn't, the
                      change <u class="">definitely doesn't work</u>.<br
                        class="">
                      <br class="">
                      If it's gone, as the bug can stop showing up, you
                      need to revert Dmitry's change and check once
                      again if the bug reappears. If it does, the change
                      <u class="">definitely works</u>.<br class="">
                      <br class="">
                      If the bug doesn't reappear, you need to switch
                      off the module for some hours, then redo the above
                      steps.<br class="">
                      <br class="">
                      <br class="">
                      <u class="">Note on restoring the config backup</u>:
                      as my later restore fix isn't included in that
                      version, you may need to create the events
                      directory manually:<br class="">
                      <blockquote class=""><tt class="">OVMS# vfs mkdir
                          /store/events</tt><tt class=""><br class="">
                        </tt><tt class=""> OVMS# config restore
                          /sd/cfgbackup.zip <password></tt><tt
                          class=""><br class="">
                        </tt></blockquote>
                      To return to normal build setup:<br class="">
                      <blockquote class=""><tt class="">~/…/esp-idf>
                          git checkout -f master</tt><tt class=""><br
                            class="">
                        </tt><tt class=""> ~/…/esp-idf> git submodule
                          update --recursive</tt><tt class=""><br
                            class="">
                        </tt><tt class=""><br class="">
                        </tt><tt class="">~/…/OVMS.V3> git checkout
                          master</tt><br class="">
                      </blockquote>
                      <br class="">
                      Please report definitive results directly to
                      Dmitry in the issue #2730.<br class="">
                      <br class="">
                      Thanks,<br class="">
                      Michael<br class="">
                      <br class="">
                      <br class="">
                      <div class="moz-cite-prefix">Am 27.11.18 um 02:02
                        schrieb Mark Webb-Johnson:<br class="">
                      </div>
                      <blockquote type="cite"
                        cite="mid:5C2B0ED3-BFA4-4629-9892-2D1B50B65EF4@webb-johnson.net"
                        class="">
                        <meta http-equiv="Content-Type"
                          content="text/html; charset=UTF-8" class="">
                        I think unmount should be on the system.shutdown
                        event. That would be after system.shuttingdown
                        and the MyBoot.RestartPending/RestartReady
                        control steps.
                        <div class=""><br class="">
                        </div>
                        <div class="">That means if a component really
                          wants to write to config (or elsewhere on
                          /store), it should do it during
                          system.shuttingdown event (and optionally do
                          the MyBoot.RestartPending(TAG)
                          and MyBoot.RestartReady(TAG) thing, like
                          simcom).</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">As you say, that doesn’t address
                          the crash-during-shuttingdown issue, but can
                          be done in a very component manner. No changes
                          to MyBoot - all done in ovms_config. Or just
                          hard-code the /store unmount before
                          esp_restart().</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">Regards, Mark.<br class="">
                          <div class=""><br class="">
                            <blockquote type="cite" class="">
                              <div class="">On 27 Nov 2018, at 12:47 AM,
                                Michael Balzer <<a
                                  href="mailto:dexter@expeedo.de"
                                  class="" moz-do-not-send="true">dexter@expeedo.de</a>>
                                wrote:</div>
                              <br class="Apple-interchange-newline">
                              <div class="">
                                <meta http-equiv="Content-Type"
                                  content="text/html; charset=UTF-8"
                                  class="">
                                <div text="#000000" bgcolor="#FFFFFF"
                                  class=""> Dimitris question about
                                  unclean shutdowns let me check our
                                  reboot code, and we actually do not
                                  take care of unmounting the config
                                  store.<br class="">
                                  <br class="">
                                  While my crash observations do not
                                  look like crash/clean makes a
                                  difference to this bug, I still think
                                  we should do that.<br class="">
                                  <br class="">
                                  The simple solution would be to do
                                  this last, in
                                  Boot::boot_shutdown_done() just before
                                  the esp_restart(), but that does not
                                  take care of crashes during shutdown,
                                  which still are too frequent.<br
                                    class="">
                                  <br class="">
                                  How about unmounting /store ASAP after
                                  the shuttingdown event (i.e. normal
                                  shutdown handling)?<br class="">
                                  Writing to the config would then have
                                  no effect during shutdown, but do we
                                  actually need (want) that to be
                                  possible?<br class="">
                                  <br class="">
                                  Regards,<br class="">
                                  Michael<br class="">
                                  <br class="">
                                  <br class="">
                                  <div class="moz-cite-prefix">Am
                                    25.11.18 um 03:12 schrieb Mark
                                    Webb-Johnson:<br class="">
                                  </div>
                                  <blockquote type="cite"
                                    cite="mid:AD733748-1B40-447A-BC16-32B037A7227A@webb-johnson.net"
                                    class="">
                                    <meta http-equiv="Content-Type"
                                      content="text/html; charset=UTF-8"
                                      class="">
                                    I second this. A fantastic effort,
                                    Michael.
                                    <div class=""><br class="">
                                    </div>
                                    <div class="">The log you provided
                                      on the GitHub issue looks really
                                      helpful, and I see Dimitri has
                                      replied:</div>
                                    <div class=""><br class="">
                                    </div>
                                    <blockquote style="margin: 0 0 0
                                      40px;&#xA;&#xA; border:&#xA;
                                      none;&#xA; padding:&#xA; 0px;"
                                      class="">
                                      <div class="">
                                        <div class=" clearfix&#xA;
                                          timeline-comment-header"
                                          style="box-sizing:
                                          border-box;&#xA;&#xA;&#xA;
                                          background-color:&#xA;
                                          rgb(246,&#xA;&#xA;&#xA;
                                          248,&#xA; 250);&#xA;
                                          border-bottom-width:&#xA;&#xA;
                                          1px;&#xA;&#xA;
                                          border-bottom-style:&#xA;&#xA;&#xA;
                                          solid;&#xA;&#xA;
                                          border-bottom-color:&#xA;&#xA;
                                          rgb(209, 213,&#xA;
                                          218);&#xA;&#xA;&#xA;&#xA;
                                          border-top-left-radius:&#xA;&#xA;
                                          3px;&#xA;&#xA;
                                          border-top-right-radius:&#xA;
                                          3px;&#xA;&#xA;&#xA;
                                          color:&#xA; rgb(88,&#xA; 96,
                                          105);&#xA;
                                          padding-left:&#xA;&#xA;
                                          15px;&#xA;&#xA;
                                          padding-right:&#xA;&#xA;
                                          15px;&#xA;&#xA;
                                          font-family:&#xA;&#xA;
                                          -apple-system,&#xA;&#xA;&#xA;
BlinkMacSystemFont,&#xA;&#xA;&#xA;&#xA; "Segoe&#xA; UI",
                                          Helvetica,&#xA;&#xA;
                                          Arial,&#xA;&#xA;
                                          sans-serif,&#xA;&#xA;&#xA;
                                          "Apple Color&#xA;&#xA;
                                          Emoji",&#xA;&#xA;
                                          "Segoe UI&#xA;&#xA;
                                          Emoji",&#xA;&#xA;
                                          "Segoe&#xA; UI&#xA;&#xA;
                                          Symbol";&#xA;
                                          font-size:&#xA; 14px;">
                                          <h3 class="
                                            timeline-comment-header-text&#xA;&#xA;&#xA;&#xA;
                                            text-normal f5"
                                            style="box-sizing:&#xA;&#xA;&#xA;
                                            border-box;&#xA;
                                            margin-bottom:&#xA;&#xA;
                                            0px;&#xA;&#xA;
                                            margin-top:&#xA;
                                            0px;&#xA;&#xA; font-size:
                                            14px&#xA;&#xA;
                                            !important;&#xA;
                                            font-weight:&#xA;&#xA;&#xA;
                                            400&#xA;&#xA; !important;
                                            max-width:&#xA;
                                            78%;&#xA;&#xA;&#xA;
                                            padding-bottom:&#xA;
                                            10px;&#xA;&#xA;
                                            padding-top:&#xA; 10px;"><span
                                              class="css-truncate"
                                              style="box-sizing:&#xA;&#xA;
                                              border-box;&#xA;&#xA;
                                              font-weight:&#xA; 600;"><a
                                                class=" text-inherit
                                                author&#xA;&#xA;
                                                css-truncate-target&#xA;&#xA;"
data-hovercard-type="user"
                                                data-hovercard-url="/hovercards?user_id=32667452"
data-octo-click="hovercard-link-click"
                                                data-octo-dimensions="link_type:self"
href="https://github.com/dmitry1945"
                                                aria-describedby="hovercard-aria-description"
                                                style="box-sizing:&#xA;
                                                border-box;&#xA;&#xA;
                                                color:&#xA;&#xA; rgb(88,
                                                96,&#xA;&#xA;
                                                105);&#xA;&#xA;
                                                text-decoration:&#xA;
                                                none;&#xA;&#xA;
                                                display:&#xA;&#xA;
                                                inline-block;&#xA;&#xA;&#xA;&#xA;
                                                max-width:&#xA;
                                                125px;&#xA;
                                                overflow:&#xA;
                                                hidden;&#xA;&#xA;&#xA;
                                                text-overflow:&#xA;&#xA;&#xA;
                                                ellipsis;&#xA;&#xA;&#xA;
                                                vertical-align:&#xA;
                                                top;&#xA;
                                                white-space:&#xA;
                                                nowrap;"
                                                moz-do-not-send="true">dmitry1945</a> </span>commented <a
href="https://github.com/espressif/esp-idf#issuecomment-441391665"
                                              id="issuecomment-441391665-permalink"
                                              class="js-timestamp
                                              timestamp"
                                              style="box-sizing:&#xA;&#xA;
                                              border-box;&#xA;&#xA;
                                              color:&#xA;&#xA;
                                              inherit;&#xA;
                                              text-decoration:&#xA;&#xA;
                                              none;&#xA;&#xA;&#xA;
                                              white-space:&#xA; nowrap;"
                                              moz-do-not-send="true">6
                                              hours ago</a><span
                                              class="js-comment-fragment"
                                              style="box-sizing:&#xA;
                                              border-box;"><include-fragment
                                                class=" d-inline
                                                js-comment-edit-history&#xA;&#xA;"
style="box-sizing:&#xA;&#xA; border-box;&#xA; display:&#xA;&#xA;&#xA;
                                                inline&#xA; !important;"></include-fragment></span></h3>
                                        </div>
                                        <div class="edit-comment-hide"
                                          style="box-sizing:&#xA;&#xA;&#xA;
                                          border-box;&#xA;&#xA;
                                          caret-color:&#xA; rgb(36,&#xA;
                                          41, 46);&#xA; color:&#xA;&#xA;
                                          rgb(36, 41,&#xA;
                                          46);&#xA;&#xA;
                                          font-family:&#xA;&#xA;&#xA;
                                          -apple-system,&#xA;
                                          BlinkMacSystemFont,&#xA;&#xA;&#xA;&#xA;
                                          "Segoe&#xA;
                                          UI",&#xA;
                                          Helvetica,&#xA;&#xA;
                                          Arial,&#xA;&#xA; sans-serif,
                                          "Apple&#xA;&#xA;
                                          Color&#xA;&#xA;
                                          Emoji",&#xA;&#xA;&#xA;
                                          "Segoe UI
                                          Emoji",&#xA;&#xA;
                                          "Segoe&#xA;
                                          UI&#xA;&#xA;&#xA;
                                          Symbol";&#xA;
                                          font-size:&#xA; 14px;&#xA;
                                          background-color:&#xA;&#xA;&#xA;&#xA;
                                          rgb(255,&#xA; 255, 255);"><task-lists
                                            disabled="" sortable=""
                                            style="box-sizing:&#xA;
                                            border-box;" class="">
                                            <table class="d-block"
                                              style="box-sizing:&#xA;&#xA;&#xA;
                                              border-box;&#xA;&#xA;&#xA;
                                              border-collapse:&#xA;&#xA;
                                              collapse;&#xA;&#xA;
                                              border-spacing:&#xA;
                                              0px;&#xA;
                                              display:&#xA;&#xA;&#xA;
                                              block&#xA;&#xA;
                                              !important;">
                                              <tbody class="d-block"
                                                style="box-sizing:&#xA;&#xA;&#xA;
                                                border-box;&#xA;&#xA;
                                                display:&#xA;&#xA;
                                                block&#xA; !important;">
                                                <tr class="d-block"
                                                  style="box-sizing:&#xA;&#xA;&#xA;&#xA;
                                                  border-box;&#xA;
                                                  display:&#xA;&#xA;
                                                  block&#xA;&#xA;
                                                  !important;">
                                                  <td class="
                                                    comment-body&#xA;&#xA;
d-block&#xA;&#xA;&#xA;&#xA; &#xA; markdown-body&#xA;&#xA;&#xA;
                                                    js-comment-body"
                                                    style="box-sizing:&#xA;&#xA;&#xA;
border-box;&#xA;&#xA;&#xA; padding:&#xA; 15px;&#xA;&#xA; font-size:
                                                    14px;&#xA;&#xA;&#xA;
                                                    line-height:&#xA;
                                                    1.5;&#xA;&#xA;&#xA;
word-wrap:&#xA;&#xA;&#xA; break-word;&#xA; overflow:&#xA;
                                                    visible;&#xA;&#xA;&#xA;&#xA;
                                                    width:&#xA;
                                                    698px;&#xA;
                                                    display:&#xA;&#xA;
                                                    block&#xA;&#xA;
                                                    !important;">
                                                    <div
                                                      style="box-sizing:&#xA;&#xA;&#xA;
border-box;&#xA;&#xA;&#xA;&#xA; margin-bottom:&#xA;&#xA; 0px&#xA;&#xA;
                                                      !important;&#xA;&#xA;
                                                      margin-top:&#xA;
                                                      0px&#xA;&#xA;
                                                      !important;"
                                                      class="">Thank
                                                      you <a
                                                        class="user-mention"
data-hovercard-type="user"
                                                        data-hovercard-url="/hovercards?user_id=2706753"
data-octo-click="hovercard-link-click"
                                                        data-octo-dimensions="link_type:self"
href="https://github.com/dexterbg"
                                                        aria-describedby="hovercard-aria-description"
style="box-sizing:&#xA;&#xA;&#xA; border-box; color:&#xA;&#xA;&#xA;
                                                        rgb(36,&#xA;&#xA;
                                                        41,&#xA;&#xA;
                                                        46);&#xA;&#xA;
                                                        text-decoration:&#xA;&#xA;
none;&#xA;&#xA;&#xA;&#xA; font-weight:&#xA;&#xA;&#xA;
                                                        600;&#xA;&#xA;&#xA;
white-space:&#xA; nowrap;" moz-do-not-send="true">@dexterbg</a>, the
                                                      place is clear.</div>
                                                  </td>
                                                </tr>
                                              </tbody>
                                            </table>
                                          </task-lists></div>
                                      </div>
                                    </blockquote>
                                    <div class="">
                                      <div class=""><br class="">
                                      </div>
                                      <div class="">Hopefully Dimitri
                                        can find the issue from here. So
                                        many little bugs in wifi and
                                        bluetooth stacks causing us
                                        random issues - it would be
                                        helpful to be able to update to
                                        the latest IDF.</div>
                                      <div class=""><br class="">
                                      </div>
                                      <div class="">Thanks, and Regards,</div>
                                      <div class="">Mark.</div>
                                      <div class=""><br class="">
                                        <blockquote type="cite" class="">
                                          <div class="">On 25 Nov 2018,
                                            at 3:22 AM, Stephen Casner
                                            <<a
                                              href="mailto:casner@acm.org"
                                              class=""
                                              moz-do-not-send="true">casner@acm.org</a>>
                                            wrote:</div>
                                          <br
                                            class="Apple-interchange-newline">
                                          <div class="">
                                            <div class="">Michael --
                                              Kudos for this Herculean
                                              effort -- Steve<br
                                                class="">
                                              <br class="">
                                              On Sat, 24 Nov 2018,
                                              Michael Balzer wrote:<br
                                                class="">
                                              <br class="">
                                              <blockquote type="cite"
                                                class="">Narrowing this
                                                down is a real PITA. The
                                                effect sometimes stops
                                                to occur, I then need to
                                                leave the module powered
                                                down for a while to get
                                                the effect back.<br
                                                  class="">
                                                Sometimes 1 hour is
                                                sufficient, currently
                                                it's been off for 2
                                                hours and still works. I
                                                had a test window
                                                yesterday evening, one
                                                this morning and one in
                                                the<br class="">
                                                afternoon. Temperature
                                                is most probably
                                                irrelevant, as the last
                                                power downs were outside
                                                in ~4-5 °C to rule this
                                                out.<br class="">
                                                <br class="">
                                                So as a "passed" can
                                                always be a false
                                                positive I need to
                                                validate every passed
                                                step by switching back
                                                to a failing version and
                                                see if that still fails.<br
                                                  class="">
                                                <br class="">
                                                It seems I now have to
                                                wait until tomorrow for
                                                the next test window,
                                                but I have bisected down
                                                to this range of just 8
                                                commits:<br class="">
                                                <br class="">
balzer@leela:~/esp/esp-idf> git rev-list
                                                9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
^9d2f7c60d9aef9860c61c2756318ada68c80fddf<br class="">
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8<br class="">
f392727abf7d56490c2f33127a59bfac42c937e0<br class="">
e834d6fffc23a6fcfc0d2e871c9235417a7fb48f<br class="">
35842d02abb5f574aaab466d46081a232fdd20a6<br class="">
f05f3fbde87a9ce45c6818f71b49cd13888fd457<br class="">
a6d6c58ecadb9759a0bacf35cd7332ac641e598d<br class="">
321b1e02052de95db60ddce87eecce5f9e04e9b8<br class="">
40486c872345584d34949b3ce83f9e956a7eea13<br class="">
                                                <br class="">
                                                ...with
                                                9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
                                                being the last
                                                identified bad commit,
                                                and
                                                9d2f7c60d9aef9860c61c2756318ada68c80fddf
                                                being the last good.<br
                                                  class="">
                                                <br class="">
                                                If I should guess now,
                                                it's probably one of
                                                Dmitry's commits on the
                                                wear leveling code.<br
                                                  class="">
                                                <br class="">
                                                Regards,<br class="">
                                                Michael<br class="">
                                                <br class="">
                                                <br class="">
                                                Am 23.11.18 um 17:20
                                                schrieb Michael Balzer:<br
                                                  class="">
                                                <blockquote type="cite"
                                                  class="">It's not a
                                                  timing issue, I've let
                                                  it reboot about 30
                                                  times without any
                                                  successful mount after
                                                  the first failure.<br
                                                    class="">
                                                  <br class="">
                                                  Going into bisecting
                                                  now...<br class="">
                                                  <br class="">
                                                  <br class="">
                                                  Am 23.11.18 um 15:54
                                                  schrieb Mark
                                                  Webb-Johnson:<br
                                                    class="">
                                                  <blockquote
                                                    type="cite" class="">
                                                    <blockquote
                                                      type="cite"
                                                      class="">It may
                                                      actually not be a
                                                      corruption of the
                                                      filesystem but
                                                      some timing issue
                                                      on the mount
                                                      procedure. To test
                                                      that we could
                                                      disable the auto
                                                      formatting on<br
                                                        class="">
                                                      mount failures.<br
                                                        class="">
                                                    </blockquote>
                                                    <br class="">
                                                    True.<br class="">
                                                    <br class="">
                                                    A couple of
                                                    Espressif guys have
                                                    jumped on the issue,
                                                    and I have provided
                                                    some more
                                                    information for
                                                    them. I think key
                                                    will be reproducing
                                                    it.<br class="">
                                                    <br class="">
                                                    <blockquote
                                                      type="cite"
                                                      class="">The issue
                                                      may also be
                                                      dependant on the
                                                      hardware version,
                                                      i.e. it could be
                                                      caused by the bug
                                                      that caused the SD
                                                      speed issue on the
                                                      first 3.1 batch.<br
                                                        class="">
                                                    </blockquote>
                                                    <br class="">
                                                    That was definitely
                                                    a hardware issue
                                                    with the CP2102
                                                    chip. I don't think
                                                    related to ESP in
                                                    any way.<br class="">
                                                    <br class="">
                                                    Regards, Mark.<br
                                                      class="">
                                                    <br class="">
                                                    <blockquote
                                                      type="cite"
                                                      class="">On 23 Nov
                                                      2018, at 10:34 PM,
                                                      Michael Balzer
                                                      <<a
                                                        href="mailto:dexter@expeedo.de"
                                                        class=""
                                                        moz-do-not-send="true">dexter@expeedo.de</a>
                                                      <<a
                                                        href="mailto:dexter@expeedo.de"
                                                        class=""
                                                        moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
                                                      wrote:<br class="">
                                                      <br class="">
                                                      It may actually
                                                      not be a
                                                      corruption of the
                                                      filesystem but
                                                      some timing issue
                                                      on the mount
                                                      procedure. To test
                                                      that we could
                                                      disable the auto
                                                      formatting on<br
                                                        class="">
                                                      mount failures.<br
                                                        class="">
                                                      <br class="">
                                                      The issue may also
                                                      be dependant on
                                                      the hardware
                                                      version, i.e. it
                                                      could be caused by
                                                      the bug that
                                                      caused the SD
                                                      speed issue on the
                                                      first 3.1 batch.<br
                                                        class="">
                                                      <br class="">
                                                      I only have tried
                                                      the idf update on
                                                      my batch 1 module
                                                      (my bench /
                                                      development
                                                      module). I think
                                                      most of our edge
                                                      testers also have
                                                      that version.<br
                                                        class="">
                                                      <br class="">
                                                      Regards,<br
                                                        class="">
                                                      Michael<br
                                                        class="">
                                                      <br class="">
                                                      <br class="">
                                                      Am 23.11.18 um
                                                      02:32 schrieb Mark
                                                      Webb-Johnson:<br
                                                        class="">
                                                      <blockquote
                                                        type="cite"
                                                        class="">I have
                                                        raised the
                                                        following github
                                                        issue to
                                                        Espressif:<br
                                                          class="">
                                                        <br class="">
                                                           <a
                                                          href="https://github.com/espressif/esp-idf/issues/2730"
                                                          class=""
                                                          moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730</a><br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
       Environment<br class="">
                                                        <br class="">
                                                             *
                                                        Development Kit:
                                                        none<br class="">
                                                             * Kit
                                                        version (for
                                                        WroverKit/PicoKit/DevKitC):
                                                        none<br class="">
                                                             * Module or
                                                        chip used:
                                                        ESP32-WROVER
                                                        16MB<br class="">
                                                             * IDF
                                                        version (run
                                                        |git describe
                                                        --tags| to find
                                                        it):
                                                        v3.2-beta1-208-g0d7f2d77c<br
                                                          class="">
                                                             * Build
                                                        System: make<br
                                                          class="">
                                                             * Compiler
                                                        version (run
                                                        |xtensa-esp32-elf-gcc
                                                        --version| to
                                                        find it):
                                                        (crosstool-NG
                                                        crosstool-ng-1.22.0-80-g6c4433a)
                                                        5.2.0<br
                                                          class="">
                                                             * Operating
                                                        System: macOS<br
                                                          class="">
                                                             * Power
                                                        Supply: USB<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
                                                               Problem
                                                        Description<br
                                                          class="">
                                                        <br class="">
                                                           TLDR: Between
                                                        May and July
                                                        2018 a change
                                                        was made to esp
                                                        idf master that
                                                        is causing
                                                        corruption on
                                                        FAT filesystems
                                                        mounted on SPI
                                                        flash.<br
                                                          class="">
                                                        <br class="">
                                                           Our project
                                                        uses a
                                                        partitions.csv
                                                        as follows:<br
                                                          class="">
                                                        <br class="">
                                                           |# Name,
                                                        Type, SubType,
                                                        Offset, Size
                                                        nvs, data, nvs,
                                                        0x9000, 0x4000
                                                        otadata, data,
                                                        ota, 0xd000,
                                                        0x2000 phy_init,
                                                        data, phy,
                                                        0xf000, 0x1000
                                                        factory,<br
                                                          class="">
                                                           app, factory,
                                                        0x10000, 4M
                                                        ota_0, app,
                                                        ota_0, , 4M
                                                        ota_1, app,
                                                        ota_1, , 4M
                                                        store, data,
                                                        fat, , 1M |<br
                                                          class="">
                                                        <br class="">
                                                           The 'store'
                                                        partition is
                                                        formatted as
                                                        FAT, as follows:<br
                                                          class="">
                                                        <br class="">
   esp_vfs_fat_mount_config_t m_store_fat;<br class="">
                                                           wl_handle_t
                                                        m_store_wlh;<br
                                                          class="">
   memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
                                                          class="">
   m_store_fat.format_if_mount_failed = true;<br class="">
   m_store_fat.max_files = 5;<br class="">
   esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
                                                        &m_store_wlh);<br
                                                          class="">
                                                        <br class="">
                                                           We have
                                                        previously used
                                                        a clone of esp
                                                        idf master,
                                                        dated around May
                                                        22 2018, without
                                                        issues. The
                                                        partition is
                                                        very reliable.<br
                                                          class="">
                                                        <br class="">
                                                           However, on
                                                        Jul 6 2018, we
                                                        updated our
                                                        clone to use the
                                                        latest esp idf
                                                        master at that
                                                        time. Shortly
                                                        afterwards,
                                                        users started to
                                                        report that
                                                        their<br
                                                          class="">
                                                           'store'
                                                        filesystem
                                                        contents were
                                                        corrupted. We
                                                        rolled back.<br
                                                          class="">
                                                        <br class="">
                                                           We have now
                                                        tried again
                                                        (updating on Oct
                                                        20 2018 to
                                                        v3.2-beta1-208-g0d7f2d77c)
                                                        and immediately
                                                        had the same
                                                        issue. Random
                                                        corruption of
                                                        FAT filesystem<br
                                                          class="">
                                                           in SPI flash.<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
         Expected Behavior<br class="">
                                                        <br class="">
                                                           No corruption
                                                        of FAT
                                                        filesystem.<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
                                                                 Actual
                                                        Behavior<br
                                                          class="">
                                                        <br class="">
                                                           Corruption of
                                                        FAT filesystem.<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
                                                                 Steps
                                                        to reproduce<br
                                                          class="">
                                                        <br class="">
                                                            1. Create a
                                                        partition in SPI
                                                        flash, and mount
                                                        FAT filesystem<br
                                                          class="">
                                                            2. Read and
                                                        write to files
                                                        on FAT
                                                        filesystem<br
                                                          class="">
                                                            3. Reboot<br
                                                          class="">
                                                            4. Observe
                                                        random
                                                        corruption and
                                                        unmountable
                                                        filesystem<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
                                                                 Code to
                                                        reproduce this
                                                        issue<br
                                                          class="">
                                                        <br class="">
   esp_vfs_fat_mount_config_t m_store_fat;<br class="">
                                                           wl_handle_t
                                                        m_store_wlh;<br
                                                          class="">
   memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
                                                          class="">
   m_store_fat.format_if_mount_failed = true;<br class="">
   m_store_fat.max_files = 5;<br class="">
   esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
                                                        &m_store_wlh);<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
                                                               Debug
                                                        Logs<br class="">
                                                        <br class="">
                                                           n/a<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
                                                               Other
                                                        items if
                                                        possible<br
                                                          class="">
                                                        <br class="">
                                                           Please advise
                                                        if you need
                                                        anything
                                                        further.<br
                                                          class="">
                                                        <br class="">
                                                        <br class="">
                                                        I think the
                                                        timeline is
                                                        correct (the
                                                        issue is in esp
                                                        idf master some
                                                        time between May
                                                        and July 2018),
                                                        but please let
                                                        me know if you
                                                        know differently
                                                        (or<br class="">
                                                        update the
                                                        github issue
                                                        with your
                                                        comments).<br
                                                          class="">
                                                        <br class="">
                                                        Regards, Mark<br
                                                          class="">
                                                        <br class="">
                                                        <blockquote
                                                          type="cite"
                                                          class="">On 23
                                                          Nov 2018, at
                                                          6:19 AM,
                                                          Michael Balzer
                                                          <<a
                                                          href="mailto:dexter@expeedo.de"
                                                          class=""
                                                          moz-do-not-send="true">dexter@expeedo.de</a>
                                                          <<a
                                                          href="mailto:dexter@expeedo.de"
                                                          class=""
                                                          moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
                                                          wrote:<br
                                                          class="">
                                                          <br class="">
                                                          esp-idf and
                                                          OVMS branches
                                                          are back to
                                                          the working
                                                          version.<br
                                                          class="">
                                                          <br class="">
                                                          In case you
                                                          also lost your
                                                          config:  I
                                                          also just
                                                          fixed a bug on
                                                          restoring into
                                                          an empty
                                                          /store
                                                          partition.<br
                                                          class="">
                                                          <br class="">
                                                          Regards,<br
                                                          class="">
                                                          Michael<br
                                                          class="">
                                                          <br class="">
                                                          <br class="">
                                                          Am 22.11.18 um
                                                          22:34 schrieb
                                                          Michael
                                                          Balzer:<br
                                                          class="">
                                                          <blockquote
                                                          type="cite"
                                                          class="">See <a
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165"
                                                          class=""
                                                          moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165</a><br
                                                          class="">
                                                          <br class="">
                                                          I'll reset
                                                          both master
                                                          branches now.<br
                                                          class="">
                                                          <br class="">
                                                          If you're
                                                          about to pull,
                                                          please wait
                                                          until I've
                                                          reverted the
                                                          branches.<br
                                                          class="">
                                                          <br class="">
                                                          Regards,<br
                                                          class="">
                                                          Michael<br
                                                          class="">
                                                          <br class="">
                                                          </blockquote>
                                                          <br class="">
                                                          --<br class="">
                                                          Michael Balzer
                                                          * Helkenberger
                                                          Weg 9 *
                                                          D-58256
                                                          Ennepetal<br
                                                          class="">
                                                          Fon 02333 /
                                                          833 5735 *
                                                          Handy 0176 /
                                                          206 989 26<br
                                                          class="">
                                                          <br class="">
_______________________________________________<br class="">
                                                          OvmsDev
                                                          mailing list<br
                                                          class="">
                                                          <a
                                                          href="mailto:OvmsDev@lists.openvehicles.com"
                                                          class=""
                                                          moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
                                                          <<a
                                                          href="mailto:OvmsDev@lists.openvehicles.com"
                                                          class=""
                                                          moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
                                                          class="">
                                                          <a
                                                          href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                                          class=""
                                                          moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                                          class="">
                                                        </blockquote>
                                                        <br class="">
                                                        <br class="">
_______________________________________________<br class="">
                                                        OvmsDev mailing
                                                        list<br class="">
                                                        <a
                                                          href="mailto:OvmsDev@lists.openvehicles.com"
                                                          class=""
                                                          moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                                          class="">
                                                        <a
                                                          class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                                          moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                                          class="">
                                                      </blockquote>
                                                      <br class="">
                                                      --<br class="">
                                                      Michael Balzer *
                                                      Helkenberger Weg 9
                                                      * D-58256
                                                      Ennepetal<br
                                                        class="">
                                                      Fon 02333 / 833
                                                      5735 * Handy 0176
                                                      / 206 989 26<br
                                                        class="">
_______________________________________________<br class="">
                                                      OvmsDev mailing
                                                      list<br class="">
                                                      <a
                                                        href="mailto:OvmsDev@lists.openvehicles.com"
                                                        class=""
                                                        moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
                                                      <<a
                                                        href="mailto:OvmsDev@lists.openvehicles.com"
                                                        class=""
                                                        moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
                                                        class="">
                                                      <a
                                                        href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                                        class=""
                                                        moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                                        class="">
                                                    </blockquote>
                                                    <br class="">
                                                    <br class="">
_______________________________________________<br class="">
                                                    OvmsDev mailing list<br
                                                      class="">
                                                    <a
                                                      href="mailto:OvmsDev@lists.openvehicles.com"
                                                      class=""
                                                      moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                                      class="">
                                                    <a
                                                      class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                                      moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                                      class="">
                                                  </blockquote>
                                                  <br class="">
                                                  --<br class="">
                                                  Michael Balzer *
                                                  Helkenberger Weg 9 *
                                                  D-58256 Ennepetal<br
                                                    class="">
                                                  Fon 02333 / 833 5735 *
                                                  Handy 0176 / 206 989
                                                  26<br class="">
                                                  <br class="">
_______________________________________________<br class="">
                                                  OvmsDev mailing list<br
                                                    class="">
                                                  <a
                                                    href="mailto:OvmsDev@lists.openvehicles.com"
                                                    class=""
                                                    moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                                    class="">
                                                  <a
                                                    class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                                    moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                                    class="">
                                                </blockquote>
                                                <br class="">
                                                --<br class="">
                                                Michael Balzer *
                                                Helkenberger Weg 9 *
                                                D-58256 Ennepetal<br
                                                  class="">
                                                Fon 02333 / 833 5735 *
                                                Handy 0176 / 206 989 26<br
                                                  class="">
                                                <br class="">
                                                <br class="">
                                              </blockquote>
                                              <br class="">
                                                       --
                                              Steve_______________________________________________<br
                                                class="">
                                              OvmsDev mailing list<br
                                                class="">
                                              <a
                                                href="mailto:OvmsDev@lists.openvehicles.com"
                                                class=""
                                                moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                                class="">
                                              <a
                                                class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                                moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                                class="">
                                            </div>
                                          </div>
                                        </blockquote>
                                      </div>
                                      <br class="">
                                    </div>
                                    <br class="">
                                    <fieldset
                                      class="mimeAttachmentHeader"></fieldset>
                                    <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                                  </blockquote>
                                  <br class="">
                                  <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                                </div>
_______________________________________________<br class="">
                                OvmsDev mailing list<br class="">
                                <a
                                  href="mailto:OvmsDev@lists.openvehicles.com"
                                  class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                  class="">
                                <a class="moz-txt-link-freetext"
                                  href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                  moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                  class="">
                              </div>
                            </blockquote>
                          </div>
                          <br class="">
                        </div>
                        <br class="">
                        <fieldset class="mimeAttachmentHeader"></fieldset>
                        <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                      </blockquote>
                      <br class="">
                      <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                      <br class="">
                      <fieldset class="mimeAttachmentHeader"></fieldset>
                      <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                    </blockquote>
                    <br class="">
                    <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                    <br class="">
                    <fieldset class="mimeAttachmentHeader"></fieldset>
                    <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                  </blockquote>
                  <br class="">
                  <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                  <br class="">
                  <fieldset class="mimeAttachmentHeader"></fieldset>
                  <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                </blockquote>
                <br class="">
                <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
              </div>
              _______________________________________________<br
                class="">
              OvmsDev mailing list<br class="">
              <a href="mailto:OvmsDev@lists.openvehicles.com" class=""
                moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                class="">
              <a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                class="">
            </div>
          </blockquote>
        </div>
        <br class="">
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
  </body>
</html>