<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    @all: we need some help on this.<br>
    <br>
    Dmitry has asked us to verify an assumption about the source of the
    problem. Unfortunately, my module has decided to not show the bug
    anymore.<br>
    <br>
    Can anyone a) reproduce the bug and b) check Dmitry's code change?
    The code change is very simple:<br>
    <blockquote><a class="moz-txt-link-freetext" href="https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779">https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779</a><br>
    </blockquote>
    …that's in file <tt>esp-idf/components/wear_levelling/WL_Flash.cpp</tt>
    line 183.<br>
    <br>
    Chances are the bug is more frequently present on V3.1 first batch
    modules.<br>
    <br>
    <u>To test a build including the bug, do</u>:<br>
    <blockquote><tt>OVMS# config backup /sd/cfgbackup.zip</tt><br>
      <br>
      <tt>~/…/esp-idf> git checkout
        5afc1b0cbbcd5903744cd2948ee235b7016eeede</tt><br>
      <tt>
        ~/…/esp-idf> git merge --no-commit
        9d609af54c63e7f949a4fbc43d4f1c13b57f49d8</tt><br>
      <tt>
        ~/…/esp-idf> git submodule update --recursive</tt><br>
      <tt>
      </tt><br>
      <tt>~/…/OVMS.V3> git checkout
        c3c908c00a09c4a841d1fec115be785c8f486270</tt><br>
      <tt>~/…/OVMS.V3> make app-flash</tt><br>
    </blockquote>
    Then do some reboots of the module.<br>
    <br>
    If the bug shows up, usually the second reboot already will result
    in a lost configuration. You'll see some warnings in the boot
    process (mount failure), and the prompt will directly be "#" as no
    module password is in place.<br>
    <br>
    If that happens, rebuild after applying the change as described by
    Dmitry, restore your config (see below), check if the bug is gone.
    If it isn't, the change <u>definitely doesn't work</u>.<br>
    <br>
    If it's gone, as the bug can stop showing up, you need to revert
    Dmitry's change and check once again if the bug reappears. If it
    does, the change <u>definitely works</u>.<br>
    <br>
    If the bug doesn't reappear, you need to switch off the module for
    some hours, then redo the above steps.<br>
    <br>
    <br>
    <u>Note on restoring the config backup</u>: as my later restore fix
    isn't included in that version, you may need to create the events
    directory manually:<br>
    <blockquote><tt>OVMS# vfs mkdir /store/events</tt><tt><br>
      </tt><tt>
        OVMS# config restore /sd/cfgbackup.zip <password></tt><tt><br>
      </tt></blockquote>
    To return to normal build setup:<br>
    <blockquote><tt>~/…/esp-idf> git checkout -f master</tt><tt><br>
      </tt><tt>
        ~/…/esp-idf> git submodule update --recursive</tt><tt><br>
      </tt><tt><br>
      </tt><tt>~/…/OVMS.V3> git checkout master</tt><br>
    </blockquote>
    <br>
    Please report definitive results directly to Dmitry in the issue
    #2730.<br>
    <br>
    Thanks,<br>
    Michael<br>
    <br>
    <br>
    <div class="moz-cite-prefix">Am 27.11.18 um 02:02 schrieb Mark
      Webb-Johnson:<br>
    </div>
    <blockquote type="cite"
      cite="mid:5C2B0ED3-BFA4-4629-9892-2D1B50B65EF4@webb-johnson.net">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      I think unmount should be on the system.shutdown event. That would
      be after system.shuttingdown and the
      MyBoot.RestartPending/RestartReady control steps.
      <div class=""><br class="">
      </div>
      <div class="">That means if a component really wants to write to
        config (or elsewhere on /store), it should do it during
        system.shuttingdown event (and optionally do
        the MyBoot.RestartPending(TAG) and MyBoot.RestartReady(TAG)
        thing, like simcom).</div>
      <div class=""><br class="">
      </div>
      <div class="">As you say, that doesn’t address the
        crash-during-shuttingdown issue, but can be done in a very
        component manner. No changes to MyBoot - all done in
        ovms_config. Or just hard-code the /store unmount before
        esp_restart().</div>
      <div class=""><br class="">
      </div>
      <div class="">Regards, Mark.<br class="">
        <div><br class="">
          <blockquote type="cite" class="">
            <div class="">On 27 Nov 2018, at 12:47 AM, Michael Balzer
              <<a href="mailto:dexter@expeedo.de" class=""
                moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
            <br class="Apple-interchange-newline">
            <div class="">
              <meta http-equiv="Content-Type" content="text/html;
                charset=UTF-8" class="">
              <div text="#000000" bgcolor="#FFFFFF" class=""> Dimitris
                question about unclean shutdowns let me check our reboot
                code, and we actually do not take care of unmounting the
                config store.<br class="">
                <br class="">
                While my crash observations do not look like crash/clean
                makes a difference to this bug, I still think we should
                do that.<br class="">
                <br class="">
                The simple solution would be to do this last, in
                Boot::boot_shutdown_done() just before the
                esp_restart(), but that does not take care of crashes
                during shutdown, which still are too frequent.<br
                  class="">
                <br class="">
                How about unmounting /store ASAP after the shuttingdown
                event (i.e. normal shutdown handling)?<br class="">
                Writing to the config would then have no effect during
                shutdown, but do we actually need (want) that to be
                possible?<br class="">
                <br class="">
                Regards,<br class="">
                Michael<br class="">
                <br class="">
                <br class="">
                <div class="moz-cite-prefix">Am 25.11.18 um 03:12
                  schrieb Mark Webb-Johnson:<br class="">
                </div>
                <blockquote type="cite"
                  cite="mid:AD733748-1B40-447A-BC16-32B037A7227A@webb-johnson.net"
                  class="">
                  <meta http-equiv="Content-Type" content="text/html;
                    charset=UTF-8" class="">
                  I second this. A fantastic effort, Michael.
                  <div class=""><br class="">
                  </div>
                  <div class="">The log you provided on the GitHub issue
                    looks really helpful, and I see Dimitri has replied:</div>
                  <div class=""><br class="">
                  </div>
                  <blockquote style="margin: 0 0 0 40px; border: none;
                    padding:&#xA; 0px;" class="">
                    <div class="">
                      <div class="clearfix timeline-comment-header"
                        style="box-sizing: border-box; background-color:
                        rgb(246,&#xA; 248, 250); border-bottom-width:
                        1px; border-bottom-style:&#xA; solid;
                        border-bottom-color: rgb(209, 213, 218);&#xA;
                        border-top-left-radius: 3px;
                        border-top-right-radius: 3px;&#xA; color:
                        rgb(88, 96, 105); padding-left: 15px;
                        padding-right:&#xA; 15px; font-family:
                        -apple-system, BlinkMacSystemFont,&#xA;
                        "Segoe UI", Helvetica, Arial,
                        sans-serif,&#xA; "Apple Color Emoji",
                        "Segoe UI Emoji",&#xA; "Segoe UI
                        Symbol"; font-size: 14px;">
                        <h3 class="timeline-comment-header-text
                          text-normal f5" style="box-sizing: border-box;
                          margin-bottom: 0px;&#xA; margin-top: 0px;
                          font-size: 14px !important; font-weight:&#xA;
                          400 !important; max-width: 78%;
                          padding-bottom: 10px;&#xA; padding-top: 10px;"><span
                            class="css-truncate" style="box-sizing:
                            border-box; font-weight: 600;"><a
                              class="css-truncate-target author
                              text-inherit" data-hovercard-type="user"
                              data-hovercard-url="/hovercards?user_id=32667452"
                              data-octo-click="hovercard-link-click"
                              data-octo-dimensions="link_type:self"
                              href="https://github.com/dmitry1945"
                              aria-describedby="hovercard-aria-description"
                              style="box-sizing: border-box; color:
                              rgb(88, 96,&#xA; 105); text-decoration:
                              none; display: inline-block;&#xA;
                              max-width: 125px; overflow: hidden;
                              text-overflow:&#xA; ellipsis;
                              vertical-align: top; white-space: nowrap;"
                              moz-do-not-send="true">dmitry1945</a> </span>commented <a
href="https://github.com/espressif/esp-idf#issuecomment-441391665"
                            id="issuecomment-441391665-permalink"
                            class="js-timestamp timestamp"
                            style="box-sizing:&#xA; border-box; color:
                            inherit; text-decoration: none;&#xA;
                            white-space: nowrap;" moz-do-not-send="true">6
                            hours ago</a><span
                            class="js-comment-fragment"
                            style="box-sizing:&#xA; border-box;"><include-fragment
                              class="js-comment-edit-history d-inline"
                              style="box-sizing: border-box; display:
                              inline&#xA; !important;"></include-fragment></span></h3>
                      </div>
                      <div class="edit-comment-hide" style="box-sizing:
                        border-box;&#xA; caret-color: rgb(36, 41, 46);
                        color: rgb(36, 41, 46);&#xA; font-family:
                        -apple-system, BlinkMacSystemFont,
                        "Segoe&#xA; UI", Helvetica, Arial,
                        sans-serif, "Apple Color&#xA; Emoji",
                        "Segoe UI Emoji", "Segoe UI&#xA;
                        Symbol"; font-size: 14px; background-color:
                        rgb(255,&#xA; 255, 255);"><task-lists
                          disabled="" sortable="" style="box-sizing:
                          border-box;" class="">
                          <table class="d-block" style="box-sizing:
                            border-box;&#xA; border-collapse: collapse;
                            border-spacing: 0px; display:&#xA; block
                            !important;">
                            <tbody class="d-block" style="box-sizing:
                              border-box;&#xA; display: block
                              !important;">
                              <tr class="d-block" style="box-sizing:
                                border-box;&#xA; display: block
                                !important;">
                                <td class=" markdown-body d-block&#xA;
                                  comment-body js-comment-body"
                                  style="box-sizing: border-box;&#xA;
                                  padding: 15px; font-size: 14px;
                                  line-height: 1.5;&#xA; word-wrap:
                                  break-word; overflow: visible;
                                  width:&#xA; 698px; display: block
                                  !important;">
                                  <div style="box-sizing: border-box;
                                    margin-bottom:&#xA; 0px !important;
                                    margin-top: 0px !important;"
                                    class="">Thank you <a
                                      class="user-mention"
                                      data-hovercard-type="user"
                                      data-hovercard-url="/hovercards?user_id=2706753"
data-octo-click="hovercard-link-click"
                                      data-octo-dimensions="link_type:self"
                                      href="https://github.com/dexterbg"
aria-describedby="hovercard-aria-description" style="box-sizing:
                                      border-box; color: rgb(36,&#xA;
                                      41, 46); text-decoration: none;
                                      font-weight:&#xA; 600;
                                      white-space: nowrap;"
                                      moz-do-not-send="true">@dexterbg</a>,
                                    the place is clear.</div>
                                </td>
                              </tr>
                            </tbody>
                          </table>
                        </task-lists></div>
                    </div>
                  </blockquote>
                  <div class="">
                    <div class=""><br class="">
                    </div>
                    <div class="">Hopefully Dimitri can find the issue
                      from here. So many little bugs in wifi and
                      bluetooth stacks causing us random issues - it
                      would be helpful to be able to update to the
                      latest IDF.</div>
                    <div class=""><br class="">
                    </div>
                    <div class="">Thanks, and Regards,</div>
                    <div class="">Mark.</div>
                    <div class=""><br class="">
                      <blockquote type="cite" class="">
                        <div class="">On 25 Nov 2018, at 3:22 AM,
                          Stephen Casner <<a
                            href="mailto:casner@acm.org" class=""
                            moz-do-not-send="true">casner@acm.org</a>>
                          wrote:</div>
                        <br class="Apple-interchange-newline">
                        <div class="">
                          <div class="">Michael -- Kudos for this
                            Herculean effort -- Steve<br class="">
                            <br class="">
                            On Sat, 24 Nov 2018, Michael Balzer wrote:<br
                              class="">
                            <br class="">
                            <blockquote type="cite" class="">Narrowing
                              this down is a real PITA. The effect
                              sometimes stops to occur, I then need to
                              leave the module powered down for a while
                              to get the effect back.<br class="">
                              Sometimes 1 hour is sufficient, currently
                              it's been off for 2 hours and still works.
                              I had a test window yesterday evening, one
                              this morning and one in the<br class="">
                              afternoon. Temperature is most probably
                              irrelevant, as the last power downs were
                              outside in ~4-5 °C to rule this out.<br
                                class="">
                              <br class="">
                              So as a "passed" can always be a false
                              positive I need to validate every passed
                              step by switching back to a failing
                              version and see if that still fails.<br
                                class="">
                              <br class="">
                              It seems I now have to wait until tomorrow
                              for the next test window, but I have
                              bisected down to this range of just 8
                              commits:<br class="">
                              <br class="">
                              balzer@leela:~/esp/esp-idf> git
                              rev-list
                              9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
                              ^9d2f7c60d9aef9860c61c2756318ada68c80fddf<br
                                class="">
                              9d609af54c63e7f949a4fbc43d4f1c13b57f49d8<br
                                class="">
                              f392727abf7d56490c2f33127a59bfac42c937e0<br
                                class="">
                              e834d6fffc23a6fcfc0d2e871c9235417a7fb48f<br
                                class="">
                              35842d02abb5f574aaab466d46081a232fdd20a6<br
                                class="">
                              f05f3fbde87a9ce45c6818f71b49cd13888fd457<br
                                class="">
                              a6d6c58ecadb9759a0bacf35cd7332ac641e598d<br
                                class="">
                              321b1e02052de95db60ddce87eecce5f9e04e9b8<br
                                class="">
                              40486c872345584d34949b3ce83f9e956a7eea13<br
                                class="">
                              <br class="">
                              ...with
                              9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
                              being the last identified bad commit, and
                              9d2f7c60d9aef9860c61c2756318ada68c80fddf
                              being the last good.<br class="">
                              <br class="">
                              If I should guess now, it's probably one
                              of Dmitry's commits on the wear leveling
                              code.<br class="">
                              <br class="">
                              Regards,<br class="">
                              Michael<br class="">
                              <br class="">
                              <br class="">
                              Am 23.11.18 um 17:20 schrieb Michael
                              Balzer:<br class="">
                              <blockquote type="cite" class="">It's not
                                a timing issue, I've let it reboot about
                                30 times without any successful mount
                                after the first failure.<br class="">
                                <br class="">
                                Going into bisecting now...<br class="">
                                <br class="">
                                <br class="">
                                Am 23.11.18 um 15:54 schrieb Mark
                                Webb-Johnson:<br class="">
                                <blockquote type="cite" class="">
                                  <blockquote type="cite" class="">It
                                    may actually not be a corruption of
                                    the filesystem but some timing issue
                                    on the mount procedure. To test that
                                    we could disable the auto formatting
                                    on<br class="">
                                    mount failures.<br class="">
                                  </blockquote>
                                  <br class="">
                                  True.<br class="">
                                  <br class="">
                                  A couple of Espressif guys have jumped
                                  on the issue, and I have provided some
                                  more information for them. I think key
                                  will be reproducing it.<br class="">
                                  <br class="">
                                  <blockquote type="cite" class="">The
                                    issue may also be dependant on the
                                    hardware version, i.e. it could be
                                    caused by the bug that caused the SD
                                    speed issue on the first 3.1 batch.<br
                                      class="">
                                  </blockquote>
                                  <br class="">
                                  That was definitely a hardware issue
                                  with the CP2102 chip. I don't think
                                  related to ESP in any way.<br class="">
                                  <br class="">
                                  Regards, Mark.<br class="">
                                  <br class="">
                                  <blockquote type="cite" class="">On 23
                                    Nov 2018, at 10:34 PM, Michael
                                    Balzer <<a
                                      href="mailto:dexter@expeedo.de"
                                      class="" moz-do-not-send="true">dexter@expeedo.de</a>
                                    <<a
                                      href="mailto:dexter@expeedo.de"
                                      class="" moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
                                    wrote:<br class="">
                                    <br class="">
                                    It may actually not be a corruption
                                    of the filesystem but some timing
                                    issue on the mount procedure. To
                                    test that we could disable the auto
                                    formatting on<br class="">
                                    mount failures.<br class="">
                                    <br class="">
                                    The issue may also be dependant on
                                    the hardware version, i.e. it could
                                    be caused by the bug that caused the
                                    SD speed issue on the first 3.1
                                    batch.<br class="">
                                    <br class="">
                                    I only have tried the idf update on
                                    my batch 1 module (my bench /
                                    development module). I think most of
                                    our edge testers also have that
                                    version.<br class="">
                                    <br class="">
                                    Regards,<br class="">
                                    Michael<br class="">
                                    <br class="">
                                    <br class="">
                                    Am 23.11.18 um 02:32 schrieb Mark
                                    Webb-Johnson:<br class="">
                                    <blockquote type="cite" class="">I
                                      have raised the following github
                                      issue to Espressif:<br class="">
                                      <br class="">
                                         <a
                                        href="https://github.com/espressif/esp-idf/issues/2730"
                                        class="" moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730</a><br
                                        class="">
                                      <br class="">
                                      <br class="">
                                             Environment<br class="">
                                      <br class="">
                                           * Development Kit: none<br
                                        class="">
                                           * Kit version (for
                                      WroverKit/PicoKit/DevKitC): none<br
                                        class="">
                                           * Module or chip used:
                                      ESP32-WROVER 16MB<br class="">
                                           * IDF version (run |git
                                      describe --tags| to find it):
                                      v3.2-beta1-208-g0d7f2d77c<br
                                        class="">
                                           * Build System: make<br
                                        class="">
                                           * Compiler version (run
                                      |xtensa-esp32-elf-gcc --version|
                                      to find it): (crosstool-NG
                                      crosstool-ng-1.22.0-80-g6c4433a)
                                      5.2.0<br class="">
                                           * Operating System: macOS<br
                                        class="">
                                           * Power Supply: USB<br
                                        class="">
                                      <br class="">
                                      <br class="">
                                             Problem Description<br
                                        class="">
                                      <br class="">
                                         TLDR: Between May and July 2018
                                      a change was made to esp idf
                                      master that is causing corruption
                                      on FAT filesystems mounted on SPI
                                      flash.<br class="">
                                      <br class="">
                                         Our project uses a
                                      partitions.csv as follows:<br
                                        class="">
                                      <br class="">
                                         |# Name, Type, SubType, Offset,
                                      Size nvs, data, nvs, 0x9000,
                                      0x4000 otadata, data, ota, 0xd000,
                                      0x2000 phy_init, data, phy,
                                      0xf000, 0x1000 factory,<br
                                        class="">
                                         app, factory, 0x10000, 4M
                                      ota_0, app, ota_0, , 4M ota_1,
                                      app, ota_1, , 4M store, data, fat,
                                      , 1M |<br class="">
                                      <br class="">
                                         The 'store' partition is
                                      formatted as FAT, as follows:<br
                                        class="">
                                      <br class="">
                                         esp_vfs_fat_mount_config_t
                                      m_store_fat;<br class="">
                                         wl_handle_t m_store_wlh;<br
                                        class="">
   memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
                                        class="">
   m_store_fat.format_if_mount_failed = true;<br class="">
                                         m_store_fat.max_files = 5;<br
                                        class="">
   esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
                                      &m_store_wlh);<br class="">
                                      <br class="">
                                         We have previously used a clone
                                      of esp idf master, dated around
                                      May 22 2018, without issues. The
                                      partition is very reliable.<br
                                        class="">
                                      <br class="">
                                         However, on Jul 6 2018, we
                                      updated our clone to use the
                                      latest esp idf master at that
                                      time. Shortly afterwards, users
                                      started to report that their<br
                                        class="">
                                         'store' filesystem contents
                                      were corrupted. We rolled back.<br
                                        class="">
                                      <br class="">
                                         We have now tried again
                                      (updating on Oct 20 2018 to
                                      v3.2-beta1-208-g0d7f2d77c) and
                                      immediately had the same issue.
                                      Random corruption of FAT
                                      filesystem<br class="">
                                         in SPI flash.<br class="">
                                      <br class="">
                                      <br class="">
                                               Expected Behavior<br
                                        class="">
                                      <br class="">
                                         No corruption of FAT
                                      filesystem.<br class="">
                                      <br class="">
                                      <br class="">
                                               Actual Behavior<br
                                        class="">
                                      <br class="">
                                         Corruption of FAT filesystem.<br
                                        class="">
                                      <br class="">
                                      <br class="">
                                               Steps to reproduce<br
                                        class="">
                                      <br class="">
                                          1. Create a partition in SPI
                                      flash, and mount FAT filesystem<br
                                        class="">
                                          2. Read and write to files on
                                      FAT filesystem<br class="">
                                          3. Reboot<br class="">
                                          4. Observe random corruption
                                      and unmountable filesystem<br
                                        class="">
                                      <br class="">
                                      <br class="">
                                               Code to reproduce this
                                      issue<br class="">
                                      <br class="">
                                         esp_vfs_fat_mount_config_t
                                      m_store_fat;<br class="">
                                         wl_handle_t m_store_wlh;<br
                                        class="">
   memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
                                        class="">
   m_store_fat.format_if_mount_failed = true;<br class="">
                                         m_store_fat.max_files = 5;<br
                                        class="">
   esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
                                      &m_store_wlh);<br class="">
                                      <br class="">
                                      <br class="">
                                             Debug Logs<br class="">
                                      <br class="">
                                         n/a<br class="">
                                      <br class="">
                                      <br class="">
                                             Other items if possible<br
                                        class="">
                                      <br class="">
                                         Please advise if you need
                                      anything further.<br class="">
                                      <br class="">
                                      <br class="">
                                      I think the timeline is correct
                                      (the issue is in esp idf master
                                      some time between May and July
                                      2018), but please let me know if
                                      you know differently (or<br
                                        class="">
                                      update the github issue with your
                                      comments).<br class="">
                                      <br class="">
                                      Regards, Mark<br class="">
                                      <br class="">
                                      <blockquote type="cite" class="">On
                                        23 Nov 2018, at 6:19 AM, Michael
                                        Balzer <<a
                                          href="mailto:dexter@expeedo.de"
                                          class=""
                                          moz-do-not-send="true">dexter@expeedo.de</a>
                                        <<a
                                          href="mailto:dexter@expeedo.de"
                                          class=""
                                          moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
                                        wrote:<br class="">
                                        <br class="">
                                        esp-idf and OVMS branches are
                                        back to the working version.<br
                                          class="">
                                        <br class="">
                                        In case you also lost your
                                        config:  I also just fixed a bug
                                        on restoring into an empty
                                        /store partition.<br class="">
                                        <br class="">
                                        Regards,<br class="">
                                        Michael<br class="">
                                        <br class="">
                                        <br class="">
                                        Am 22.11.18 um 22:34 schrieb
                                        Michael Balzer:<br class="">
                                        <blockquote type="cite" class="">See
                                          <a
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165"
                                            class=""
                                            moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165</a><br
                                            class="">
                                          <br class="">
                                          I'll reset both master
                                          branches now.<br class="">
                                          <br class="">
                                          If you're about to pull,
                                          please wait until I've
                                          reverted the branches.<br
                                            class="">
                                          <br class="">
                                          Regards,<br class="">
                                          Michael<br class="">
                                          <br class="">
                                        </blockquote>
                                        <br class="">
                                        --<br class="">
                                        Michael Balzer * Helkenberger
                                        Weg 9 * D-58256 Ennepetal<br
                                          class="">
                                        Fon 02333 / 833 5735 * Handy
                                        0176 / 206 989 26<br class="">
                                        <br class="">
_______________________________________________<br class="">
                                        OvmsDev mailing list<br class="">
                                        <a
                                          href="mailto:OvmsDev@lists.openvehicles.com"
                                          class=""
                                          moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
                                        <<a
                                          href="mailto:OvmsDev@lists.openvehicles.com"
                                          class=""
                                          moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
                                          class="">
                                        <a
                                          href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                          class=""
                                          moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                          class="">
                                      </blockquote>
                                      <br class="">
                                      <br class="">
_______________________________________________<br class="">
                                      OvmsDev mailing list<br class="">
                                      <a
                                        href="mailto:OvmsDev@lists.openvehicles.com"
                                        class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                        class="">
                                      <a class="moz-txt-link-freetext"
                                        href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                        moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                        class="">
                                    </blockquote>
                                    <br class="">
                                    --<br class="">
                                    Michael Balzer * Helkenberger Weg 9
                                    * D-58256 Ennepetal<br class="">
                                    Fon 02333 / 833 5735 * Handy 0176 /
                                    206 989 26<br class="">
_______________________________________________<br class="">
                                    OvmsDev mailing list<br class="">
                                    <a
                                      href="mailto:OvmsDev@lists.openvehicles.com"
                                      class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
                                    <<a
                                      href="mailto:OvmsDev@lists.openvehicles.com"
                                      class="" moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
                                      class="">
                                    <a
                                      href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                      class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                      class="">
                                  </blockquote>
                                  <br class="">
                                  <br class="">
_______________________________________________<br class="">
                                  OvmsDev mailing list<br class="">
                                  <a
                                    href="mailto:OvmsDev@lists.openvehicles.com"
                                    class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                    class="">
                                  <a class="moz-txt-link-freetext"
                                    href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                    moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                    class="">
                                </blockquote>
                                <br class="">
                                --<br class="">
                                Michael Balzer * Helkenberger Weg 9 *
                                D-58256 Ennepetal<br class="">
                                Fon 02333 / 833 5735 * Handy 0176 / 206
                                989 26<br class="">
                                <br class="">
_______________________________________________<br class="">
                                OvmsDev mailing list<br class="">
                                <a
                                  href="mailto:OvmsDev@lists.openvehicles.com"
                                  class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                  class="">
                                <a class="moz-txt-link-freetext"
                                  href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                  moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                  class="">
                              </blockquote>
                              <br class="">
                              --<br class="">
                              Michael Balzer * Helkenberger Weg 9 *
                              D-58256 Ennepetal<br class="">
                              Fon 02333 / 833 5735 * Handy 0176 / 206
                              989 26<br class="">
                              <br class="">
                              <br class="">
                            </blockquote>
                            <br class="">
                                                       --
                            Steve_______________________________________________<br
                              class="">
                            OvmsDev mailing list<br class="">
                            <a
                              href="mailto:OvmsDev@lists.openvehicles.com"
                              class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                              class="">
                            <a class="moz-txt-link-freetext"
                              href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                              moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                              class="">
                          </div>
                        </div>
                      </blockquote>
                    </div>
                    <br class="">
                  </div>
                  <br class="">
                  <fieldset class="mimeAttachmentHeader"></fieldset>
                  <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
                </blockquote>
                <br class="">
                <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
              </div>
              _______________________________________________<br
                class="">
              OvmsDev mailing list<br class="">
              <a href="mailto:OvmsDev@lists.openvehicles.com" class=""
                moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                class="">
              <a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                class="">
            </div>
          </blockquote>
        </div>
        <br class="">
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
  </body>
</html>