<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Is it just my impression or has the wifi speed decreased with that
    workaround?<br>
    <br>
    It's especially apparent on the firmware version info. That's just
    23K, but takes some seconds to load. It was slow before though, I
    need to make a comparison.<br>
    <br>
    I hope the workaround doesn't shift the issue to core 0 (running the
    wifi stack)…<br>
    <br>
    Regards,<br>
    Michael<br>
    <br>
    <br>
    <div class="moz-cite-prefix">Am 30.01.19 um 19:19 schrieb Michael
      Balzer:<br>
    </div>
    <blockquote type="cite"
      cite="mid:98bbdf3e-1fb3-c8f5-c228-ec41c3792360@expeedo.de">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      Rolled out to "my" edgers. I'll watch crash counters during the
      next days and report.<br>
      <br>
      Regards,<br>
      Michael<br>
      <br>
      <br>
      <div class="moz-cite-prefix">Am 30.01.19 um 12:44 schrieb Mark
        Webb-Johnson:<br>
      </div>
      <blockquote type="cite"
        cite="mid:822711AA-42DE-4A94-93C7-B247C749FAB2@webb-johnson.net">
        <meta http-equiv="Content-Type" content="text/html;
          charset=UTF-8">
        Yes, I suspect there is something in there related to the core.
        I worry that (a) it is not as simple as 2MB/2MB, and (b) we run
        code on both cores anyway. We may have just fixed the test case
        (which runs on core #1), but not code running on core #0.
        <div class=""><br class="">
        </div>
        <div class="">Anyway, it is worth a try in the wild to see if it
          reduces the volume of errors, while Espressif continue to work
          on it.</div>
        <div class=""><br class="">
        </div>
        <div class="">Kludgy wonderfulness is in commit
          3.2.001-29-g311307cd.</div>
        <div class=""><br class="">
        </div>
        <div class="">Regards, Mark.<br class="">
          <div><br class="">
            <blockquote type="cite" class="">
              <div class="">On 30 Jan 2019, at 7:39 PM, Michael Balzer
                <<a href="mailto:dexter@expeedo.de" class=""
                  moz-do-not-send="true">dexter@expeedo.de</a>>
                wrote:</div>
              <br class="Apple-interchange-newline">
              <div class="">
                <meta http-equiv="Content-Type" content="text/html;
                  charset=UTF-8" class="">
                <div text="#000000" bgcolor="#FFFFFF" class="">
                  <div class="moz-cite-prefix">Oh my…<br class="">
                    <br class="">
                    Can it be the first 2 MB are taboo for core 1 and
                    the second 2 MB for core 0?<br class="">
                    <br class="">
                    Anyway, we won't need those 2 MB for now, and
                    hopefully this will be fixed by Espressif soon.<br
                      class="">
                    <br class="">
                    So I'd say yes, commit.<br class="">
                    <br class="">
                    Regards,<br class="">
                    Michael<br class="">
                    <br class="">
                    <br class="">
                    Am 30.01.19 um 12:26 schrieb Mark Webb-Johnson:<br
                      class="">
                  </div>
                  <blockquote type="cite"
                    cite="mid:2F06DE6F-A324-4137-90C7-B7450C61A0BA@webb-johnson.net"
                    class="">
                    <meta http-equiv="Content-Type" content="text/html;
                      charset=UTF-8" class="">
                    I tried changing CONFIG_SPIRAM_SIZE in sdkconfig
                    directly, but it just reset back to the default 4MB.
                    <div class=""><br class="">
                    </div>
                    <div class="">So changed esp-idf
                      itself, components/esp32/Kconfig, and added a
                      prompt for that config option. I could then change
                      it in menuconfig and the change persisted.</div>
                    <div class=""><br class="">
                    </div>
                    <div class="">I then tested:</div>
                    <div class=""><br class="">
                    </div>
                    <blockquote style="margin: 0 0 0 40px; border:
                      none;&#xA; padding:&#xA; 0px;" class="">
                      <div class="">
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (0)
                              cpu_start: App cpu up.</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (715)
                              spiram: SPI SRAM memory test OK</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (716)
                              heap_init: Initializing. RAM available for
                              dynamic allocation:</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (716)
                              heap_init: At 3FFAE6E0 len 00001920 (6
                              KiB): DRAM</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (723)
                              heap_init: At 3FFBA920 len 000256E0 (149
                              KiB): DRAM</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (728)
                              heap_init: At 3FFE0440 len 00003BC0 (14
                              KiB): D/IRAM</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (735)
                              heap_init: At 3FFE4350 len 0001BCB0 (111
                              KiB): D/IRAM</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (742)
                              heap_init: At 400953CC len 0000AC34 (43
                              KiB): IRAM</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (747)
                              cpu_start: Pro cpu start user code</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (752)
                              spiram: Adding pool of 2048K of external
                              SPI memory to heap allocator</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (158)
                              cpu_start: Starting scheduler on PRO CPU.</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (0)
                              cpu_start: Starting scheduler on APP CPU.</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">This is
                              ESP32 chip with 2 CPU cores, WiFi/BT/BLE,
                              silicon revision 1, 16MB external flash</span></font></div>
                      </div>
                      <div class=""><font class="" face="Andale Mono"><span
                            style="font-size: 14px;" class="">…</span></font></div>
                      <div class="">
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">I (9566)
                              test_task: testing mode 1 loopcnt 1000</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">E (9576)
                              test_task: loop #1: corruption detected at
                              pos=497 size=498</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">4348864OVMS
                              WIFI BLE BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class=""><br
                                class="">
                            </span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">#1:
                              size=498, data @0x3f901ca0</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">4348864OVMS
                              WIFI BLE BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class=""><br
                                class="">
                            </span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">E (9806)
                              test_task: loop #5: corruption detected at
                              pos=497 size=498</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">4348864OVMS
                              WIFI BLE BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class=""><br
                                class="">
                            </span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">E
                              (10036) test_task: loop #9: corruption
                              detected at pos=829 size=830</span></font></div>
                        <div class=""><font class="" face="Andale Mono"><span
                              style="font-size: 14px;" class="">4348864OVMS
                              WIFI BLE BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23)4348864OVMS WIFI BLE
                              BT cores=2
                              rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                              (build idf v3.2-dev-1419-g5afc1b0cb-dirty
                              Jan 27 2019 15:08:23</span></font></div>
                      </div>
                    </blockquote>
                    <div class="">
                      <div class=""><br class="">
                      </div>
                      <div class="">which looks like a FAIL to me?</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">So, then I tried something a little
                        funky:</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">
                        <ol class="MailOutline">
                          <li class="">Go back to
                            normal CONFIG_SPIRAM_SIZE 4MB<br class="">
                            <br class="">
                          </li>
                          <li class="">Add this
                            to main/test_memory_corruption.cpp
                            app_main():<br class="">
                            <br class="">
                            <font class="" face="Andale Mono"><span
                                style="font-size:&#xA; 14px;" class="">void*
                                big2mb = heap_caps_malloc(1024*1024*2,
                                MALLOC_CAP_SPIRAM);<br class="">
                                printf("Big 2MB spi memory allocation is
                                at %p\n",big2mb);</span></font><br
                              class="">
                            <br class="">
                          </li>
                          <li class="">Rebuild and run...</li>
                        </ol>
                      </div>
                    </div>
                    <div class=""><br class="">
                    </div>
                    <div class="">Here is what I get:</div>
                    <div class=""><br class="">
                    </div>
                    <blockquote style="margin: 0 0 0 40px; border:
                      none;&#xA; padding:&#xA; 0px;" class="">
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (0) cpu_start: App cpu up.</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1151) spiram: SPI SRAM
                                memory test OK</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1152) heap_init:
                                Initializing. RAM available for dynamic
                                allocation:</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1152) heap_init: At 3FFAE6E0
                                len 00001920 (6 KiB): DRAM</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1159) heap_init: At 3FFBA920
                                len 000256E0 (149 KiB): DRAM</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1164) heap_init: At 3FFE0440
                                len 00003BC0 (14 KiB): D/IRAM</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1171) heap_init: At 3FFE4350
                                len 0001BCB0 (111 KiB): D/IRAM</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1179) heap_init: At 400953CC
                                len 0000AC34 (43 KiB): IRAM</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1184) cpu_start: Pro cpu
                                start user code</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (1188) spiram: Adding pool of
                                4096K of external SPI memory to heap
                                allocator</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (206) cpu_start: Starting
                                scheduler on PRO CPU.</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (0) cpu_start: Starting
                                scheduler on APP CPU.</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">...</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">This is ESP32 chip with 2 CPU
                                cores, WiFi/BT/BLE, silicon revision 1,
                                16MB external flash</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">Big 2MB spi memory allocation
                                is at 0x3f800028</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">...</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (291) wifi station:
                                ESP_WIFI_MODE_APSTA</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (291) wifi: wifi driver task:
                                3ffc25c4, prio:23, stack:3584, core=0</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">...</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (3281) wifi station: retry to
                                connect to the AP</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (3281) wifi station: connect
                                to the AP fail</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class=""><br class="">
                              </span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (6111) wifi station: retry to
                                connect to the AP</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (6111) wifi station: connect
                                to the AP fail</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class=""><br class="">
                              </span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (8461) mem_stresser: started
                                on core 0 mem @0x3fa01b98</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (8461) mem_stresser: started
                                on core 1 mem @0x3fa81bac</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (8951) wifi station: retry to
                                connect to the AP</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (8951) wifi station: connect
                                to the AP fail</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class=""><br class="">
                              </span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (9461) test_task: running on
                                core 1</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class=""><br class="">
                              </span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (9561) test_task: testing
                                mode 1 loopcnt 1000</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">#1: size=1039, data @0x3fb01ca4</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">4348864OVMS WIFI BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                                (build idf
                                v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
                                2019 15:08:23)4348864OVMS WIFI BLE BT
                                cores=2
                                rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                                (build idf
                                v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
                                2019 15:08:23)4348864OVMS WIFI BLE BT
                                cores=2
                                rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                                (build idf
                                v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
                                2019 15:08:23)4348864OVMS WIFI BLE BT
                                cores=2
                                rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                                (build idf
                                v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
                                2019 15:08:23)4348864OVMS WIFI BLE BT
                                cores=2
                                rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                                (build idf
                                v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
                                2019 15:08:23)4348864OVMS WIFI BLE BT
                                cores=2
                                rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
                                (build idf
                                v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
                                2019 15:08:23)4348864OVMS WIFI BLE BT
                                cores=2 rev=ESP32/1</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class=""><br class="">
                              </span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (11781) wifi station: retry
                                to connect to the AP</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (11781) wifi station: connect
                                to the AP fail</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class=""><br class="">
                              </span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (14621) wifi station: retry
                                to connect to the AP</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (14621) wifi station: connect
                                to the AP fail</span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class=""><br class="">
                              </span></font></div>
                        </div>
                      </div>
                      <div class="">
                        <div class="">
                          <div class=""><font class="" face="Andale
                              Mono"><span style="font-size: 14px;"
                                class="">I (17451) wifi station: connect
                                to the AP fail</span></font></div>
                        </div>
                      </div>
                    </blockquote>
                    <div class="">
                      <div class="">
                        <div class=""><br class="">
                        </div>
                        <div class="">I left it running for a few
                          minutes, but no faults shown. Modes 1 through
                          4 all clean.</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">So, to take the funkiness to the
                          next level, I added the same to ovms_main,
                          right at the start (actually in the log level
                          setter, as that seemed to be the first thing
                          that initialises). Somebody else gets in for
                          28bytes of SPIRAM (or maybe that is a
                          structure header), but anyway...</div>
                        <div class=""><br class="">
                        </div>
                      </div>
                    </div>
                    <blockquote style="margin: 0 0 0 40px; border:
                      none;&#xA; padding:&#xA; 0px;" class="">
                      <div class="">
                        <div class="">
                          <div class="">
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1742) spiram: SPI SRAM memory test OK</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1743) heap_init: Initializing. RAM
                                  available for dynamic allocation:</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1743) heap_init: At 3FFAFF10 len
                                  000000F0 (0 KiB): DRAM</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1750) heap_init: At 3FFCEDE8 len
                                  00011218 (68 KiB): DRAM</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1755) heap_init: At 3FFE0440 len
                                  00003BC0 (14 KiB): D/IRAM</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1762) heap_init: At 3FFE4350 len
                                  0001BCB0 (111 KiB): D/IRAM</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1768) heap_init: At 4009F530 len
                                  00000AD0 (2 KiB): IRAM</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1775) cpu_start: Pro cpu start user
                                  code</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (1779) spiram: Adding pool of 4096K of
                                  external SPI memory to heap allocator</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (125) ovms_main: Set default logging
                                  level for * to INFO</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">I
                                  (126) ovms_main: Pre-allocated 2MB of
                                  SPIRAM at 0x3f800028 (workaround ESP32
                                  bug)</span></font></div>
                            <div class=""><font class=""
                                face="Andale&#xA; Mono"><span
                                  style="font-size: 14px;" class="">...</span></font></div>
                            <div class="">
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">I
                                    (17783) webserver: Launching Web
                                    Server</span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">I
                                    (17823) ssh: Launching SSH Server</span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">OVMS#
                                    test string 500 2</span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">#1:
                                    stdlen = 1027</span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">1985504OVMS
                                    WIFI BLE BT cores=2
                                    rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
                                    (build idf
                                    v3.1-dev-2835-g151269458-dirty Jan
                                    30 2019 19:17:20)012.61985504OVMS
                                    WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
                                    (build idf
                                    v3.1-dev-2835-g151269458-dirty Jan
                                    30 2019 19:17:20)012.61985504OVMS
                                    WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
                                    (build idf
                                    v3.1-dev-2835-g151269458-dirty Jan
                                    30 2019 19:17:20)012.61985504OVMS
                                    WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
                                    (build idf
                                    v3.1-dev-2835-g151269458-dirty Jan
                                    30 2019 19:17:20)012.61985504OVMS
                                    WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
                                    (build idf
                                    v3.1-dev-2835-g151269458-dirty Jan
                                    30 2019 19:17:20)012.61985504OVMS
                                    WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
                                    (build idf
                                    v3.1-dev-2835-g151269458-dirty Jan
                                    30 2019 19:17:20)012.61985504</span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class=""><br
                                      class="">
                                  </span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">finished</span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">OVMS#</span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class=""><br
                                      class="">
                                  </span></font></div>
                              <div class=""><font class=""
                                  face="Andale&#xA; Mono"><span
                                    style="font-size: 14px;" class="">OVMS#
                                    module memory<br class="">
                                    Free 8-bit 76160/230212, 32-bit
                                    812/2720, SPIRAM 1886108/4194252<br
                                      class="">
                                    --Task--     Total DRAM D/IRAM  
                                    IRAM SPIRAM   +/- DRAM D/IRAM   IRAM
                                    SPIRAM<br class="">
                                    main*              8368      0    
                                     0   8456      +8368     +0     +0
                                     +8456<br class="">
                                    no task*           5376      0    
                                     0      0      +5376     +0     +0  
                                      +0<br class="">
                                    esp_timer         34808      0  
                                     6442123100     +34808     +0  
                                    +644+2123100<br class="">
                                    OVMS Events       36352  41040    
                                     0  35120     +36352 +41040     +0
                                    +35120<br class="">
                                    OVMS DukTape        188      0    
                                     0  34668       +188     +0     +0
                                    +34668<br class="">
                                    OVMS CanRx         3124      0    
                                     0  26168      +3124     +0     +0
                                    +26168<br class="">
                                    ipc0               7776      0    
                                     0      0      +7776     +0     +0  
                                      +0<br class="">
                                    ipc1                 12      0    
                                     0      0        +12     +0     +0  
                                      +0<br class="">
                                    Tmr Svc               0     88    
                                     0      0         +0    +88     +0  
                                      +0<br class="">
                                    tiT                 232    468    
                                     0    664       +232   +468     +0  
                                    +664<br class="">
                                    wifi                600   8896    
                                     0   3324       +600  +8896     +0
                                     +3324<br class="">
                                    OVMS Console          0    200  
                                    1180  26332         +0   +200  +1180
                                    +26332<br class="">
                                    mdns                  0    104    
                                     0      4         +0   +104     +0  
                                      +4<br class="">
                                    OVMS NetMan           0    216    
                                     0     84         +0   +216     +0  
                                     +84</span></font></div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                    <div class="">
                      <div class=""><br class="">
                      </div>
                      <div class="">I think that wins as the most ugly
                        kludge to date.</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">Shall I commit?</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">Regards, Mark.</div>
                      <div class=""><br class="">
                        <blockquote type="cite" class="">
                          <div class="">On 30 Jan 2019, at 5:00 PM, Mark
                            Webb-Johnson <<a
                              href="mailto:mark@webb-johnson.net"
                              class="" moz-do-not-send="true">mark@webb-johnson.net</a>>
                            wrote:</div>
                          <br class="Apple-interchange-newline">
                          <div class="">
                            <meta http-equiv="Content-Type"
                              content="text/html; charset=UTF-8"
                              class="">
                            <div style="word-wrap: break-word;&#xA;
                              -webkit-nbsp-mode:&#xA; space;
                              line-break:&#xA; after-white-space;"
                              class=""><br class="">
                              <div class=""><br class="">
                                <blockquote type="cite" class="">
                                  <div class="">On 30 Jan 2019, at 4:41
                                    PM, Michael Balzer <<a
                                      href="mailto:dexter@expeedo.de"
                                      class="" moz-do-not-send="true">dexter@expeedo.de</a>>
                                    wrote:</div>
                                  <br class="Apple-interchange-newline">
                                  <div class="">
                                    <meta http-equiv="Content-Type"
                                      content="text/html; charset=UTF-8"
                                      class="">
                                    <div text="#000000"
                                      bgcolor="#FFFFFF" class="">
                                      <div class="moz-cite-prefix">Mark,<br
                                          class="">
                                        <br class="">
                                        issue #2892 mentions using only
                                        the lower 2 MB of SPI RAM as a
                                        workaround. Where are our
                                        allocations placed by the
                                        allocator? Does it fill from the
                                        middle or end? If not we don't
                                        use the upper half yet.<br
                                          class="">
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                                <div class=""><br class="">
                                </div>
                                I saw that, and had a look. I don’t see
                                anything in makeconfig to limit to 2MB
                                (not 4MB).</div>
                              <div class=""><br class="">
                              </div>
                              <div class="">I do see
                                a CONFIG_SPIRAM_SIZE=4194304 in
                                sdkconfig, but not sure how it gets
                                there. Perhaps we can just change there
                                to 2MB and try?</div>
                              <div class=""><br class="">
                              </div>
                              <div class="">I did see this in the
                                documentation:</div>
                              <div class=""><br class="">
                              </div>
                              <blockquote style="margin: 0 0 0
                                40px;&#xA; border: none;&#xA; padding:
                                0px;" class="">
                                <div class=""><span
                                    style="caret-color:&#xA; rgb(64,
                                    64,&#xA; 64); color: rgb(64,&#xA;
                                    64, 64); font-family:
                                    Lato,&#xA;&#xA; proxima-nova,
                                    "Helvetica&#xA; Neue",
                                    Arial,&#xA; sans-serif;&#xA;
                                    font-size: 16px;&#xA;
                                    background-color:&#xA; rgb(252,
                                    252,&#xA; 252);" class="">During
                                    ESP-IDF startup, external RAM is
                                    mapped into the data address space
                                    starting at at address 0x3F800000
                                    (byte-accessible). The length of
                                    this region is the same as the
                                    SPIRAM size (up to the limit of
                                    4MiB).</span></div>
                              </blockquote>
                              <div class=""><br class="">
                              </div>
                              <div class="">So, maybe we can look at the
                                address our allocations come from to see
                                their offset from 0x3F800000? If it is
                                coming from the top 2MB, then perhaps we
                                start with a big 2MB allocation that we
                                never use?</div>
                              <div class=""> <br class="">
                                <blockquote type="cite" class="">
                                  <div class="">
                                    <div text="#000000"
                                      bgcolor="#FFFFFF" class="">
                                      <div class="moz-cite-prefix">A
                                        possible workaround for the most
                                        apparent issues with this
                                        (string assembly) could be to
                                        use char buffers instead of
                                        std::string. I was thinking
                                        about doing that at least for
                                        the websocket stream.<br
                                          class="">
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                                <div class=""><br class="">
                                </div>
                                <div class="">It seems strange that we
                                  are only seeing it with std::string.
                                  Maybe that just stresses the system
                                  more, or does a lot of reallocations?</div>
                                <br class="">
                                <blockquote type="cite" class="">
                                  <div class="">
                                    <div text="#000000"
                                      bgcolor="#FFFFFF" class="">
                                      <div class="moz-cite-prefix"> But
                                        if this really is a hardware
                                        issue it can affect all objects
                                        in SPI RAM, appending to
                                        std::string then only triggers
                                        this more often.<br class="">
                                        <br class="">
                                        Another workaround could be to
                                        run everything on core 0.<br
                                          class="">
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                                <div class=""><br class="">
                                </div>
                                Urgh.</div>
                              <div class=""><br class="">
                                <blockquote type="cite" class="">
                                  <div class="">
                                    <div text="#000000"
                                      bgcolor="#FFFFFF" class="">
                                      <div class="moz-cite-prefix">Regards,<br
                                          class="">
                                        Michael<br class="">
                                        <br class="">
                                        <br class="">
                                        Am 30.01.19 um 03:33 schrieb
                                        Mark Webb-Johnson:<br class="">
                                      </div>
                                      <blockquote type="cite"
                                        cite="mid:4737F304-A344-4BD8-9B0C-95366DB7A40F@webb-johnson.net"
                                        class="">
                                        <meta http-equiv="Content-Type"
                                          content="text/html;
                                          charset=UTF-8" class="">
                                        Michael,
                                        <div class=""><br class="">
                                        </div>
                                        <div class="">Espressif’s
                                          response (and linking to that
                                          other issue) sounds like
                                          others are seeing this.</div>
                                        <div class=""><br class="">
                                        </div>
                                        <div class="">Lousy timing (with
                                          Chinese New Year next week),
                                          so don’t expect anything quick
                                          from Espressif. I guess we’ll
                                          just have to live with it
                                          until they can find a
                                          workaround. It sounds like the
                                          issue is at the hardware level
                                          and a compiler patch will be
                                          needed.</div>
                                        <div class=""><br class="">
                                        </div>
                                        <div class="">Regards, Mark.<br
                                            class="">
                                          <div class=""><br class="">
                                            <blockquote type="cite"
                                              class="">
                                              <div class="">On 30 Jan
                                                2019, at 3:07 AM,
                                                Michael Balzer <<a
                                                  href="mailto:dexter@expeedo.de"
                                                  class=""
                                                  moz-do-not-send="true">dexter@expeedo.de</a>>
                                                wrote:</div>
                                              <br
                                                class="Apple-interchange-newline">
                                              <div class="">
                                                <meta
                                                  http-equiv="Content-Type"
                                                  content="text/html;
                                                  charset=UTF-8"
                                                  class="">
                                                <div text="#000000"
                                                  bgcolor="#FFFFFF"
                                                  class=""> → <a
                                                    class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/issues/3006"
                                                    moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/3006</a><br
                                                    class="">
                                                  <br class="">
                                                  Regards,<br class="">
                                                  Michael<br class="">
                                                  <br class="">
                                                  <br class="">
                                                  <div
                                                    class="moz-cite-prefix">Am
                                                    28.01.19 um 20:46
                                                    schrieb Michael
                                                    Balzer:<br class="">
                                                  </div>
                                                  <blockquote
                                                    type="cite"
                                                    cite="mid:b193b172-9a83-6df5-7b61-9e0cbfa46281@expeedo.de"
                                                    class="">
                                                    <meta
                                                      http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
                                                    To clarify: the bug
                                                    is most likely not
                                                    restricted to the
                                                    case of building a
                                                    message in a buffer.
                                                    It can possibly
                                                    cause corruptions in
                                                    any RAM section, so
                                                    can well be
                                                    responsible for
                                                    many/most of the
                                                    unidentified crashes
                                                    and stack/heap
                                                    corruptions we're
                                                    experiencing.<br
                                                      class="">
                                                    <br class="">
                                                    Regards,<br class="">
                                                    Michael<br class="">
                                                    <br class="">
                                                    <br class="">
                                                    <div
                                                      class="moz-cite-prefix">Am
                                                      28.01.19 um 20:43
                                                      schrieb Michael
                                                      Balzer:<br
                                                        class="">
                                                    </div>
                                                    <blockquote
                                                      type="cite"
                                                      cite="mid:06d0ea65-f4d5-e2c5-9ca1-b0ad63bfa3b1@expeedo.de"
                                                      class="">
                                                      <meta
                                                        http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
                                                      For those not
                                                      following the
                                                      github discussion:
                                                      I'm pretty sure
                                                      I've nailed the
                                                      bug down.<br
                                                        class="">
                                                      <br class="">
                                                      I have reproduced
                                                      the bug in a
                                                      simple test
                                                      project and intend
                                                      to raise an issue
                                                      with Espressif on
                                                      this.<br class="">
                                                      <br class="">
                                                      <a
                                                        class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457965435"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457965435</a><br
                                                        class="">
                                                      <br class="">
                                                      If you'd like to
                                                      test this on your
                                                      module, configure
                                                      the project to
                                                      your wifi
                                                      credentials, then
                                                      use "make flash".
                                                      As the test
                                                      project is small,
                                                      this normally will
                                                      not erase your
                                                      OVMS config
                                                      partition, but a
                                                      backup is always
                                                      recommended.<br
                                                        class="">
                                                      <br class="">
                                                      Regards,<br
                                                        class="">
                                                      Michael<br
                                                        class="">
                                                      <br class="">
                                                      <br class="">
                                                      <div
                                                        class="moz-cite-prefix">Am
                                                        24.01.19 um
                                                        21:19 schrieb
                                                        Michael Balzer:<br
                                                          class="">
                                                      </div>
                                                      <blockquote
                                                        type="cite"
                                                        cite="mid:e4159a1c-b283-25b0-f1b5-ce570b529a1c@expeedo.de"
                                                        class="">
                                                        <meta
                                                          http-equiv="content-type"
content="text/html; charset=UTF-8" class="">
                                                        Everyone please
                                                        have a look at…<br
                                                          class="">
                                                        <br class="">
                                                        <a
                                                          class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457334248"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457334248</a><br
                                                          class="">
                                                        <br class="">
                                                        Please try to
                                                        reproduce the
                                                        bug on your
                                                        modules.<br
                                                          class="">
                                                        <br class="">
                                                        I'm open for
                                                        explanations.<br
                                                          class="">
                                                        <br class="">
                                                        I thought this
                                                        might be some
                                                        copy-on-write
                                                        bug with
                                                        std::string, but
                                                        the gcc 5.x
                                                        libstdc++ does
                                                        no longer use
                                                        that
                                                        implementation
                                                        (wouldn't be
                                                        C++11 compliant
                                                        as well). I also
                                                        tried moving all
                                                        strings to
                                                        temporary
                                                        buffers, but
                                                        modes 5 & 6
                                                        eliminated this
                                                        explanation as
                                                        well.<br
                                                          class="">
                                                        <br class="">
                                                        My remaining
                                                        theories:<br
                                                          class="">
                                                        <ul class="">
                                                          <li class="">A
                                                          task writing
                                                          out of bounds
                                                          (but only
                                                          0-bytes?)</li>
                                                          <li class="">A
                                                          hardware issue
                                                          only affecting
                                                          some modules<br
                                                          class="">
                                                          </li>
                                                        </ul>
                                                        A hardware issue
                                                        only affecting
                                                        some percentage
                                                        of ESP32 could
                                                        explain this as
                                                        well as the
                                                        strange heap
                                                        corruptions that
                                                        seem to affect
                                                        some modules
                                                        especially
                                                        often.<br
                                                          class="">
                                                        <br class="">
                                                        Regards,<br
                                                          class="">
                                                        Michael<br
                                                          class="">
                                                        <br class="">
                                                      </blockquote>
                                                    </blockquote>
                                                  </blockquote>
                                                </div>
                                              </div>
                                            </blockquote>
                                          </div>
                                        </div>
                                      </blockquote>
                                    </div>
                                  </div>
                                </blockquote>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                    </div>
                  </blockquote>
                  <br class="">
                  <pre class="moz-signature" cols="144">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
                </div>
                _______________________________________________<br
                  class="">
                OvmsDev mailing list<br class="">
                <a href="mailto:OvmsDev@lists.openvehicles.com" class=""
                  moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                  class="">
                <a class="moz-txt-link-freetext"
                  href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                  moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                  class="">
              </div>
            </blockquote>
          </div>
          <br class="">
        </div>
        <br>
        <fieldset class="mimeAttachmentHeader"></fieldset>
        <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
      </blockquote>
      <br>
      <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="160">-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
  </body>
</html>