<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Yep, I agree. It looks like core #0 is using the top 2MB and core #1 the bottom 2MB.<div class=""><br class=""></div><div class="">The question is will we see more problems with this workaround than without it?</div><div class=""><br class=""></div><div class="">I doubt if anything will come out of Espressif China for several weeks. Chinese New Year is here now and everything is shutting down for the break. Factories will be down for a month - businesses perhaps a fortnight.</div><div class=""><br class=""></div><div class="">At least we have confirmed that it is the same issue as:</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><h1 class="gh-header-title" style="box-sizing: border-box; font-size: 32px; margin: 0px 150px 0px 0px; font-weight: 400; line-height: 1.125; overflow-wrap: break-word; color: rgb(36, 41, 46); font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);"><span class="js-issue-title" style="box-sizing: border-box;">[TW#28180] PSRAM Cache Issue stills exist </span><span class="gh-header-number" style="box-sizing: border-box; color: rgb(163, 170, 177); letter-spacing: -1px;">#2892</span></h1></div></blockquote><div class=""><br class=""></div><div class="">Regards, Mark<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 31 Jan 2019, at 5:36 AM, Michael Balzer <<a href="mailto:dexter@expeedo.de" class="">dexter@expeedo.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
Comparison made, difference confirmed. Load times:<br class="">
<blockquote class=""><b class=""><tt class="">Pre-Workaround: </tt></b><br class="">
<tt class="">firmware 32.7 KB 5.08 s</tt><br class="">
<tt class="">firmware 32.7 KB 4.96 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.08 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.06 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.09 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.17 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.00 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.06 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.04 s</tt><br class="">
<tt class="">firmware 32.7 KB 4.96 s</tt><br class="">
<br class="">
<b class=""><tt class="">Post-Workaround: </tt></b><br class="">
<tt class="">firmware 32.7 KB 6.99 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.05 s</tt><br class="">
<tt class="">firmware 32.7 KB 4.83 s</tt><br class="">
<tt class="">firmware 32.7 KB 6.42 s</tt><br class="">
<tt class="">firmware 32.7 KB 6.50 s</tt><br class="">
<tt class="">firmware 32.7 KB 6.49 s</tt><br class="">
<tt class="">firmware 32.7 KB 6.40 s</tt><br class="">
<tt class="">firmware 32.7 KB 4.94 s</tt><br class="">
<tt class="">firmware 32.7 KB 5.17 s</tt><br class="">
<tt class="">firmware 32.7 KB 6.54 s</tt><br class="">
</blockquote>
<br class="">
This looks like the bug hits the wifi or lwip stack now, causing
frequent retransmits of corrupted packets.<br class="">
<br class="">
I just also tried running the test project with that 2 MB allocation
and starting the test task on core 0: the corruptions show again. So
it's reasonable to assume core 0 can only work with the lower 2 MB
and core 1 only with the higher 2 MB.<br class="">
<br class="">
Would it be an option to split the SPI RAM into two heaps and use
each depending on the core requesting memory?<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 30.01.19 um 21:01 schrieb Michael
Balzer:<br class="">
</div>
<blockquote type="cite" cite="mid:0c443a8b-be03-6268-c7bf-170a803a67a7@expeedo.de" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
Is it just my impression or has the wifi speed decreased with that
workaround?<br class="">
<br class="">
It's especially apparent on the firmware version info. That's just
23K, but takes some seconds to load. It was slow before though, I
need to make a comparison.<br class="">
<br class="">
I hope the workaround doesn't shift the issue to core 0 (running
the wifi stack)…<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 30.01.19 um 19:19 schrieb Michael
Balzer:<br class="">
</div>
<blockquote type="cite" cite="mid:98bbdf3e-1fb3-c8f5-c228-ec41c3792360@expeedo.de" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
Rolled out to "my" edgers. I'll watch crash counters during the
next days and report.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 30.01.19 um 12:44 schrieb Mark
Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:822711AA-42DE-4A94-93C7-B247C749FAB2@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
Yes, I suspect there is something in there related to the
core. I worry that (a) it is not as simple as 2MB/2MB, and (b)
we run code on both cores anyway. We may have just fixed the
test case (which runs on core #1), but not code running on
core #0.
<div class=""><br class="">
</div>
<div class="">Anyway, it is worth a try in the wild to see if
it reduces the volume of errors, while Espressif continue to
work on it.</div>
<div class=""><br class="">
</div>
<div class="">Kludgy wonderfulness is in commit
3.2.001-29-g311307cd.</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 30 Jan 2019, at 7:39 PM, Michael Balzer
<<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">Oh my…<br class="">
<br class="">
Can it be the first 2 MB are taboo for core 1 and
the second 2 MB for core 0?<br class="">
<br class="">
Anyway, we won't need those 2 MB for now, and
hopefully this will be fixed by Espressif soon.<br class="">
<br class="">
So I'd say yes, commit.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 30.01.19 um 12:26 schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:2F06DE6F-A324-4137-90C7-B7450C61A0BA@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
I tried changing CONFIG_SPIRAM_SIZE in sdkconfig
directly, but it just reset back to the default
4MB.
<div class=""><br class="">
</div>
<div class="">So changed esp-idf
itself, components/esp32/Kconfig, and added a
prompt for that config option. I could then
change it in menuconfig and the change
persisted.</div>
<div class=""><br class="">
</div>
<div class="">I then tested:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px;
border:
none;
padding:
0px;" class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (0) cpu_start: App cpu up.</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (715) spiram: SPI SRAM memory
test OK</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (716) heap_init:
Initializing. RAM available for dynamic
allocation:</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (716) heap_init: At 3FFAE6E0
len 00001920 (6 KiB): DRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (723) heap_init: At 3FFBA920
len 000256E0 (149 KiB): DRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (728) heap_init: At 3FFE0440
len 00003BC0 (14 KiB): D/IRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (735) heap_init: At 3FFE4350
len 0001BCB0 (111 KiB): D/IRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (742) heap_init: At 400953CC
len 0000AC34 (43 KiB): IRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (747) cpu_start: Pro cpu
start user code</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (752) spiram: Adding pool of
2048K of external SPI memory to heap
allocator</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (158) cpu_start: Starting
scheduler on PRO CPU.</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (0) cpu_start: Starting
scheduler on APP CPU.</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">This is ESP32 chip with 2 CPU
cores, WiFi/BT/BLE, silicon revision 1,
16MB external flash</span></font></div>
</div>
<div class=""><font class="" face="Andale Mono"><span style="font-size: 14px;" class="">…</span></font></div>
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I (9566) test_task: testing
mode 1 loopcnt 1000</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">E (9576) test_task: loop #1:
corruption detected at pos=497 size=498</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">4348864OVMS WIFI BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">#1: size=498, data @0x3f901ca0</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">4348864OVMS WIFI BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">E (9806) test_task: loop #5:
corruption detected at pos=497 size=498</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">4348864OVMS WIFI BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">E (10036) test_task: loop #9:
corruption detected at pos=829 size=830</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">4348864OVMS WIFI BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23</span></font></div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">which looks like a FAIL to me?</div>
<div class=""><br class="">
</div>
<div class="">So, then I tried something a
little funky:</div>
<div class=""><br class="">
</div>
<div class="">
<ol class="MailOutline">
<li class="">Go back to
normal CONFIG_SPIRAM_SIZE 4MB<br class="">
<br class="">
</li>
<li class="">Add this
to main/test_memory_corruption.cpp
app_main():<br class="">
<br class="">
<font class="" face="Andale Mono"><span style="font-size:
14px;" class="">void*
big2mb = heap_caps_malloc(1024*1024*2,
MALLOC_CAP_SPIRAM);<br class="">
printf("Big 2MB spi memory allocation
is at %p\n",big2mb);</span></font><br class="">
<br class="">
</li>
<li class="">Rebuild and run...</li>
</ol>
</div>
</div>
<div class=""><br class="">
</div>
<div class="">Here is what I get:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px;
border:
none;
padding:
0px;" class="">
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(0) cpu_start: App cpu up.</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1151) spiram: SPI SRAM memory test OK</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1152) heap_init: Initializing. RAM
available for dynamic allocation:</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1152) heap_init: At 3FFAE6E0 len
00001920 (6 KiB): DRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1159) heap_init: At 3FFBA920 len
000256E0 (149 KiB): DRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1164) heap_init: At 3FFE0440 len
00003BC0 (14 KiB): D/IRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1171) heap_init: At 3FFE4350 len
0001BCB0 (111 KiB): D/IRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1179) heap_init: At 400953CC len
0000AC34 (43 KiB): IRAM</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1184) cpu_start: Pro cpu start user
code</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1188) spiram: Adding pool of 4096K of
external SPI memory to heap allocator</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(206) cpu_start: Starting scheduler on
PRO CPU.</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(0) cpu_start: Starting scheduler on
APP CPU.</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">...</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">This
is ESP32 chip with 2 CPU cores,
WiFi/BT/BLE, silicon revision 1, 16MB
external flash</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">Big
2MB spi memory allocation is at
0x3f800028</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">...</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(291) wifi station:
ESP_WIFI_MODE_APSTA</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(291) wifi: wifi driver task:
3ffc25c4, prio:23, stack:3584, core=0</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">...</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(3281) wifi station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(3281) wifi station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(6111) wifi station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(6111) wifi station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(8461) mem_stresser: started on core 0
mem @0x3fa01b98</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(8461) mem_stresser: started on core 1
mem @0x3fa81bac</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(8951) wifi station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(8951) wifi station: connect to the AP
fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(9461) test_task: running on core 1</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(9561) test_task: testing mode 1
loopcnt 1000</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">#1:
size=1039, data @0x3fb01ca4</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">4348864OVMS
WIFI BLE BT cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2
rev=ESP32/1WLAN-214677-52wifi103.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.2-dev-1419-g5afc1b0cb-dirty Jan 27
2019 15:08:23)4348864OVMS WIFI BLE BT
cores=2 rev=ESP32/1</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(11781) wifi station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(11781) wifi station: connect to the
AP fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(14621) wifi station: retry to connect
to the AP</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(14621) wifi station: connect to the
AP fail</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
</div>
</div>
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(17451) wifi station: connect to the
AP fail</span></font></div>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">I left it running for a few
minutes, but no faults shown. Modes 1
through 4 all clean.</div>
<div class=""><br class="">
</div>
<div class="">So, to take the funkiness to the
next level, I added the same to ovms_main,
right at the start (actually in the log
level setter, as that seemed to be the first
thing that initialises). Somebody else gets
in for 28bytes of SPIRAM (or maybe that is a
structure header), but anyway...</div>
<div class=""><br class="">
</div>
</div>
</div>
<blockquote style="margin: 0 0 0 40px;
border:
none;
padding:
0px;" class="">
<div class="">
<div class="">
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1742) spiram: SPI SRAM memory test
OK</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1743) heap_init: Initializing. RAM
available for dynamic allocation:</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1743) heap_init: At 3FFAFF10 len
000000F0 (0 KiB): DRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1750) heap_init: At 3FFCEDE8 len
00011218 (68 KiB): DRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1755) heap_init: At 3FFE0440 len
00003BC0 (14 KiB): D/IRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1762) heap_init: At 3FFE4350 len
0001BCB0 (111 KiB): D/IRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1768) heap_init: At 4009F530 len
00000AD0 (2 KiB): IRAM</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1775) cpu_start: Pro cpu start user
code</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(1779) spiram: Adding pool of 4096K
of external SPI memory to heap
allocator</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(125) ovms_main: Set default logging
level for * to INFO</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(126) ovms_main: Pre-allocated 2MB
of SPIRAM at 0x3f800028 (workaround
ESP32 bug)</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">...</span></font></div>
<div class="">
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(17783) webserver: Launching Web
Server</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">I
(17823) ssh: Launching SSH Server</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">OVMS#
test string 500 2</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">#1:
stdlen = 1027</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">1985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty Jan
30 2019 19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty Jan
30 2019 19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty Jan
30 2019 19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty Jan
30 2019 19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty Jan
30 2019 19:17:20)012.61985504OVMS
WIFI BLE BT cores=2
rev=ESP32/168HIGHWAYS-29wifi163.2.001-22-g14d16854-dirty/factory/edge
(build idf
v3.1-dev-2835-g151269458-dirty Jan
30 2019 19:17:20)012.61985504</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">finished</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">OVMS#</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class=""><br class="">
</span></font></div>
<div class=""><font class="" face="Andale
Mono"><span style="font-size: 14px;" class="">OVMS#
module memory<br class="">
Free 8-bit 76160/230212, 32-bit
812/2720, SPIRAM 1886108/4194252<br class="">
--Task-- Total DRAM D/IRAM
IRAM SPIRAM +/- DRAM D/IRAM
IRAM SPIRAM<br class="">
main* 8368 0
0 8456 +8368 +0 +0
+8456<br class="">
no task* 5376 0
0 0 +5376 +0 +0
+0<br class="">
esp_timer 34808 0
6442123100 +34808 +0
+644+2123100<br class="">
OVMS Events 36352 41040
0 35120 +36352 +41040 +0
+35120<br class="">
OVMS DukTape 188 0
0 34668 +188 +0 +0
+34668<br class="">
OVMS CanRx 3124 0
0 26168 +3124 +0 +0
+26168<br class="">
ipc0 7776 0
0 0 +7776 +0 +0
+0<br class="">
ipc1 12 0
0 0 +12 +0 +0
+0<br class="">
Tmr Svc 0 88
0 0 +0 +88 +0
+0<br class="">
tiT 232 468
0 664 +232 +468 +0
+664<br class="">
wifi 600 8896
0 3324 +600 +8896 +0
+3324<br class="">
OVMS Console 0 200
1180 26332 +0 +200
+1180 +26332<br class="">
mdns 0 104
0 4 +0 +104 +0
+4<br class="">
OVMS NetMan 0 216
0 84 +0 +216 +0
+84</span></font></div>
</div>
</div>
</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">I think that wins as the most ugly
kludge to date.</div>
<div class=""><br class="">
</div>
<div class="">Shall I commit?</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 30 Jan 2019, at 5:00 PM,
Mark Webb-Johnson <<a href="mailto:mark@webb-johnson.net" class="" moz-do-not-send="true">mark@webb-johnson.net</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
line-break:
after-white-space;" class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 30 Jan 2019, at
4:41 PM, Michael Balzer <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">Mark,<br class="">
<br class="">
issue #2892 mentions using
only the lower 2 MB of SPI RAM
as a workaround. Where are our
allocations placed by the
allocator? Does it fill from
the middle or end? If not we
don't use the upper half yet.<br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
I saw that, and had a look. I don’t
see anything in makeconfig to limit to
2MB (not 4MB).</div>
<div class=""><br class="">
</div>
<div class="">I do see
a CONFIG_SPIRAM_SIZE=4194304 in
sdkconfig, but not sure how it gets
there. Perhaps we can just change
there to 2MB and try?</div>
<div class=""><br class="">
</div>
<div class="">I did see this in the
documentation:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0
40px;
border: none;
padding:
0px;" class="">
<div class=""><span style="caret-color:
rgb(64,
64,
64); color:
rgb(64,
64, 64);
font-family:
Lato,
proxima-nova,
"Helvetica
Neue",
Arial,
sans-serif;
font-size:
16px;
background-color:
rgb(252,
252,
252);" class="">During ESP-IDF startup,
external RAM is mapped into the
data address space starting at at
address 0x3F800000
(byte-accessible). The length of
this region is the same as the
SPIRAM size (up to the limit of
4MiB).</span></div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">So, maybe we can look at
the address our allocations come from
to see their offset from 0x3F800000?
If it is coming from the top 2MB, then
perhaps we start with a big 2MB
allocation that we never use?</div>
<div class=""> <br class="">
<blockquote type="cite" class="">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">A
possible workaround for the
most apparent issues with this
(string assembly) could be to
use char buffers instead of
std::string. I was thinking
about doing that at least for
the websocket stream.<br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">It seems strange that we
are only seeing it with std::string.
Maybe that just stresses the system
more, or does a lot of
reallocations?</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">
But if this really is a
hardware issue it can affect
all objects in SPI RAM,
appending to std::string then
only triggers this more often.<br class="">
<br class="">
Another workaround could be to
run everything on core 0.<br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Urgh.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 30.01.19 um 03:33 schrieb
Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:4737F304-A344-4BD8-9B0C-95366DB7A40F@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
Michael,
<div class=""><br class="">
</div>
<div class="">Espressif’s
response (and linking to
that other issue) sounds
like others are seeing this.</div>
<div class=""><br class="">
</div>
<div class="">Lousy timing
(with Chinese New Year next
week), so don’t expect
anything quick from
Espressif. I guess we’ll
just have to live with it
until they can find a
workaround. It sounds like
the issue is at the hardware
level and a compiler patch
will be needed.</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 30 Jan
2019, at 3:07 AM,
Michael Balzer <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""> → <a class="moz-txt-link-freetext" href="https://github.com/espressif/esp-idf/issues/3006" moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/3006</a><br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am
28.01.19 um 20:46
schrieb Michael
Balzer:<br class="">
</div>
<blockquote type="cite" cite="mid:b193b172-9a83-6df5-7b61-9e0cbfa46281@expeedo.de" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
To clarify: the
bug is most likely
not restricted to
the case of
building a message
in a buffer. It
can possibly cause
corruptions in any
RAM section, so
can well be
responsible for
many/most of the
unidentified
crashes and
stack/heap
corruptions we're
experiencing.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am
28.01.19 um
20:43 schrieb
Michael Balzer:<br class="">
</div>
<blockquote type="cite" cite="mid:06d0ea65-f4d5-e2c5-9ca1-b0ad63bfa3b1@expeedo.de" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
For those not
following the
github
discussion: I'm
pretty sure I've
nailed the bug
down.<br class="">
<br class="">
I have
reproduced the
bug in a simple
test project and
intend to raise
an issue with
Espressif on
this.<br class="">
<br class="">
<a class="moz-txt-link-freetext" href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457965435" moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457965435</a><br class="">
<br class="">
If you'd like to
test this on
your module,
configure the
project to your
wifi
credentials,
then use "make
flash". As the
test project is
small, this
normally will
not erase your
OVMS config
partition, but a
backup is always
recommended.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am
24.01.19 um
21:19 schrieb
Michael
Balzer:<br class="">
</div>
<blockquote type="cite" cite="mid:e4159a1c-b283-25b0-f1b5-ce570b529a1c@expeedo.de" class="">
<meta http-equiv="content-type" content="text/html; charset=UTF-8" class="">
Everyone
please have a
look at…<br class="">
<br class="">
<a class="moz-txt-link-freetext" href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457334248" moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/189#issuecomment-457334248</a><br class="">
<br class="">
Please try to
reproduce the
bug on your
modules.<br class="">
<br class="">
I'm open for
explanations.<br class="">
<br class="">
I thought this
might be some
copy-on-write
bug with
std::string,
but the gcc
5.x libstdc++
does no longer
use that
implementation
(wouldn't be
C++11
compliant as
well). I also
tried moving
all strings to
temporary
buffers, but
modes 5 &
6 eliminated
this
explanation as
well.<br class="">
<br class="">
My remaining
theories:<br class="">
<ul class="">
<li class="">A
task writing
out of bounds
(but only
0-bytes?)</li>
<li class="">A
hardware issue
only affecting
some modules<br class="">
</li>
</ul>
A hardware
issue only
affecting some
percentage of
ESP32 could
explain this
as well as the
strange heap
corruptions
that seem to
affect some
modules
especially
often.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
</blockquote>
</blockquote>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class="">
<pre class="moz-signature" cols="144">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></div></blockquote></div><br class=""></div></body></html>