Everyone, the poor JSON encoding performance reminded me of another issue we've had with Duktape since integration: the overall slowly degrading performance of the whole system over time. I did some new analysis on this effect using the task monitoring, which showed it's become worse lately up to the point where Duktape needs more than a second to perform the garbage collection after just 24 hours of uptime. It turned out this effect is also one of the major causes for the very bad JSON encoding performance, i.e. needing up to 10 seconds for a simple object of 5 x 1440 values, and eating 15-20% of CPU time on a permanent base (instead of the tolerable 5-6% right after boot). Doing a "script reload" would help, but only temporarily and not near the point of a freshly booted system. I didn't find anything about such an effect for Duktape in general, so I turned to examining possible ESP32 specific causes, and finally found the culprit: the esp-idf memory management, or more precisely, it's issues with fragmentation in SPIRAM. Duktape is probably one of the main fragmentation drivers due to the nature of Javascript memory management (garbage collection). There has been some discussion on this by neoniousTR, who also was involved in the SPIRAM bug hunt: https://www.esp32.com/viewtopic.php?t=8628 The dlmalloc adaptation of neoniousTR unfortunately isn't usable for us, as it exceeds our available IRAM. It also was based on a much earlier esp-idf release, and doesn't support the extensions and options we use. Espressif have introduced a new memory manager to address these issues, but only in esp-idf release 4.3. We'll need to go there one day, but that's nothing we can do short term: https://github.com/espressif/esp-idf/releases/tag/v4.3 → "Heap: Switched heap algorithm to one based on TLSF, improves performance especially when using a high number of allocations in PSRAM" So I've tried another option: I've removed the Duktape heap from the standard system memory management. I've added a separate umm_malloc instance just for the heap, which can work very fast because it doesn't need to take care of locking or poisoning. On Duktape startup, a fixed amount of SPIRAM gets allocated for the Javascript heap. The amount is 512 KB by default, which should be sufficient for most cases. I've added a configuration for this, config module duktape.heapsize, and a meminfo command to provide some insight. The results are: * Overall Duktape speedup of at least factor 3, up to factor 8 * Garbage collection runs now need below 100 ms even with large scripts running * Duktape average CPU usage dropped to 1-2% * Barely noticeable overall performance degradation after a week of continuous operation Of course, your mileage may vary, feedback is welcome. It still degrades, which is kind of expected, as Duktape isn't the only system component doing dynamic memory allocations. I think esp-idf 4.3 may offer a significant overall performance increase for us, just by replacing the current memory management by a more sophisticated implementation. As a side effect, JSON encoding is now much faster (by up to factor 8). CBOR has become only about 30% faster, which is probably due to CBOR needing much less memory allocations. CBOR is still factor 3 faster on average than JSON, so still the preferred choice for storage & transmission. Details & code: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/1bcc... Note: there are three new Kconfig variables that need to be set to enable the new Duktape memory management. The sdkconfig.default.hw31 includes the new configuration, if doing a "make", simply accept all defaults. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Michael, Thanks for performing this diagnosis and enhancement. Has anyone analyzed the work required to move to current esp-idf? How many things will break? I imagine the heap instrumentation I added would be one of them. Perhaps we could divide up that work among several developers. I would likely need to get another module, though, because my current v3.0 bench unit without SPIRAM probably wouldn't be usable. -- Steve On Sat, 4 Sep 2021, Michael Balzer wrote:
Everyone,
the poor JSON encoding performance reminded me of another issue we've had with Duktape since integration: the overall slowly degrading performance of the whole system over time.
I did some new analysis on this effect using the task monitoring, which showed it's become worse lately up to the point where Duktape needs more than a second to perform the garbage collection after just 24 hours of uptime. It turned out this effect is also one of the major causes for the very bad JSON encoding performance, i.e. needing up to 10 seconds for a simple object of 5 x 1440 values, and eating 15-20% of CPU time on a permanent base (instead of the tolerable 5-6% right after boot).
Doing a "script reload" would help, but only temporarily and not near the point of a freshly booted system.
I didn't find anything about such an effect for Duktape in general, so I turned to examining possible ESP32 specific causes, and finally found the culprit: the esp-idf memory management, or more precisely, it's issues with fragmentation in SPIRAM. Duktape is probably one of the main fragmentation drivers due to the nature of Javascript memory management (garbage collection).
There has been some discussion on this by neoniousTR, who also was involved in the SPIRAM bug hunt: https://www.esp32.com/viewtopic.php?t=8628
The dlmalloc adaptation of neoniousTR unfortunately isn't usable for us, as it exceeds our available IRAM. It also was based on a much earlier esp-idf release, and doesn't support the extensions and options we use.
Espressif have introduced a new memory manager to address these issues, but only in esp-idf release 4.3. We'll need to go there one day, but that's nothing we can do short term: https://github.com/espressif/esp-idf/releases/tag/v4.3 → "Heap: Switched heap algorithm to one based on TLSF, improves performance especially when using a high number of allocations in PSRAM"
So I've tried another option: I've removed the Duktape heap from the standard system memory management. I've added a separate umm_malloc instance just for the heap, which can work very fast because it doesn't need to take care of locking or poisoning.
On Duktape startup, a fixed amount of SPIRAM gets allocated for the Javascript heap. The amount is 512 KB by default, which should be sufficient for most cases. I've added a configuration for this, config module duktape.heapsize, and a meminfo command to provide some insight.
The results are:
* Overall Duktape speedup of at least factor 3, up to factor 8 * Garbage collection runs now need below 100 ms even with large scripts running * Duktape average CPU usage dropped to 1-2% * Barely noticeable overall performance degradation after a week of continuous operation
Of course, your mileage may vary, feedback is welcome.
It still degrades, which is kind of expected, as Duktape isn't the only system component doing dynamic memory allocations.
I think esp-idf 4.3 may offer a significant overall performance increase for us, just by replacing the current memory management by a more sophisticated implementation.
As a side effect, JSON encoding is now much faster (by up to factor 8). CBOR has become only about 30% faster, which is probably due to CBOR needing much less memory allocations. CBOR is still factor 3 faster on average than JSON, so still the preferred choice for storage & transmission.
Details & code: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/1bcc...
Note: there are three new Kconfig variables that need to be set to enable the new Duktape memory management. The sdkconfig.default.hw31 includes the new configuration, if doing a "make", simply accept all defaults.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Craig, you're probably qualified best to answer this, as you already did some steps to port to idf 4.0? https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/360 But I guess there have been quite some changes since 4.0 also. We'll have to try and solve the issues one by one. Probably best to get for-v3.3 including the SIM7600 support stable and merged into master first. Mark, I'll merge the heap change into for-v3.3 next. Regards, Michael Am 04.09.21 um 21:52 schrieb Stephen Casner:
Michael,
Thanks for performing this diagnosis and enhancement.
Has anyone analyzed the work required to move to current esp-idf? How many things will break? I imagine the heap instrumentation I added would be one of them. Perhaps we could divide up that work among several developers. I would likely need to get another module, though, because my current v3.0 bench unit without SPIRAM probably wouldn't be usable.
-- Steve
On Sat, 4 Sep 2021, Michael Balzer wrote:
Everyone,
the poor JSON encoding performance reminded me of another issue we've had with Duktape since integration: the overall slowly degrading performance of the whole system over time.
I did some new analysis on this effect using the task monitoring, which showed it's become worse lately up to the point where Duktape needs more than a second to perform the garbage collection after just 24 hours of uptime. It turned out this effect is also one of the major causes for the very bad JSON encoding performance, i.e. needing up to 10 seconds for a simple object of 5 x 1440 values, and eating 15-20% of CPU time on a permanent base (instead of the tolerable 5-6% right after boot).
Doing a "script reload" would help, but only temporarily and not near the point of a freshly booted system.
I didn't find anything about such an effect for Duktape in general, so I turned to examining possible ESP32 specific causes, and finally found the culprit: the esp-idf memory management, or more precisely, it's issues with fragmentation in SPIRAM. Duktape is probably one of the main fragmentation drivers due to the nature of Javascript memory management (garbage collection).
There has been some discussion on this by neoniousTR, who also was involved in the SPIRAM bug hunt: https://www.esp32.com/viewtopic.php?t=8628
The dlmalloc adaptation of neoniousTR unfortunately isn't usable for us, as it exceeds our available IRAM. It also was based on a much earlier esp-idf release, and doesn't support the extensions and options we use.
Espressif have introduced a new memory manager to address these issues, but only in esp-idf release 4.3. We'll need to go there one day, but that's nothing we can do short term: https://github.com/espressif/esp-idf/releases/tag/v4.3 → "Heap: Switched heap algorithm to one based on TLSF, improves performance especially when using a high number of allocations in PSRAM"
So I've tried another option: I've removed the Duktape heap from the standard system memory management. I've added a separate umm_malloc instance just for the heap, which can work very fast because it doesn't need to take care of locking or poisoning.
On Duktape startup, a fixed amount of SPIRAM gets allocated for the Javascript heap. The amount is 512 KB by default, which should be sufficient for most cases. I've added a configuration for this, config module duktape.heapsize, and a meminfo command to provide some insight.
The results are:
* Overall Duktape speedup of at least factor 3, up to factor 8 * Garbage collection runs now need below 100 ms even with large scripts running * Duktape average CPU usage dropped to 1-2% * Barely noticeable overall performance degradation after a week of continuous operation
Of course, your mileage may vary, feedback is welcome.
It still degrades, which is kind of expected, as Duktape isn't the only system component doing dynamic memory allocations.
I think esp-idf 4.3 may offer a significant overall performance increase for us, just by replacing the current memory management by a more sophisticated implementation.
As a side effect, JSON encoding is now much faster (by up to factor 8). CBOR has become only about 30% faster, which is probably due to CBOR needing much less memory allocations. CBOR is still factor 3 faster on average than JSON, so still the preferred choice for storage & transmission.
Details & code: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/1bcc...
Note: there are three new Kconfig variables that need to be set to enable the new Duktape memory management. The sdkconfig.default.hw31 includes the new configuration, if doing a "make", simply accept all defaults.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
On 9/4/21 1:39 PM, Michael Balzer wrote:
Craig, you're probably qualified best to answer this, as you already did some steps to port to idf 4.0?
https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/360
But I guess there have been quite some changes since 4.0 also. We'll have to try and solve the issues one by one.
I haven't looked at this in some time. You can learn a lot by just trying to build with a different idf. And for me there's the meta problem that it's some work for me to make a FreeBSD port that targets a different/newer toolchain. But I will support any work in this area.
Probably best to get for-v3.3 including the SIM7600 support stable and merged into master first. Mark, I'll merge the heap change into for-v3.3 next.
That sounds right to me. And since I have a (frankenstein) SIM7600 modem board, I booted Mark's latest for-v3.3 version on my dev unit Thursday but I have 0 dBm signal and the gps says "Connected on channel: #1" and does not have lock. I think my test antenna situation is good and the sim card was working with the sim5360. dashboard.hologram.io shows my production boards both being active in the last hour, the dev unit says "ready" but has no last active date or usage. So anyway, I have some debugging ahead of me this weekend... Craig OVMS# cell MODEM Status Model: SIM7600 Network Registration: Searching Provider: Signal: 0 dBm Mode: State: NetWait Mux: Status up PPP: Not running GPS: Connected on channel: #1 OVMS# location status Currently at 0.000000,0.000000 (without GPS lock) There are 2 locations defined No active locations
Craig, Regarding GPS lock, check AT+GPSAUTO? It should be disabled (0). If not, do a AT&F, then AT&W, and reboot. I suspect this may be a fault in simcom firmware / production with modules delivered not at factory defaults; as one of my two test modules was showing GPSAUTO 1, while a AT&F factory reset put it back to 0. I still haven’t worked around that in the 3.3 firmware driver yet (problem being we can’t configure the GPS while it is running, and GPSAUTO=1 starts the GPS on startup). If not that, set log level to debug, and send me what you see. I’ve tidied up the modem (now cellular) driver debugging code in 3.3 and that should clearly show the commands and responses to/from the modem now. Regards, Mark
On 5 Sep 2021, at 5:11 AM, Craig Leres <leres@xse.com> wrote:
On 9/4/21 1:39 PM, Michael Balzer wrote:
Craig, you're probably qualified best to answer this, as you already did some steps to port to idf 4.0? https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/360 But I guess there have been quite some changes since 4.0 also. We'll have to try and solve the issues one by one.
I haven't looked at this in some time. You can learn a lot by just trying to build with a different idf.
And for me there's the meta problem that it's some work for me to make a FreeBSD port that targets a different/newer toolchain. But I will support any work in this area.
Probably best to get for-v3.3 including the SIM7600 support stable and merged into master first. Mark, I'll merge the heap change into for-v3.3 next.
That sounds right to me. And since I have a (frankenstein) SIM7600 modem board, I booted Mark's latest for-v3.3 version on my dev unit Thursday but I have 0 dBm signal and the gps says "Connected on channel: #1" and does not have lock. I think my test antenna situation is good and the sim card was working with the sim5360. dashboard.hologram.io shows my production boards both being active in the last hour, the dev unit says "ready" but has no last active date or usage. So anyway, I have some debugging ahead of me this weekend...
Craig
OVMS# cell MODEM Status Model: SIM7600 Network Registration: Searching Provider: Signal: 0 dBm Mode: State: NetWait Mux: Status up PPP: Not running GPS: Connected on channel: #1 OVMS# location status Currently at 0.000000,0.000000 (without GPS lock) There are 2 locations defined No active locations _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
On 9/4/21 5:47 PM, Mark Webb-Johnson wrote:
Regarding GPS lock, check AT+GPSAUTO? It should be disabled (0). If not, do a AT&F, then AT&W, and reboot. I suspect this may be a fault in simcom firmware / production with modules delivered not at factory defaults; as one of my two test modules was showing GPSAUTO 1, while a AT&F factory reset put it back to 0. I still haven’t worked around that in the 3.3 firmware driver yet (problem being we can’t configure the GPS while it is running, and GPSAUTO=1 starts the GPS on startup).
Helpful; after figuring out AT+GPSAUTO? -> AT+CGPSAUTO?, it was indeed enabled, and factory defaults+reboot fixed it. Running with "log level verbose cellular" and looking at the SIM7600 AT command manual I see the module doesn't seem to know what time it is (CCLK): D (554006) cellular: mux-tx #3: AT+CREG?;+CCLK?;+CSQ;+COPS? D (554016) cellular: mux-rx-line #3: +CREG: 1,2 D (554026) cellular: mux-rx-line #3: +CCLK: "80/04/01,02:10:04" D (554026) cellular: mux-rx-line #3: +CSQ: 99,99 D (554026) cellular: mux-rx-line #3: +COPS: 0 Perhaps related to not being registered. I'm not making much sense of the others and scanning the pdf I don't see anything else that might be useful. Something I can try without much effort is to swap in my spare SIM5380 modem, that might reveal some other issue with my setup. Regarding how to deal with the GPSAUTO=1 problem, what comes to mind would be a button on the cellular page that says "click here to fix the GPS and reboot". And I guess I've never used the guided setup but it could detect/fix this too. Nice work on the simcom rework; if I can get this working I'd seriously think about moving the SIM7600 to one of my cars. Craig
On 9/4/21 6:45 PM, Craig Leres wrote:
Something I can try without much effort is to swap in my spare SIM5380 modem, that might reveal some other issue with my setup.
I tried this and the sim5380 was able to register right away and I could see that it was active on the hologram.io dashboard. So I'm sure now that my antenna works at least to the u.fl pigtail. I checked resistance between the center pin of the ant3 u.fl socket and the pin on sim5380 module and it was single digits M ohms. But I see now that both ant2 and ant3 on my sim7600 module are shorted to ground. So that's a problem. I remember trying to move the u.fl socket from ant3 to ant2 and giving up and soldering new sockets in both locations. And I found the socket I removed. Comparing it to some unused new sockets (mouser 712-CONUFL001-SMD) I see they this stupid tab that comes off of the grounded part of the connector. I thought all 4 sides of the old u.fl were soldered but looking at the old one I removed, I see now that only 3 were and the 4th just had solder on the trace. Oops. Well this should be an easy fix (assuming I haven't hurt the simcom -- but if so I have two new sim7600 and one sim7100 handy). Craig
As it turns out, u.fl sockets are directional (if I had looked more closely I would have seen this). The socket I replaced at ant3 had the correct orientation but after removing it all 4 pins are still shorted together. This makes me think ant3 is not really used with a sim7600. The socket on ant2 was reversed. Installing a new one correctly solved the short circuit but now it appears to be open and I'm still getting no signal (+CSQ: 99,99). So I guess now I need to swap the simcom module. Craig
Craig, For SIM7600, ANT1 is GPS, and ANT2 is GSM. ANT3 should not be connected. Regards, Mark
On 6 Sep 2021, at 1:40 AM, Craig Leres <leres@xse.com> wrote:
As it turns out, u.fl sockets are directional (if I had looked more closely I would have seen this). The socket I replaced at ant3 had the correct orientation but after removing it all 4 pins are still shorted together. This makes me think ant3 is not really used with a sim7600.
The socket on ant2 was reversed. Installing a new one correctly solved the short circuit but now it appears to be open and I'm still getting no signal (+CSQ: 99,99). So I guess now I need to swap the simcom module.
Craig
Although it checked out ok when I first reinstalled it, the ant2 u.fl center conductor was not making the trip to the sma connector. I put some flux on it and reheated it and was able to get some M ohms at first but then nothing. I printed the sim7600 hardware design manual page that shows the layout of the pins. Interestingly, the top pins seem to be reversed (?!?!), for example gnss_ant buzzes out as the third pin from the top right instead of third pin from the top left as shown in the pdf. But aux_ant buzzes ok. But I see my problem: R6 and R7 are not populated. Looking around ant3, I see R8 and R9 are 0 ohm jumpers, so I think I need to move them. (C8 and C9 are also empty but so are C10 or C11 so guessing nothing to do there.) Craig
On 9/5/21 7:51 PM, Craig Leres wrote:
Interestingly, the top pins seem to be reversed (?!?!), for example gnss_ant buzzes out as the third pin from the top right instead of third pin from the top left as shown in the pdf.
Looking at the schematic I see now gnss_ant and the ant1 u.fl have a resistor and capacitor in series but vbat has a 10 ohm resistor and inductor. So that explains why I buzzed the wrong pin... Craig
Progress! I was a little worried because I was measuring ~2 ohms between the antenna pin and ground but I tried it anyway and I have signal now. I saw crashes the first few times the cell network came up but it eventually settled down and is working now. Maybe an edge case in the dhcp code vs. first time network registration? FYI I'm running 3.2.016-377-g1e6da7e5 (for-v3.3 branch). Craig ------------------------------------------------------------- [initial boot/crash] OVMS> I (30972) cellular: State: Enter NetWait state I (30972) gsm-nmea: Startup OVMS> ena Password: I (31422) webserver: HTTP GET / I (31502) webserver: HTTP GET /assets/script.js I (31882) webserver: HTTP GET /apple-touch-icon.png I (32022) webserver: HTTP GET /cfg/cellular OVMS# I (35972) cellular: Network Registration status: DeniedRegistration I (36172) cellular: Signal Quality is: 24 (-65 dBm) OVMS# simcom Unrecognised command I (40322) cellular: Signal Quality is: 20 (-73 dBm) OVMS# cell MODEM Status Model: SIM7600 Network Registration: DeniedRegistration Provider: Signal: -73 dBm Mode: State: NetWait Mux: Status up PPP: Not running GPS: Connected on channel: #1 I (43042) cellular: Network Registration status: RegisteredRoaming I (43972) cellular: State: Enter NetStart state I (45032) cellular: PPP Connection is ready to start I (45972) cellular: State: Enter NetMode state I (45972) gsm-ppp: Initialising... OVMS# log level verbose cellular Logging level for cellular set to verbose OVMS# cell MODEM Status Model: SIM7600 Network Registration: RegisteredRoaming Provider: Signal: -73 dBm Mode: State: NetMode Mux: Status up PPP: Not connected GPS: Connected on channel: #1 D (52362) cellular: mux-rx-line #3: NO CARRIER D (52362) cellular: mux-rx-line #4: NO CARRIER D (52372) cellular: mux-rx-line #3: +PPPD: DISCONNECTED I (52372) cellular: PPP Connection disconnected D (52372) cellular: mux-rx-line #4: +PPPD: DISCONNECTED I (52372) cellular: PPP Connection disconnected OVMS# Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was un handled. Core 1 register dump: PC : 0x400fd4b8 PS : 0x00060830 A0 : 0x800fd5c5 A1 : 0x3f fca010 A2 : 0x3ffb5140 A3 : 0xa5a5a5a5 A4 : 0x3f488bd6 A5 : 0x3f fb5148 A6 : 0x00000073 A7 : 0x000000a5 A8 : 0x800fd481 A9 : 0x3f fc9ff0 A10 : 0x3ffca058 A11 : 0x3ffca050 A12 : 0x400f895c A13 : 0x00 000000 A14 : 0x3ffc91ac A15 : 0x00000001 SAR : 0x00000008 EXCCAUSE: 0x00 00001c EXCVADDR: 0xa5a5a663 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xff ffffff ELF file SHA256: f80a889fca6f51ca Backtrace: 0x400fd4b8:0x3ffca010 0x400fd5c2:0x3ffca090 0x4022dfa9:0x3ffca0b0 Rebooting... ets Jul 29 2019 12:21:46 rst:0xc (SW_CPU_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 ------------------------------------------------------------- [decoded backtace] ice 485 % ./backtrace.sh 0x400fd4b8:0x3ffca010 0x400fd5c2:0x3ffca090 0x4022dfa9:0x3ffca0b0 + xtensa-esp32-elf-addr2line -e build/ovms3.elf 0x400fd4b8:0x3ffca010 0x400fd5c2:0x3ffca090 0x4022dfa9:0x3ffca0b0 main/ovms_notify.cpp:739 main/ovms_notify.cpp:739 /home/ice/u0/leres/esp/openvehicles-xtensa-esp32-elf/components/lwip/lwip/src/core/ipv4/dhcp.c:916 ------------------------------------------------------------- [later successful boot/network registration] OVMS# cell MODEM Status Model: SIM7600 Network Registration: RegisteredRoaming Provider: T-Mobile Hologram Signal: -73 dBm Mode: LTE,Online State: NetMode Mux: Status up PPP: Connected on channel: #2 GPS: Connected on channel: #1 D (174995) cellular: mux-tx #3: AT+CREG?;+CCLK?;+CSQ;+CPSI?;+COPS? D (175015) cellular: mux-rx-line #3: +CPSI: LTE,Online,310-260,0x417B,23125765,176,EUTRAN-BAND12,5035,2,2,-132,-1012,-741,9 D (175015) cellular: mux-rx-line #3: +CREG: 1,5 D (175015) cellular: mux-rx-line #3: +CCLK: "21/09/05,20:29:00-28" D (175015) cellular: mux-rx-line #3: +CSQ: 20,99 D (175015) cellular: mux-rx-line #3: +COPS: 0,0,"T-Mobile Hologram",7
I agree. Let’s get 3.3 out the door. Any change to IDF 4.x will require a freeze on changes to the core frameworks we have while we make that change. It will certainly be easier if we have a few people helping out with this (in particular where we have changed/extended the Espressif stuff). I have high hopes that this will improve stability and performance, as we are so far behind Espressif now. Another issue to be addressed is flash size. Currently we have 4MB factory, 4MB OTA0, 4MB OTA1, and the rest for /store, etc. Given our image is now 3.5MB, we are going to hit these limits sometime soon. An alternative arrangement is to just switch to two firmware partitions of 6GB. Current code runs from one, with new firmware updates downloaded to the other. We could do this upgrade in-place for existing users. Regards, Mark.
On 5 Sep 2021, at 4:39 AM, Michael Balzer <dexter@expeedo.de> wrote:
Craig, you're probably qualified best to answer this, as you already did some steps to port to idf 4.0?
https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/360
But I guess there have been quite some changes since 4.0 also. We'll have to try and solve the issues one by one.
Probably best to get for-v3.3 including the SIM7600 support stable and merged into master first. Mark, I'll merge the heap change into for-v3.3 next.
Regards, Michael
Am 04.09.21 um 21:52 schrieb Stephen Casner:
Michael,
Thanks for performing this diagnosis and enhancement.
Has anyone analyzed the work required to move to current esp-idf? How many things will break? I imagine the heap instrumentation I added would be one of them. Perhaps we could divide up that work among several developers. I would likely need to get another module, though, because my current v3.0 bench unit without SPIRAM probably wouldn't be usable.
-- Steve
On Sat, 4 Sep 2021, Michael Balzer wrote:
Everyone,
the poor JSON encoding performance reminded me of another issue we've had with Duktape since integration: the overall slowly degrading performance of the whole system over time.
I did some new analysis on this effect using the task monitoring, which showed it's become worse lately up to the point where Duktape needs more than a second to perform the garbage collection after just 24 hours of uptime. It turned out this effect is also one of the major causes for the very bad JSON encoding performance, i.e. needing up to 10 seconds for a simple object of 5 x 1440 values, and eating 15-20% of CPU time on a permanent base (instead of the tolerable 5-6% right after boot).
Doing a "script reload" would help, but only temporarily and not near the point of a freshly booted system.
I didn't find anything about such an effect for Duktape in general, so I turned to examining possible ESP32 specific causes, and finally found the culprit: the esp-idf memory management, or more precisely, it's issues with fragmentation in SPIRAM. Duktape is probably one of the main fragmentation drivers due to the nature of Javascript memory management (garbage collection).
There has been some discussion on this by neoniousTR, who also was involved in the SPIRAM bug hunt: https://www.esp32.com/viewtopic.php?t=8628
The dlmalloc adaptation of neoniousTR unfortunately isn't usable for us, as it exceeds our available IRAM. It also was based on a much earlier esp-idf release, and doesn't support the extensions and options we use.
Espressif have introduced a new memory manager to address these issues, but only in esp-idf release 4.3. We'll need to go there one day, but that's nothing we can do short term: https://github.com/espressif/esp-idf/releases/tag/v4.3 → "Heap: Switched heap algorithm to one based on TLSF, improves performance especially when using a high number of allocations in PSRAM"
So I've tried another option: I've removed the Duktape heap from the standard system memory management. I've added a separate umm_malloc instance just for the heap, which can work very fast because it doesn't need to take care of locking or poisoning.
On Duktape startup, a fixed amount of SPIRAM gets allocated for the Javascript heap. The amount is 512 KB by default, which should be sufficient for most cases. I've added a configuration for this, config module duktape.heapsize, and a meminfo command to provide some insight.
The results are:
* Overall Duktape speedup of at least factor 3, up to factor 8 * Garbage collection runs now need below 100 ms even with large scripts running * Duktape average CPU usage dropped to 1-2% * Barely noticeable overall performance degradation after a week of continuous operation
Of course, your mileage may vary, feedback is welcome.
It still degrades, which is kind of expected, as Duktape isn't the only system component doing dynamic memory allocations.
I think esp-idf 4.3 may offer a significant overall performance increase for us, just by replacing the current memory management by a more sophisticated implementation.
As a side effect, JSON encoding is now much faster (by up to factor 8). CBOR has become only about 30% faster, which is probably due to CBOR needing much less memory allocations. CBOR is still factor 3 faster on average than JSON, so still the preferred choice for storage & transmission.
Details & code: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/1bcc...
Note: there are three new Kconfig variables that need to be set to enable the new Duktape memory management. The sdkconfig.default.hw31 includes the new configuration, if doing a "make", simply accept all defaults.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Branch for-v3.3 now includes the Duktape umm heap update. Regards, Michael Am 04.09.21 um 22:39 schrieb Michael Balzer:
Probably best to get for-v3.3 including the SIM7600 support stable and merged into master first. Mark, I'll merge the heap change into for-v3.3 next.
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Thanks for this, Michael. Most helpful. Mark.
On 7 Sep 2021, at 11:29 PM, Michael Balzer <dexter@expeedo.de> wrote:
Signed PGP part Branch for-v3.3 now includes the Duktape umm heap update.
Regards, Michael
Am 04.09.21 um 22:39 schrieb Michael Balzer:
Probably best to get for-v3.3 including the SIM7600 support stable and merged into master first. Mark, I'll merge the heap change into for-v3.3 next.
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
participants (4)
-
Craig Leres -
Mark Webb-Johnson -
Michael Balzer -
Stephen Casner