[Ovmsdev] Duktape heap memory management

[Michael Balzer]

Everyone,

the poor JSON encoding performance reminded me of another issue we've 
had with Duktape since integration: the overall slowly degrading 
performance of the whole system over time.

I did some new analysis on this effect using the task monitoring, which 
showed it's become worse lately up to the point where Duktape needs more 
than a second to perform the garbage collection after just 24 hours of 
uptime. It turned out this effect is also one of the major causes for 
the very bad JSON encoding performance, i.e. needing up to 10 seconds 
for a simple object of 5 x 1440 values, and eating 15-20% of CPU time on 
a permanent base (instead of the tolerable 5-6% right after boot).

Doing a "script reload" would help, but only temporarily and not near 
the point of a freshly booted system.

I didn't find anything about such an effect for Duktape in general, so I 
turned to examining possible ESP32 specific causes, and finally found 
the culprit: the esp-idf memory management, or more precisely, it's 
issues with fragmentation in SPIRAM. Duktape is probably one of the main 
fragmentation drivers due to the nature of Javascript memory management 
(garbage collection).

There has been some discussion on this by neoniousTR, who also was 
involved in the SPIRAM bug hunt: https://www.esp32.com/viewtopic.php?t=8628

The dlmalloc adaptation of neoniousTR unfortunately isn't usable for us, 
as it exceeds our available IRAM. It also was based on a much earlier 
esp-idf release, and doesn't support the extensions and options we use.

Espressif have introduced a new memory manager to address these issues, 
but only in esp-idf release 4.3. We'll need to go there one day, but 
that's nothing we can do short term: 
https://github.com/espressif/esp-idf/releases/tag/v4.3 → "Heap: Switched 
heap algorithm to one based on TLSF, improves performance especially 
when using a high number of allocations in PSRAM"

So I've tried another option: I've removed the Duktape heap from the 
standard system memory management. I've added a separate umm_malloc 
instance just for the heap, which can work very fast because it doesn't 
need to take care of locking or poisoning.

On Duktape startup, a fixed amount of SPIRAM gets allocated for the 
Javascript heap. The amount is 512 KB by default, which should be 
sufficient for most cases. I've added a configuration for this, config 
module duktape.heapsize, and a meminfo command to provide some insight.

The results are:

  * Overall Duktape speedup of at least factor 3, up to factor 8
  * Garbage collection runs now need below 100 ms even with large
    scripts running
  * Duktape average CPU usage dropped to 1-2%
  * Barely noticeable overall performance degradation after a week of
    continuous operation

Of course, your mileage may vary, feedback is welcome.

It still degrades, which is kind of expected, as Duktape isn't the only 
system component doing dynamic memory allocations.

I think esp-idf 4.3 may offer a significant overall performance increase 
for us, just by replacing the current memory management by a more 
sophisticated implementation.

As a side effect, JSON encoding is now much faster (by up to factor 8). 
CBOR has become only about 30% faster, which is probably due to CBOR 
needing much less memory allocations. CBOR is still factor 3 faster on 
average than JSON, so still the preferred choice for storage & transmission.

Details & code: 
https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/1bcc8dd84405df81e61577a90909e7b43c25baa3

Note: there are three new Kconfig variables that need to be set to 
enable the new Duktape memory management. The sdkconfig.default.hw31 
includes the new configuration, if doing a "make", simply accept all 
defaults.

Regards,
Michael

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210904/f598262b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210904/f598262b/attachment.sig>