[Ovmsdev] Duktape heap memory management

Sun Sep 5 08:41:29 HKT 2021

I agree. Let’s get 3.3 out the door.

Any change to IDF 4.x will require a freeze on changes to the core frameworks we have while we make that change. It will certainly be easier if we have a few people helping out with this (in particular where we have changed/extended the Espressif stuff). I have high hopes that this will improve stability and performance, as we are so far behind Espressif now.

Another issue to be addressed is flash size. Currently we have 4MB factory, 4MB OTA0, 4MB OTA1, and the rest for /store, etc. Given our image is now 3.5MB, we are going to hit these limits sometime soon. An alternative arrangement is to just switch to two firmware partitions of 6GB. Current code runs from one, with new firmware updates downloaded to the other. We could do this upgrade in-place for existing users.

Regards, Mark.

> On 5 Sep 2021, at 4:39 AM, Michael Balzer <dexter at expeedo.de> wrote:
> 
>  Craig, you're probably qualified best to answer this, as you already did some steps to port to idf 4.0?
> 
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/360
> 
> But I guess there have been quite some changes since 4.0 also. We'll have to try and solve the issues one by one.
> 
> Probably best to get for-v3.3 including the SIM7600 support stable and merged into master first. Mark, I'll merge the heap change into for-v3.3 next.
> 
> Regards,
> Michael
> 
> 
> Am 04.09.21 um 21:52 schrieb Stephen Casner:
>> Michael,
>> 
>> Thanks for performing this diagnosis and enhancement.
>> 
>> Has anyone analyzed the work required to move to current esp-idf?  How
>> many things will break?  I imagine the heap instrumentation I added
>> would be one of them.  Perhaps we could divide up that work among
>> several developers.  I would likely need to get another module,
>> though, because my current v3.0 bench unit without SPIRAM probably
>> wouldn't be usable.
>> 
>>                                                         -- Steve
>> 
>> On Sat, 4 Sep 2021, Michael Balzer wrote:
>> 
>>> Everyone,
>>> 
>>> the poor JSON encoding performance reminded me of another issue we've had with
>>> Duktape since integration: the overall slowly degrading performance of the
>>> whole system over time.
>>> 
>>> I did some new analysis on this effect using the task monitoring, which showed
>>> it's become worse lately up to the point where Duktape needs more than a
>>> second to perform the garbage collection after just 24 hours of uptime. It
>>> turned out this effect is also one of the major causes for the very bad JSON
>>> encoding performance, i.e. needing up to 10 seconds for a simple object of 5 x
>>> 1440 values, and eating 15-20% of CPU time on a permanent base (instead of the
>>> tolerable 5-6% right after boot).
>>> 
>>> Doing a "script reload" would help, but only temporarily and not near the
>>> point of a freshly booted system.
>>> 
>>> I didn't find anything about such an effect for Duktape in general, so I
>>> turned to examining possible ESP32 specific causes, and finally found the
>>> culprit: the esp-idf memory management, or more precisely, it's issues with
>>> fragmentation in SPIRAM. Duktape is probably one of the main fragmentation
>>> drivers due to the nature of Javascript memory management (garbage
>>> collection).
>>> 
>>> There has been some discussion on this by neoniousTR, who also was involved in
>>> the SPIRAM bug hunt: https://www.esp32.com/viewtopic.php?t=8628
>>> 
>>> The dlmalloc adaptation of neoniousTR unfortunately isn't usable for us, as it
>>> exceeds our available IRAM. It also was based on a much earlier esp-idf
>>> release, and doesn't support the extensions and options we use.
>>> 
>>> Espressif have introduced a new memory manager to address these issues, but
>>> only in esp-idf release 4.3. We'll need to go there one day, but that's
>>> nothing we can do short term:
>>> https://github.com/espressif/esp-idf/releases/tag/v4.3 → "Heap: Switched heap
>>> algorithm to one based on TLSF, improves performance especially when using a
>>> high number of allocations in PSRAM"
>>> 
>>> So I've tried another option: I've removed the Duktape heap from the standard
>>> system memory management. I've added a separate umm_malloc instance just for
>>> the heap, which can work very fast because it doesn't need to take care of
>>> locking or poisoning.
>>> 
>>> On Duktape startup, a fixed amount of SPIRAM gets allocated for the Javascript
>>> heap. The amount is 512 KB by default, which should be sufficient for most
>>> cases. I've added a configuration for this, config module duktape.heapsize,
>>> and a meminfo command to provide some insight.
>>> 
>>> The results are:
>>> 
>>>  * Overall Duktape speedup of at least factor 3, up to factor 8
>>>  * Garbage collection runs now need below 100 ms even with large
>>>    scripts running
>>>  * Duktape average CPU usage dropped to 1-2%
>>>  * Barely noticeable overall performance degradation after a week of
>>>    continuous operation
>>> 
>>> Of course, your mileage may vary, feedback is welcome.
>>> 
>>> It still degrades, which is kind of expected, as Duktape isn't the only system
>>> component doing dynamic memory allocations.
>>> 
>>> I think esp-idf 4.3 may offer a significant overall performance increase for
>>> us, just by replacing the current memory management by a more sophisticated
>>> implementation.
>>> 
>>> As a side effect, JSON encoding is now much faster (by up to factor 8). CBOR
>>> has become only about 30% faster, which is probably due to CBOR needing much
>>> less memory allocations. CBOR is still factor 3 faster on average than JSON,
>>> so still the preferred choice for storage & transmission.
>>> 
>>> Details & code:
>>> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/1bcc8dd84405df81e61577a90909e7b43c25baa3
>>> 
>>> Note: there are three new Kconfig variables that need to be set to enable the
>>> new Duktape memory management. The sdkconfig.default.hw31 includes the new
>>> configuration, if doing a "make", simply accept all defaults.
>>> 
>>> Regards,
>>> Michael
>>> 
>>> --
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>> 
>>> 
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> 
> -- 
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210905/254500cb/attachment.htm>