[Ovmsdev] 2019 / scripting framework
Mark Webb-Johnson
mark at webb-johnson.net
Thu Jan 17 09:21:48 HKT 2019
I debugged the memory leak. Whenever I call the PubSub.publish function, Duktape allocates about 288bytes of SPIRAM, and that grows with each call. But, every minute or so, a bunch of RAM gets magically freed. So, I tried:
OVMS# module memory
OVMS# script eval 'PubSub.publish("test","test");’
OVMS# script eval 'PubSub.publish("test","test");’
OVMS# script eval 'PubSub.publish("test","test");’
OVMS# script eval 'PubSub.publish("test","test");’
OVMS# script eval 'PubSub.publish("test","test");’
OVMS# module memory
OVMS# script eval 'Duktape.gc()’
OVMS# module memory
and no overall memory leak. The Duktape.gc() freed all the allocations. The issue is just general delayed memory garbage collection within Duktape.
Over time, relying only on automatic garbage collection (which seems to be every minute or so), it looks like this:
OVMS# module memory
Free 8-bit 79168/230296, 32-bit 900/2808, SPIRAM 3993620/4194252
--Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM
OVMS DukTape 188 0 0 29844 +0 +0 +0 +864
OVMS DukTape 188 0 0 30996 +0 +0 +0 +1152
OVMS DukTape 188 0 0 35640 +0 +0 +0 +4644
OVMS DukTape 188 0 0 38808 +0 +0 +0 +3168
OVMS DukTape 188 0 0 40824 +0 +0 +0 +2016
OVMS DukTape 188 0 0 15668 +0 +0 +0 -25156
OVMS DukTape 188 0 0 17756 +0 +0 +0 +2088
I don’t see the base SPIRAM memory allocation changing much, even after running for a couple of hours. I just see the SPIRAM rise up to 40KB, then drop down to 16KB, repeatedly. Anyway, I added ’script compact’ to be able to manually force a garbage collection (equivalent to "script eval 'Duktape.gc()’”).
OVMS# module memory
Free 8-bit 79168/230296, 32-bit 900/2808, SPIRAM 3991068/4194252
--Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM
OVMS DukTape 188 0 0 31636 +0 +0 +0 +13880
OVMS# script compact
Compacting javascript memory
I (263066) script: Duktape: Compacting DukTape memory
OVMS# module memory
Free 8-bit 79168/230296, 32-bit 900/2808, SPIRAM 4014624/4194252
--Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM
OVMS DukTape 188 0 0 15092 +0 +0 +0 -16544
I also included Michael’s JSON.js module as an internal module, as it is pretty useful. Now that I’ve implemented two of these, I know how to do it and will write some better support utility to make including these easier. I used https://javascript-minifier.com <https://javascript-minifier.com/> to reduce the code size (the original module code is in the jsmodsrc directory of ovms_script component, and minimised version in jsmodembed ready for embedding into firmware). Now, after boot you can simply call:
OVMS# script eval 'JSON.print(this);'
{
"OvmsCommand": function () { [native code] },
"OvmsLocationStatus": function () { [native code] },
"OvmsMetricFloat": function () { [native code] },
"OvmsMetricValue": function () { [native code] },
"assert": function () { [native code] },
"print": function () { [native code] },
"PubSub": {
"publish": function () { [ecmascript code] },
"subscribe": function () { [ecmascript code] },
"clearAllSubscriptions": function () { [ecmascript code] },
"clearSubscriptions": function () { [ecmascript code] },
"unsubscribe": function () { [ecmascript code] }
},
"JSON": {
"print": function () { [ecmascript code] }
}
}
Scripting from /store/scripts works well, and the ovmsmain.js is loaded on startup (AutoInit). Michael’s web based editor also works well for this.
OVMS# vfs cat /store/scripts/ovmsmain.js
print("Hello world\n");
With events now enabled, you can use the PubSub framework to subscribe to events, and have the code run persistently (most importantly with persistent state). Much better than having to evaluate fire-and-forget scripts off flash. The usual place would be in ovmsmain.js, but you can also modify the live running environment directly:
OVMS# script eval 'var myTicker=function(msg,data){print("Event: "+msg+"\n");}; PubSub.subscribe("ticker.10",myTicker);'
I (950636) script: Event: ticker.10
I (960636) script: Event: ticker.10
I (970636) script: Event: ticker.10
An example would be to create a function ‘arrivedHome()’ then PubSub.subscribe(‘location.enter.homeloc’,arrivedHome). Within arrivedHome() you could do whatever is required.
Another example would be to create a function ‘chargingReady()’ and hook it into the vehicle.charge.pilot.on event. Then, when the pilot signal appears, the function could call OvmsLocationStatus() to see if you are home, and act appropriately.
The ticker.* events can be used to periodically poll for things.
Perhaps a javascript function to check if we have been home for an hour, but wifi hasn’t connected, so power off/on the wifi?
The engine itself is pretty much feature complete and usable at the moment. If you change the ovmsmain.js, or just want to clear out the environment, you can ’script reload’ and everything is restored to what it was at boot.
Work now needs to be done on the support modules, to make things easier. Sure, we can call OvmsCommand(‘lock 1234’) to lock the car, but it would be much cleaner to have a vehicle support module so we can just call vehicle.lock(‘1234’). Other parts of the system also need to be extended into this, for things like notifications. Individual vehicles can also extend their own functionality. This all runs in SPIRAM, so memory should not be an issue.
Looking in the extras/console directory of Duktape source tree, there is an example for a pure C implementation of a console object. That is exactly what we want - an object with property functions that can be called. My next job is to add support for that, similar to our existing RegisterDuktapeFunction. Then we can implement objects for vehicles, metrics, etc.
Regards, Mark
> On 16 Jan 2019, at 3:22 PM, Mark Webb-Johnson <mark at webb-johnson.net> wrote:
>
> Well, the javascript framework re-work has now been committed, and is working well for me. Only one more major outstanding TODO remaining. Some comments:
>
> To do anything useful with modules, I had to increase the stack size. It is now 12,288 bytes. I’ve seen just under 10KB in normal use, so a few more KB added as a margin. It won’t work with the old 8192 byte default. At least it doesn’t use IRAM, and the majority of storage (code, execution stack, etc) is in SPIRAM.
>
> Duktape now starts up as part of the normal AutoInit system (so if it does crash, at least the rest of the system will continue to run).
>
> Modules have been fully implemented. Very nice.
>
> An internal module system has been implemented, and a first module 'int/pubsub' provided embedded in the firmware.
>
> Breaking Change: The old ’script <path>’ command has changed to ’script run <path>’. The ‘.’ alternative is unchanged.
>
> A ’script reload’ command has been provided to reload the framework.
>
> A ’script eval’ command has been provided as a quick-and-dirty javascript evaluator (for testing purposes).
>
> TODO: There is a memory leak in the PubSub event framework, so I’ve disabled it for the moment.
>
> For example:
>
> OVMS# script eval 'JSON=require("lib/JSON");'
> OVMS# script eval 'JSON.print(this)'
> {
> "OvmsCommand": function () { [native code] },
> "OvmsLocationStatus": function () { [native code] },
> "OvmsMetricFloat": function () { [native code] },
> "OvmsMetricValue": function () { [native code] },
> "assert": function () { [native code] },
> "print": function () { [native code] },
> "PubSub": {
> "publish": function () { [ecmascript code] },
> "subscribe": function () { [ecmascript code] },
> "clearAllSubscriptions": function () { [ecmascript code] },
> "clearSubscriptions": function () { [ecmascript code] },
> "unsubscribe": function () { [ecmascript code] }
> },
> "JSON": {
> "print": function () { [ecmascript code] }
> }
>
> OVMS# script reload
> Reloading javascript engine
> I (277675) script: Duktape: Clearing existing context
> I (277685) script: Duktape: Creating heap
> I (277685) script: Duktape: Initialising module system
> I (277685) script: Duktape: Pre-Registered function OvmsCommand
> I (277685) script: Duktape: Pre-Registered function OvmsLocationStatus
> I (277685) script: Duktape: Pre-Registered function OvmsMetricFloat
> I (277685) script: Duktape: Pre-Registered function OvmsMetricValue
> I (277685) script: Duktape: Pre-Registered function assert
> I (277685) script: Duktape: Pre-Registered function print
> I (277685) script: Duktape: Preload internal module PubSub
> I (277825) script: Duktape: Executing ovmsmain.js
> I (277835) script: Hello world
>
> Still a bunch to do, but the basics seem there and working well.
>
> Regards, Mark.
>
>> On 11 Jan 2019, at 9:23 AM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>
>> Your example is a little more involved than my ‘hello world’, and much cooler. Perhaps Robert can use it as a starting point in the user guide.
>>
>> Any ideas how to handle events? We need an event module in javascript, and some way for extension scripts/modules to hook into the reception of events? The events are already routed into the duktape task - just not decided what to do with them yet. This leads into the bigger question of a server-side framework to run in the Duktape task. I can see some common pub/sub frameworks:
>>
>> PubSub.JS <https://github.com/mroderick/PubSubJS>
>> Subtopic <https://github.com/pmelander/Subtopic>
>> Amplify JS <http://amplifyjs.com/api/pubsub/>
>> Postal JS <https://github.com/postaljs/postal.js>
>>
>> The last one (postal js) seems a good balance of complexity vs functionality, but goes beyond the minimum we need and may be too big. The first one is the absolute minimum, and the compressed (min) version is <2KB.
>>
>> I guess the bigger question is does anyone know of any very lightweight server side frameworks that we should be looking at (rather than rolling our own - aka OpenVehicles.JS)?
>>
>> It also would seem to make more sense to have the C interfaces as module objects with methods. So, rather than calling OvmsMetricFloat(), you get a handle to a metric object and then call a method to obtain its float value. Not sure how Duktape handles that, but I will check. Having a rich set of interfaces to our objects would be very useful and clean. Another example would be the vehicle object with methods StartCharge, StopCharge, UnlockCar, etc - rather than calling OvmsCommand(“charge start").
>>
>> Regards, Mark.
>>
>>> On 11 Jan 2019, at 12:38 AM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>
>>> Works like a charm :)
>>>
>>> I've added the autoinit option to the web UI.
>>>
>>> Important for all script writers is to keep in mind we now have a global shared context.
>>>
>>> That is very useful to share data between scripts, but needs a little bit of discipline to avoid polluting the global context with local variables and functions. Any "var", "const", "function" etc. placed into the top level of a script file now is added to the global context.
>>>
>>> The best approach is to encourage users to use closures by default for all their scripts, i.e.
>>> (function(){
>>> … user code …
>>> })();
>>> The same rule applies to web plugins.
>>>
>>> Node style modules also work nicely. A simple test module:
>>> // Module : JSON (not compatible with the browser component!)
>>> // State : test/demo
>>> // Install as : /store/scripts/lib/JSON.js
>>> // Load : JSON = require("lib/JSON");
>>> // Use : JSON.print(object);
>>>
>>> exports.print = function(obj, ind) {
>>> var type = typeof obj;
>>> if (type == "object" && Array.isArray(obj)) type = "array";
>>> if (!ind) ind = '';
>>>
>>> switch (type) {
>>> case "string":
>>> print('"' + obj.replace(/\"/g, '\\\"') + '"');
>>> break;
>>> case "array":
>>> print('[\n');
>>> for (var i = 0; i < obj.length; i++) {
>>> print(ind + ' ');
>>> exports.print(obj[i], ind + ' ');
>>> if (i != obj.length-1) print(',');
>>> print('\n');
>>> }
>>> print(ind + ']');
>>> break;
>>> case "object":
>>> print('{\n');
>>> var keys = Object.keys(obj);
>>> for (var i = 0; i < keys.length; i++) {
>>> print(ind + ' "' + keys[i] + '": ');
>>> exports.print(obj[keys[i]], ind + ' ');
>>> if (i != keys.length-1) print(',');
>>> print('\n');
>>> }
>>> print(ind + '}');
>>> break;
>>> default:
>>> print(obj);
>>> }
>>>
>>> if (ind == '') print('\n');
>>> }
>>>
>>> Using it:
>>> JSON = require("lib/JSON");
>>>
>>> print("Global context:\n");
>>> JSON.print(this);
>>>
>>> (function(){
>>> var x = { a: 42, b: "a \"foo\" is no 'bar'", c: [1,2,3], d: { sub: true }, e: ["q","w",17, { e1:22, e2:33 }] };
>>> print("\nLocal object:\n");
>>> JSON.print(x);
>>> })();
>>>
>>> Execution:
>>> OVMS# . test.js
>>> Global context:
>>> {
>>> "print": function () { [native code] },
>>> "assert": function () { [native code] },
>>> "OvmsMetricValue": function () { [native code] },
>>> "OvmsMetricFloat": function () { [native code] },
>>> "OvmsLocationStatus": function () { [native code] },
>>> "OvmsCommand": function () { [native code] },
>>> "JSON": {
>>> "print": function () { [ecmascript code] }
>>> }
>>> }
>>>
>>> Local object:
>>> {
>>> "a": 42,
>>> "b": "a \"foo\" is no 'bar'",
>>> "c": [
>>> 1,
>>> 2,
>>> 3
>>> ],
>>> "d": {
>>> "sub": true
>>> },
>>> "e": [
>>> "q",
>>> "w",
>>> 17,
>>> {
>>> "e1": 22,
>>> "e2": 33
>>> }
>>> ]
>>> }
>>>
>>> Nice :)
>>>
>>> Regards,
>>> Michael
>>>
>>>
>>> Am 10.01.19 um 10:10 schrieb Mark Webb-Johnson:
>>>> I’ve committed my work on a production javascript framework. The changes are quite extensive, and with the one exception of ‘ovmsprint’, should be backwards compatible.
>>>>
>>>> The overall goal is to be able to run a javascript code system alongside our standard firmware code. Users could then write javascript, and modify scripts already running on the device (without firmware changes). Unlike the previous version (where scripts were loaded, compiled, run, then dumped), this approach is designed to keep scripts in memory; this vastly increases the opportunities for what can be done with the system.
>>>>
>>>> Changes made include:
>>>>
>>>> Move scripting to be a component
>>>> Make DukTape use SPIRAM for as much as possible
>>>> Run the javascript engine in it's own task (OVMS DukTape)
>>>> Change the way extensions functions are registered
>>>> (just call RegisterDuktapeFunction, rather than messing around with the internals of Duktape)
>>>> Catch compilation and parsing errors (fail gracefully - finally!)
>>>> Support 'print' and 'assert' javascript framework
>>>> Output (via print) goes to current console, or logged if no console
>>>> Autoinit (and run) /store/scripts/ovmsmain.js
>>>> Support node.js style modules
>>>> Control with config auto javascript (default enable)
>>>>
>>>> The main TODO is to provide a facility for reloading the engine (new scripts/modules). At the moment, the module needs to be rebooted (ugly). There is also a lot of work still to do on making some actually useful modules. And we also need to work out what to do with events (now that they can be delivered to running javascripts).
>>>>
>>>> But, anyway, for the moment:
>>>>
>>>> I (128) script: Initialising SCRIPTS (1600)
>>>> I (132) script: Using DUKTAPE javascript engine
>>>> I (0) script: Duktape: Creating heap
>>>> I (10) script: Duktape: Initialising module system
>>>> I (20) script: Duktape: Scripting task is running
>>>> I (906) script: Duktape: Executing ovmsmain.js
>>>> I (936) script: Hello world
>>>>
>>>> OVMS# vfs cat /store/scripts/ovmsmain.js
>>>> print("Hello world\n”);
>>>>
>>>> OVMS# test javascript
>>>> Javascript 1+2=3
>>>>
>>>> Regards, Mark.
>>>>
>>>>> On 8 Jan 2019, at 11:00 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>>
>>>>> Mark,
>>>>>
>>>>> welcome back, awesome news on the JS and FCC stuff :)
>>>>>
>>>>> I've got some fixes and some feedback on the web plugins in progress, I'll get that done until the weekend (got some free days left).
>>>>>
>>>>> Regards,
>>>>> Michael
>>>>>
>>>>>
>>>>> Am 08.01.19 um 05:27 schrieb Mark Webb-Johnson:
>>>>>> Firstly, a very belated merry xmas, and happy 2019 to everyone. I hope that you all got to spend good times with friends and family, as I did. I’ve been travelling, and just got back home to a very large eMail inbox.
>>>>>>
>>>>>> During my travels, I did manage to re-work the javascript framework to be single task based. That seems to work well, and now I’m back I will test on my car. The new framework is backwards compatible - apart from memory usage (one more task with a stack), it has no impact to the flow. It will, however, allow us to extend this a lot more and become much more useful. Assuming no issues, I will be able to commit that soon.
>>>>>>
>>>>>> As we are up to 144 commits since 3.1.011, perhaps it is well past due for 3.1.012? I was hoping that Espressif would have fixed the bug related to wear levelling version upgrades that Michael helped to identify, but not yet. Any objections to a 3.1.012 release to EAP at the end of this week?
>>>>>>
>>>>>> The CE/FCC certification work is progressing. No issues so far. I’ll let you know as soon as we get near completion for this.
>>>>>>
>>>>>> I’m working through my inbox now, and will reply individually to questions there.
>>>>>>
>>>>>> Regards, Mark.
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>>
>>>>> --
>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>
>>> --
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20190117/f21f090c/attachment.htm>
More information about the OvmsDev
mailing list