[Ovmsdev] RETOOLS web

Mark Webb-Johnson mark at webb-johnson.net
Mon Feb 25 21:18:35 HKT 2019


Michael,

Looks good:



I played a roadster crtd log back through it, using a DBC to control the message multiplexing. No crashes.

I then added this live:

BO_ 836 TPMS: 8 TPMS
  SG_ v_tp_fl_p : 0|8 at 0+ (0.362976,0) [0|1000] "psi" TPMS
  SG_ v_tp_fr_p : 16|8 at 0+ (0.362976,0) [0|1000] "psi" TPMS
  SG_ v_tp_rl_p : 32|8 at 0+ (0.362976,0) [0|1000] "psi" TPMS
  SG_ v_tp_rr_p : 48|8 at 0+ (0.362976,0) [0|1000] "psi" TPMS
  SG_ v_tp_fl_t : 8|8 at 0+ (1,-40) [-40|100] "C" TPMS
  SG_ v_tp_fr_t : 24|8 at 0+ (1,-40) [-40|100] "C" TPMS
  SG_ v_tp_rl_t : 40|8 at 0+ (1,-40) [-40|100] "C" TPMS
  SG_ v_tp_rr_t : 56|8 at 0+ (1,-40) [-40|100] "C" TPMS

And got this:

key records ms last
can1/344 24 583 57 48 56 49 75 4d 76 4e | WHVIuMvN
dbc/v_tp_fl_p: 31.5789 psi
dbc/v_tp_fr_p: 31.2159 psi
dbc/v_tp_rl_p: 42.4682 psi
dbc/v_tp_rr_p: 42.8312 psi
dbc/v_tp_fl_t: 32 C
dbc/v_tp_fr_t: 33 C
dbc/v_tp_rl_t: 37 C
dbc/v_tp_rr_t: 38 C

That looks good.

Only issue so far is that when I switch to page #2, it immediately switches back to page #1. I guess it thinks there is new data, but the link was idle at the time. 

D (2097680) notify: Created entry type 'stream' subtype 'retools.status' size 269 has 1 readers pending
D (2097680) notify: Created entry type 'stream' subtype 'retools.list.update' size 3 has 1 readers pending
D (2098680) notify: Created entry type 'stream' subtype 'retools.status' size 269 has 1 readers pending
D (2098690) notify: Created entry type 'stream' subtype 'retools.list.update' size 3 has 1 readers pending
D (2099680) notify: Created entry type 'stream' subtype 'retools.status' size 269 has 1 readers pending
D (2099680) notify: Created entry type 'stream' subtype 'retools.list.update' size 3 has 1 readers pending

It seems re_stream_changed always sends “[]”, even if no changes. Easy to fix, but perhaps better to address in the Javascript.

I’m going to continue with the live DBC editing commands now. That is the pain point for using this in the real world (as the current approach is onerous:  (a) detaching the DBC from the CAN bus, (b) switching to editor, (c) editing the DBC, (d) switch to command and ‘dbc autoload’, then (e) re-attaching the DBC to the CAN bus). It can, however, all be done from the web interface. Also building up DBC files for Tesla Roadster, to see just how much can be done in that format. I think I also need to get my DBC implementation to support strings.

Regards, Mark.

> On 24 Feb 2019, at 9:38 PM, Michael Balzer <dexter at expeedo.de> wrote:
> 
> Mark,
> 
> the crash was no memory corruption, it was a race condition in the RE init (the queue was created after the task).
> 
> DataTables integration and the first draft of the RE tools web UI is now in the hub. It's already usable but far from finished, so I didn't include it as a standard page. Install "web/retools.htm" as a plugin to use & work on it.
> 
> The layout is following your draft, and I've added some rules for small devices.
> 
> I've added some control buttons and status output, the table stream updates and a detail call on row clicks. Btw: it may be better to send detail info to the browser using the stream updates (for example in hidden columns), so the browser doesn't need to request them.
> 
> The table is mostly standard DataTable, I've only enabled the responsive plugin, that does dynamic hiding & drilldown of columns. I've included most of the plugins, so there are many additional options:
> Included libraries:
> DataTables 1.10.18, Buttons 1.5.4, Column visibility 1.5.4, ColReorder 1.5.0, FixedColumns 3.2.5, FixedHeader 3.1.4, KeyTable 2.5.0, Responsive 2.2.2, RowGroup 1.1.0, RowReorder 1.2.4, Scroller 1.5.0, Select 1.2.6
> see https://datatables.net/extensions/index <https://datatables.net/extensions/index>
> The web UI listens to stream updates for the table and the text displays using this scheme:
> Subtype "retools/list/set" → replace list by payload
> Subtype "retools/list/update" → update/extend list by payload
> Subtype "retools/<boxid>" → set payload as text content in box <boxid>, boxid e.g. "status", "info" or "extra"
> Example: notify raise text stream retools/extra "hi there"
> 
> Text boxes are currently <samp> with monospace font & text to html conversion. If you'd like to display HTML, replace "$('<div/>').text(msg.value).html()" by just "msg.value" on line 175.
> 
> Table updates are sent as JSON arrays with HTML markup for the changed/discovered highlighting. This currently uses <b> & <u>, which are styled accordingly. Command "re stream list" creates a message with all rows (no highlighting), "re stream changed" includes only changed rows (with highlighting).
> 
> I've added a first stream sender on the per second ticker. That currently simply sends all changed rows and could be optimized to only send those changed since the last tx. It could also make sense to raise the update frequency then.
> 
> The status box monitors some "retools" events I introduced, the stream ticker also updates the box -- if the stream subscription & updating is going to be permanent (as now), the event coupling can be removed.
> 
> Regards,
> Michael
> 
> 
> Am 23.02.19 um 10:18 schrieb Michael Balzer:
>> Side note: starting the RE server from the web shell currently consistently results in a crash from xQueueReceive() that looks like our "old friend", the memory corruption.
>> 
>> Starting from the async console or ssh or via MP works. No idea yet how that could make a difference for xQueueReceive(), I'll assume it's the PSRAM bug for now.
>> 
>> 
>> D (38792) webcommand: HttpCommandStream[0x3fa2c358] init: handler=0x3fa2f6a0 command='re start' verbosity=65535
>> I (38812) webcommand: HttpCommandStream[0x3fa2c358]: 1996204 bytes free, executing: re start
>> assertion "res == coreID || res == portMUX_FREE_VAL" failed: file "/home/balzer/esp/esp-idf/components/freertos/portmux_impl.inc.h", line 105, function: vPortCPUAcquireMutexIntsDisabledExtram
>> abort() was called at PC 0x40112f67 on core 1
>> 
>> Backtrace: 0x40091188:0x3ffea650 0x40091383:0x3ffea670 0x40112f67:0x3ffea690 0x40093ce3:0x3ffea6c0         0x40095ac0:0x3ffea6f0 0x4016c08a:0x3ffea730 0x4016c0c1:0x3ffea770
>> 
>> Rebooting...
>> 
>> 
>> 
>> Am 16.02.19 um 14:00 schrieb Mark Webb-Johnson:
>>> Thanks, Michael.
>>> 
>>> I can handle the RE side of things - just haven’t done a lot of work in JavaScript (too many people better at it than me at my day job, so I leave it to them). I can also probably extend it once the basics are working.
>>> 
>>> Just need a basic UI framework, and some way to push the data from RE.
>>> 
>>> No hurry.
>>> 
>>> Regards, Mark
>>> 
>>> On 16 Feb 2019, at 8:43 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>> 
>>>> Mark,
>>>> 
>>>> I haven't seen the necessity for a complex plugin just for table views up to now. Normally it's pretty simple to make dynamic tables just using HTML5 and jQuery. For example, see the file selector widget I created. That uses a table to display the file entries, it's dynamic, sortable & responsive with just a few lines of code.
>>>> 
>>>> DataTables adds a lot of features and simplifies manipulations of the data. I like the automatic responsive drilldown shown here:
>>>> https://datatables.net/extensions/responsive/examples/styling/bootstrap.html <https://datatables.net/extensions/responsive/examples/styling/bootstrap.html>
>>>> 
>>>> It's got some weight of course (about half of the charts lib with just the most basic extensions), so we would add it as an optional asset. On a first look it will need some customization for us, as it uses the glyphicons for the bootstrap 3 variant, which we don't include for size reasons.
>>>> 
>>>> What do others think? And which of the extensions should be included?
>>>> 
>>>> Overall, the UI as drafted shouldn't be hard to implement. I'll see if I can create a first version, but probably won't have the time before next weekend.
>>>> 
>>>> Regards,
>>>> Michael
>>>> 
>>>> 
>>>> Am 13.02.19 um 13:05 schrieb Mark Webb-Johnson:
>>>>> 
>>>>> Michael, this is probably for you - but if anybody else can help please do…
>>>>> 
>>>>> I am trying to see how hard it would be to do a web based interface to the RETOOLS system. Presumably this would appear in the menu as Tools / Reverse Engineering (or some such abbreviation if that is too long). The basic screen would look something like this:
>>>>> 
>>>>> +----------------------------------------------------------------------------------------------------+
>>>>> |                                                                                                    |
>>>>> | +----------+  +------+-------+------------------------------------+------+  +--------------------+ |
>>>>> | |          |  | hdr1 |  hdr2 |                  hdr3              | hdr4 |  |                    | |
>>>>> | |          |  +------+-------+------------------------------------+------+  |                    | |
>>>>> | | Controls |  |      |       |                                    |      |  |        Info        | |
>>>>> | |          |  | col1 |  col2 |                  col3              | col4 |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | +----------+  |      |       |                                    |      |  +--------------------+ |
>>>>> |               |      |       |                                    |      |                         |
>>>>> | +----------+  |      |       |                                    |      |  +--------------------+ |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |  Status  |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |             TABLE                  |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |        Extra       | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | |          |  |      |       |                                    |      |  |                    | |
>>>>> | +----------+  |      |       |                                    |      |  +--------------------+ |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               |      |       |                                    |      |                         |
>>>>> |               +------+-------+------------------------------------+------+                         |
>>>>> |                                                                                                    |
>>>>> +----------------------------------------------------------------------------------------------------+
>>>>> 
>>>>> The controls are just a set of controls - things that bring up dialogs for more information or perform actions. Like start/stop RE, switch modes, manipulate DBC files, etc. Left justified in the frame, and perhaps 10% of the width. Height can dynamically adjust with the text content.
>>>>> 
>>>>> The Status, Info, and Extra panes are just a dynamically updated text panes. Main code needs to be able to send text here and have it shown in the appropriate pane. Right justified in the frame and perhaps 30% of the width. Height can dynamically adjust with the text content.
>>>>> 
>>>>> The TABLE is a table. Probably 5 columns by many rows (anywhere from 20 to 200 or so). Either paged, scrolling, or just very tall. Paged is probably preferable. Would need a simple mechanism to dynamically add/update a row (identified by ‘key’ column and in sorted order), presumably using the web socket. Also need something to be able to dynamically clear the entire table. The RETOOLS code would use this to update the table in realtime (presumably over a web socket). It would be good if the table could do things like search filtering of displayed rows, column sorting, etc (all client side - something like datatables.net <http://datatables.net/>).
>>>>> 
>>>>> For example, look at this output:
>>>>> 
>>>>> OVMS# re dbc list
>>>>> key                     records     ms last
>>>>> can1/100:0081                 2  22000 01 02 03 04 05          | .....
>>>>> can1/101                      1  44000 01 02 03 04             | ....
>>>>>   dbc/DI_Signal1: 129 km
>>>>>   dbc/DI_Signal2: 1 km
>>>>>   dbc/DI_Signal3: 130 kWh
>>>>>   dbc/DI_Signal4: 122 Degrees
>>>>> 
>>>>> The idea is to be able to show the key, records, ms, last, and ascii dump in the main table. Then, clicking on a row would show the statistics for that message in the Info pane, and DBC signals in the Extra pane.
>>>>> 
>>>>> I can handle the RETOOLS side, but the javascript is beyond me. I also had a look at the ovms web server framework, but it doesn’t seem to be able to do the table part (yet)?
>>>>> 
>>>>> Any idea how complex this is?
>>>>> 
>>>>> Regards, Mark.
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>> 
>>>> -- 
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>> 
>>> 
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>> 
>> -- 
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>> 
>> 
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
> 
> -- 
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20190225/e68e53a1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-4.tiff
Type: image/tiff
Size: 76676 bytes
Desc: not available
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20190225/e68e53a1/attachment-0001.tiff>


More information about the OvmsDev mailing list