[Ovmsdev] Locations and scripts
Mark Webb-Johnson
mark at webb-johnson.net
Wed Jun 19 14:10:39 HKT 2019
I have committed and pushed what I have. It has not had extensive testing - for which I need to get it in some cars and get some real world data. A rough synopsis of what has been done follows:
A major set of extensions (and basic re-write) of the CAN logging framework.
The system is now split into three:
The logging framework itself
The formatters (to convert to/from formats like pcap, crtd, etc)
The loggers (to read/write to different locations such as monitor, vfs, tcp, etc)
Also standardised are the formatters (now shared between can logging and retools), and the filters (so software filters can be defined and handled in a standard way).
The tcpclient and tcpserver loggers have not yet been implemented. They are there purely as stubs at the moment.
This is a very large change, and has not had extensive testing yet. There will likely be a number of changes in the coming few days / weeks.
It has turned into a rather interesting framework, with quite a lot of overlap vs retools. I think we can move quite a bit that retools does into this, to simplify that code and let retools do just the analysis part (without worrying about collection data and moving it around the network). It will be interesting to see what happens once tcpclient and tcpserver get implemented (my next task, and much simpler than what has been done to date).
Regards, Mark.
> On 17 Jun 2019, at 5:08 PM, Mark Webb-Johnson <mark at webb-johnson.net> wrote:
>
>
> I’ve completed most of the work of this restructuring, and the new arrangement looks good. The new work uses CAN_log_message_t (rather than CAN_frame_t) for most things other than basic frame tx/rx, and that is based on Michael’s can log CAN_LogMsg_t (almost unchanged, apart from timestamp). I’ve added a canfilter class to be able to do software filtering (not just in logging, but also in general). Then candump* has moved to canformat* to virtualise all the different formats (CRTD, PCAP, etc) for can dumps (with a modular registration of formats so we can add more formats easily, without change to core components).
>
> The last change I have to make is to canlog. I want it to use the canfilter and canformat systems so all it is doing is logging (and not involved in filtering, or formatting - both of which are now general purpose facilities). It seems that the ‘re serve’ framework can move into this, and perhaps we can log to things other than the file system.
>
> The changes were quite extensive:
>
> Everything in the can component
> retools component
> mcp2515 component
> esp32can component
> dbc component
>
> I should be done with the changes in the next day or two, and then will do some testing in my car. Hopefully should be able to merge my commits later this week.
>
> Regards, Mark.
>
>> On 12 Jun 2019, at 9:00 PM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>
>> I also see the filter system in canlog could be very useful as a general filtering system. I suggest to move that into the central CAN class, and allow such software filters to be attached to listeners, callbacks, and loggers.
>>
>> By refactoring this, I think we can get some very useful functionality.
>>
>> @Michael, one question, for you: What is the purpose of RegisterCallback (and the callbacks vs listeners, in general). Listeners get their frames delivered via a queue (so must have a task running). Callbacks are simple function callbacks running synchronously with the CAN task. From what I can see, the only user of callbacks is vehicle_twizy where CanResponder is set as the callback on CAN1. Why is that necessary, given that the standard vehicle object has IncomingFrameCan1() to receive frames. Is this required for performance (running synchronously with the CAN receive task), or some other reason why it can’t run from the vehicle task on queue receive of the frame? In other words, why can’t we just call CanResponder() from within IncomingFrameCan1()?
>>
>> Regards, Mark.
>>
>>> On 11 Jun 2019, at 8:49 PM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>>
>>> It has become a little duplicated:
>>>
>>> struct CAN_frame_t
>>> {
>>> canbus* origin; // Origin of the frame
>>> CAN_FIR_t FIR; // Frame information record
>>> uint32_t MsgID; // Message ID
>>> union
>>> {
>>> uint8_t u8[8]; // Payload byte access
>>> uint32_t u32[2]; // Payload u32 access (Att: little endian!)
>>> uint64_t u64; // Payload u64 access (Att: little endian!)
>>> } data;
>>>
>>> esp_err_t Write(canbus* bus=NULL, TickType_t maxqueuewait=0); // bus: NULL=origin
>>> };
>>>
>>> typedef struct
>>> {
>>> uint32_t timestamp;
>>> canbus* bus;
>>> CAN_LogEntry_t type;
>>> union
>>> {
>>> CAN_frame_t frame;
>>> CAN_status_t status;
>>> char* text;
>>> };
>>> } CAN_LogMsg_t;
>>>
>>> typedef struct
>>> {
>>> CAN_frame_t last;
>>> uint32_t rxcount;
>>> struct __attribute__((__packed__))
>>> {
>>> struct {
>>> uint8_t Ignore:1; // 0x01
>>> uint8_t Changed:1; // 0x02
>>> uint8_t Discovered:1; // 0x04
>>> uint8_t :1; // 0x08
>>> uint8_t :1; // 0x10
>>> uint8_t :1; // 0x20
>>> uint8_t :1; // 0x40
>>> uint8_t :1; // 0x80
>>> } b;
>>> uint8_t dc; // Data bytes changed
>>> uint8_t dd; // Data bytes discovered
>>> uint8_t spare;
>>> } attr;
>>> } re_record_t;
>>>
>>> CAN_frame_t has origin for the can bus the message arrived on, while CAN_LogMsg_t has frame.origin and bus (presumably because bus is needed for status and text messages).
>>>
>>> Then we have candump and canlog virtual interfaces (canlog supports ’trace’ and ‘crtd’, while candump supports ‘crtd’ and ‘pap’).
>>>
>>> The users of these higher level structures are RETOOLS and CANLOG. I suggest the following changes:
>>>
>>> Have a single format conversion virtual class, and then initial implementations for pcap, crtd and trace. That can virtualise the conversion of a CAN message to be logged/transmitted/received to/from the external format. This can include a factory for registration of types (by textual name), as well as support functions for command registration (including whether they have support for reading, writing, or both). This just provides format conversion and does not provide any file handling or transmission functions. Canlog and Candump get merged to form this.
>>>
>>> Perhaps base this on CAN_LogMsg_t, but I suggest a more generic name (perhaps CAN_message_t, to differentiate that this is a can message rather than a simple frame). It is CAN_frame_t + additional support for stats and general text messages.
>>>
>>> Tidy up the CAN_LogMsg_t bus vs frame.origin issue.
>>>
>>> Simply put, separate the formatting into a standalone virtual implementations, then leave retools and canlog to handle the actual RE stuff or file logging, or whatever else is required with the data that gets converted.
>>>
>>> Does that make sense?
>>>
>>> Regards, Mark.
>>>
>>>> On 11 Jun 2019, at 3:01 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>
>>>> Mark,
>>>>
>>>> when implementing the can log framework I introduced the CAN_LogMsg_t specifically to capture the original timestamp of the frame, so it wouldn't get lost if
>>>> the logging task gets behind, and to transport other log data types than just frames to the loggers.
>>>>
>>>> I would also expect the RE tools stream to produce event info & statistics entries like can log does, as that is very helpful in analysing. Maybe rebasing
>>>> candump on can log would be better?
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>>
>>>> Am 11.06.19 um 08:37 schrieb Mark Webb-Johnson:
>>>>> This really should be unified, and probably not too hard to do. Crazy to have two so similar frameworks. I think we can just have one virtual class for converting CAN frames to/from other formats, and then implementations for CRTD, PCAP, etc.
>>>>>
>>>>> I do remember that last time I looked at this, canlog used CAN_LogMsg_t, and candump used CAN_frame_t (with CAN_LogMsg_t having some other fields for logging statistics, etc), and that made it non-trivial. Perhaps canlog should be built on top of candump?
>>>>>
>>>>> I will have a look at it.
>>>>>
>>>>> Regards, Mark.
>>>>>
>>>>>> On 6 Jun 2019, at 4:06 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>>>
>>>>>> Steve,
>>>>>>
>>>>>> Mark quoted from the RE tools candump_crtd class, which has its own crtd formatter.
>>>>>>
>>>>>> For the canlog framework I currently get the timestamp from esp_log_timestamp(). Should be easy to change that to gettimeofday().
>>>>>>
>>>>>> Regards,
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> Am 05.06.19 um 18:39 schrieb Stephen Casner:
>>>>>>> Mark,
>>>>>>>
>>>>>>> If by "unix julian time" you mean a timestamp like 1559751698 (the
>>>>>>> time as I write this), than that is not what Greg and I have observed
>>>>>>> in the CRTD logs. Here are the first lines in our files:
>>>>>>>
>>>>>>> 151766.574 CXX Info Type:crtd; Path:'/sd/gps.crtd'; Filter:1:100-100; Vehicle:TR2N;
>>>>>>>
>>>>>>> 383603.293 CXX Info Type:crtd; Path:'/sd/location.crtd'; Filter:1:100-100; Vehicle:TR1N;
>>>>>>>
>>>>>>> Yet OVMS does have the current time from NTP:
>>>>>>>
>>>>>>> OVMS# time
>>>>>>> Time Zone:
>>>>>>> UTC Time: 2019-06-05 16:35:45 UTC
>>>>>>> Local Time: 2019-06-05 16:35:45 GMT
>>>>>>> Provider: ntp
>>>>>>>
>>>>>>> PROVIDER STRATUM UPDATE TIME
>>>>>>> *ntp 1 57 Wed Jun 5 16:35:44 2019
>>>>>>>
>>>>>>> Who calls candump_crtd::get() with the timeval that goes into the
>>>>>>> sprintf() you mention?
>>>>>>>
>>>>>>> -- Steve
>>>>>>>
>>>>>>> On Wed, 5 Jun 2019, Mark Webb-Johnson wrote:
>>>>>>>
>>>>>>>> The CRTD timestamps should be unix julian time (not uptime). Produced by this:
>>>>>>>>
>>>>>>>> sprintf(m_buf,"%ld.%06ld %sR%s %0*X",
>>>>>>>> time->tv_sec, time->tv_usec,
>>>>>>>> busnumber,
>>>>>>>> (frame->FIR.B.FF == CAN_frame_std) ? "11":"29",
>>>>>>>> (frame->FIR.B.FF == CAN_frame_std) ? 3 : 8,
>>>>>>>> frame->MsgID);
>>>>>>>>
>>>>>>>> Once module gets time from the vehicle (or gps), it should show the correct timestamps.
>>>>>>>>
>>>>>>>> Regards, Mark.
>>>>>>> _______________________________________________
>>>>>>> OvmsDev mailing list
>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>
>>>> --
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20190619/60eaf8b7/attachment.htm>
More information about the OvmsDev
mailing list