[Ovmsdev] CAN BUS logging to flash storage
Michael Balzer
dexter at expeedo.de
Fri Jan 19 07:17:18 HKT 2018
OK, first implementation is done & pushed. Documentation is missing, I'll add that ASAP.
Basic usage:
OVMS > can ?
can1 CANx framework
can2 CANx framework
can3 CANx framework
log CAN logging framework
OVMS > can log ?
crtd ...format logging
off Stop logging
status Logging status
trace Logging to syslog
OVMS > can log status
CAN logging inactive.
OVMS > can log trace ?
Usage: can log trace [filter1] [filter2] [filter3]
Filter: <bus> / <id>[-<id>] / <bus>:<id>[-<id>]
Example: 2:2a0-37f
OVMS > can log crtd ?
Usage: can log crtd <path> [filter1] [filter2] [filter3]
Filter: <bus> / <id>[-<id>] / <bus>:<id>[-<id>]
Example: 2:2a0-37f
OVMS > can log trace
CAN logging active: Type:trace; Path:''; Filter:off; Vehicle:;
Note: info logging is done at log level debug, frame logging at verbose
OVMS > can can2 tx standard 201 12 34 56 11 22 33 44 55
E (358776) canlog: Error can2 rxpkt=0 txpkt=1 errflags=0x8000 rxerr=63 txerr=0 rxovr=0 txovr=0 txdelay=0
OVMS > can log off
Closing log: Type:trace; Path:''; Filter:off; Vehicle:;
CAN logging stopped.
OVMS > can log crtd /sd/test.crtd 1 2:780-7ff
CAN logging active: Type:crtd; Path:'/sd/test.crtd'; Filter:1:0-ffffffff,2:780-7ff; Vehicle:;
OVMS > can can1 rx standard 100 12 34 56 11 22 33 44 55
OVMS > can can1 tx standard 101 12 34 56 11 22 33 44 55
OVMS > can can2 rx standard 200 12 34 56 11 22 33 44 55
OVMS > can can2 tx standard 201 12 34 56 11 22 33 44 55
OVMS > can can2 rx standard 7d2 12 34 56 11 22 33 44 55
OVMS > can can2 tx standard 7d3 12 34 56 11 22 33 44 55
OVMS > can log off
Closing log: Type:crtd; Path:'/sd/test.crtd'; Filter:1:0-ffffffff,2:780-7ff; Vehicle:;
CAN logging stopped.
OVMS > vfs cat /sd/test.crtd
459.576 CXX Info Type:crtd; Path:'/sd/test.crtd'; Filter:1:0-ffffffff,2:780-7ff; Vehicle:;
476.706 1R11 100 12 34 56 11 22 33 44 55
480.366 1T11 101 12 34 56 11 22 33 44 55
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=28 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=28 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=37 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=46 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=55 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=64 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=73 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8000a6 rxerr=82 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x44000 rxerr=99 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x44000 rxerr=99 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8040a6 rxerr=100 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8040a6 rxerr=109 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8040a6 rxerr=118 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8040a6 rxerr=127 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x204000 rxerr=135 txerr=0 rxovr=0 txovr=0 txdelay=0
480.366 1CEV Error rxpkt=1 txpkt=1 errflags=0x8040a6 rxerr=135 txerr=0 rxovr=0 txovr=0 txdelay=0
522.616 2R11 7D2 12 34 56 11 22 33 44 55
522.616 2CEV TX_Queue T11 7D3 12 34 56 11 22 33 44 55
Vehicle events are also logged, and if you issue "can <bus> status", the packet statistics will be logged.
The crtd logger is fast enough for the (single bus) Twizy at 500 kbit without filters. I still have no means to test multiple buses.
The current implementation limits the user to one active can logger, but that's just for ease of use and not fixed. We can add support for multiple parallel
loggers, but I doubt there's an actual use case for this.
Btw, as a next logger, we could implement the pcapng format: https://wiki.wireshark.org/Development/PcapNg
Regards,
Michael
Am 14.01.2018 um 18:13 schrieb Michael Balzer:
> Mark,
>
> thanks for the info, I'll add a documentation file on this.
>
> It just turned out writing into a file is far too slow to be done inline with the frame processing… and yes, I _could_ have anticipated that ;)
>
> Maybe idf 3.x will perform better on file I/O, but a separate logging task is better anyway.
>
> Regards,
> Michael
>
>
> Am 12.01.2018 um 01:44 schrieb Mark Webb-Johnson:
>> Thanks.
>>
>> I really should formally document the CRTD format properly somewhere. Anyway, here are my old notes on it, and an example:
>>
>> +++++
>>
>> CRTD Example:
>>
>> 1320724293.000 CXX OVMS Tesla Roadster cando2crtd converted log
>> 1320724293.000 CXX OVMS Tesla roadster log: trip.20111108.csv
>> 1320724294.072 R11 420 00 96 96
>> 1320724294.073 R11 588 00 00 86
>> 1320724294.073 R11 280 00 00 03 00 00 00 00 00
>> 1320724294.073 R11 001 00 00 00 00
>> 1320724294.073 R11 590 01 00 00 00 00 00 00 00
>> 1320724294.113 R11 400 01 01 00 00 00 00 4C 1D
>>
>>
>> Generators should use whatever line termination the host system uses. For Unix and embedded systems that is LF. Parsers should support all CR, LF, and CRLF
>> variants.
>>
>> Generators should output hex fields in upper-case and front zero padded (for clarity). Parsers should support both upper and lower case, as well as variable
>> length.
>>
>> By convention, the .crtd extension is used for these files. It is suggested that generators prefix all crtd files with CXX lines documenting the generator,
>> comment and source of log file.
>>
>> Space-separated fields are:
>>
>> * Timestamp (can be seconds, milliseconds, or microseconds - zero prefixed and full length padded for ms and us).
>> * Type:
>> o C*: comment
>> + CXX: general comment (rest of line is a comment)
>> + CEV: user-generated event (rest of line is event description)
>> o R11: rx standard frame
>> o R29: rx extended frame
>> o T11: tx standard frame
>> o T29: tx extended frame
>>
>>
>> Other field notes:
>>
>> * For the R?? and T?? types, the rest of the line is hex ID followed by hex data bytes (up to 8).
>> * In the case of systems supporting multiple CAN buses, the type can be prefixed with the bus number (eg; 1R11, 2T29, etc). Messages from different CAN bus
>> captures can be written to one shared crtd file, or individual files for each bus - entirely up to the generator.
>>
>>
>> +++++
>>
>> Regards, Mark.
>>
>>> On 11 Jan 2018, at 8:59 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>
>>> Mark,
>>>
>>> I'll add the logging support along with the TX queue.
>>>
>>> Regards,
>>> Michael
>>>
>>>
>>> Am 10.01.2018 um 01:27 schrieb Mark Webb-Johnson:
>>>> Probably best to split this off as a separate eMail thread.
>>>>
>>>> I think the implementation would be relatively simple.
>>>>
>>>> * Create a canlog class with a FILE* m_log member variable, and a virtual function Log() that is called with a direction (tx/rx) and frame to log.
>>>> o The constructor should be passed the vfs file path to log to. It should fopen the m_log file to that.
>>>> o The destructor should fclose the m_log file.
>>>> * Create a canlog_crtd class, derived from canlog, that supports Log() to log to m_log in CRTD format.
>>>> * Optionally create other canlog_* classes, for other logging formats that we want to support.
>>>> * Add a canlog* m_log member variable in canbus.
>>>> o Initialise m_log to NULL in the constructor.
>>>> o In the destructor, if m_log is not null, delete it’s object and set it to NULL.
>>>> o Have a command ‘can log’ like ‘can trace’ that specifies a vfs file path (remember to check it with OvmsConfig ProtectedPath) and logging format.
>>>> It will then ’new’ a canlog_* style object (depending on logging format).
>>>> * In can::IncomingFrame, log incoming CAN messages with m_log->Log(), in the same way as m_trace is handled.
>>>> * In canbus::Write, log outgoing CAN messages with m_log->Log(), in the same way as m_trace is handled.
>>>>
>>>>
>>>> Volunteer?
>>>>
>>>> Regards, Mark.
>>>>
>>>>> On 10 Jan 2018, at 8:07 AM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>>>>
>>>>> Now SD CARD support is coming, anybody want to step forward and extend components/can to support CAN bus logging to a file (in crtd format, or perhaps
>>>>> support one or two popular formats)? Implementation would be in class canbus, very similar to m_trace; just need to have a FILE* m_log and appropriate
>>>>> commands / logging of packet write/read in the same place that trace does it.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.teslaclub.hk
>>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>>>
>>>
>>> --
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.teslaclub.hk <mailto:OvmsDev at lists.teslaclub.hk>
>>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>>
>>
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.teslaclub.hk
>> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.teslaclub.hk
> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20180119/9c86c793/attachment.htm>
More information about the OvmsDev
mailing list