[Ovmsdev] Automatic naming of CAN log files

[Michael Balzer]

Ludovic,

most of your thoughts were also mine on designing the file logging 
scheme, I suggest taking that as a reference & inspiration.

I suggest controlling this through a similar config scheme, introducing 
more can config param instances as needed. There's currently only the 
queue size:

OVMS# con lis can
can (readable writeable)
   log.queuesize: 300

The file logging sync period feature is what you need to not lose a 
complete file on crash, it also enables following a file by "vfs tail" 
and transferring the file content written so far without stopping the 
logger.

Regards,
Michael


Am 09.11.22 um 22:52 schrieb Ludovic LANGE:
>
> Hi,
>
> I'm planning to implement an "automatic naming" of the CAN log files 
> for the VFS target. It would also take care of the file name rotation 
> according to some criterion (file size, and/or log duration).
>
> I've created an issue to follow it here 
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/748 
> (and some others also) but let me first share what I have in mind:
>
>
> The use case is the following: I want to setup OVMS as a CAN bus 
> datalogger. It should be always-on, always and continuously logging, 
> event after a restart.
>
> (For that, my current solution is to use event scripts, cf 
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/766 
> )
>
> Logging to the same file, in that case, present some challenges:
>
>   * file size would grow indefinitely, which could be then make it
>     difficult to copy / network send somewhere
>   * file would always be opened, which could prevent any action on it
>     (if we want to send it somewhere else, or remove it)
>   * in case of crash / corruption the whole file could be lost
>   * in case of reboot you'd end overwriting the file.
>
> Of course we should change the file name, but it's not easy to do it 
> in a script.
>
> It may be possible to do this with Duktape and `|OvmsCommand.Exec|` 
> but I fear that we may miss events if we stop / start logging on a 
> highly loaded bus.
>
>
> A proposal to solve this would be to add a new mode in 
> `canlog_vfs.cpp` that would cater for this automatic naming 
> (inspiration for this feature coming from another product).
>
>
> In this mode you don't pass the path as a parameter, but instead a 
> `log prefix` that will be used to construct the file path.
>
> Syntax could be something like : `can log start vfs-auto crtd <log 
> prefix> [filter1] ... [filterN]` (no filename).
>
> It would automatically generate a log file name, taking into account 
> the splits / rotations that can be configured:
>
>   * if asked for by configuration - rotate file name (= increment file
>     name) after a file size is reached
>   * if asked for by configuration - rotate file name (= increment file
>     name) after a certain logging duration has been reached
>
>
> Naming would be 'automatic' with something like :
> `/sd/log/<vehicleid>/<#power on>/<log prefix>/<#rotations>.<log 
> extension>`
>
> where:
> * `/sd/log/` is arbitrary. Could be coming from configuration if 
> wanted (esp. for module without sd card)
> * `<vehicleid>` is there to help to declutter your sdcard (if you 
> share it with multiple modules), or to declutter your storage server 
> (when 
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/749 
> has been completed). We may have to make it a valid filename.
> * `<#power on>` is the number of time the module has powered on ; it 
> is an always incrementing number expressed as an integer on 8 digits. 
> Depending on the configuration, some modules will be always-on ; 
> others will stop more frequently - the number will overflow after 
> 99.999.999 power on, which will certainly see a change of sd card, so 
> no fear of unwanted overwriting of files
> * `<log prefix>` is a way to distinguish between multiple concurrent 
> can logs active.
> * `<#rotations>` is a counter of rotations for this file and this 
> power on ; it is an integer on 8 digits ; reset to `00000001` at each 
> poweroff and that will increment each time there is a file name rotation.
>
> some configuration items cater for the file name rotation:
> * `split_size_mb` : when the file has reached this size, the file is 
> closed, the element `<#rotations>` is incremented, and a new file with 
> the 'automatic' name is opened.
> * `split_duration_s` : when the file has logged for this duration, the 
> file is closed, the element `<#rotations>` is incremented, and a new 
> file with the 'automatic' name is opened.
>
>
>
> example of file name:  `/sd/log/mycar/00000024/can1-25k/00000005.crtd`
>
>
> (Of course we need a way to either stop logging once the VFS is full, 
> or to auto-remove some log files under this path)
>
>
> Let me know if you have any thoughts / concerns about this.
>
>
> Regards,
>
> Ludovic
>

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20221112/b5d365f2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20221112/b5d365f2/attachment.sig>