<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Ludovic,<br>
<br>
this is far more than I ever needed for my vehicle CAN logging, but
I can see this making sense in vehicle or ECU development, and in
gathering initial data for reverse engineering from users in the
field. With this, users can easily collect data over a couple of
days taking photos & notes, and then send the collected data to
the developer. That's currently hard to do, as users would need to
monitor & manage the continuous logging.<br>
<br>
Your config scheme looks appropriate. I assume "vfs-auto" will set
another config instance to enable continuation after a crash/reboot?<br>
<br>
Please avoid "_" on event names. Maybe you should raise e.g.
"can.log.rotate.pre" synchronously before the actual rotation, so
handlers can add some last info to the file, and
"can.log.rotate.post" after.<br>
<br>
Adding a status dump before closing a file would be good so you can
see if there were drops. That currently needs a "can canX status" to
be triggered. Maybe clearing the counters on transition to the next
file then also makes sense.<br>
<br>
A command to log arbitrary comments would also make sense, that
could be used by users and scripts.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 26.11.22 um 23:31 schrieb Ludovic
LANGE:<br>
</div>
<blockquote type="cite"
cite="mid:a2e4a4b6-cf7a-bcc4-0c30-f8181050e357@lange.nom.fr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">Hi,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">To follow on this previous idea, I'd
like to share a Proof Of Concept of my proposal in the `<font
face="monospace">748-split-can-log-files</font>` branch for
comments - you can see the diff here
<a class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/compare/master...llange:Open-Vehicle-Monitoring-System-3:748-split-can-log-files"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/compare/master...llange:Open-Vehicle-Monitoring-System-3:748-split-can-log-files</a></div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">It does not (yet !) include all the
suggestions received - notably absent is the proposal of file
logging sync period from Michael below - but I'll try to take
them into account.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">There is an open issue
<a class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/748"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/748</a>
to gather public comments regarding the code if you prefer to do
it on GitHub.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">(When talking of automatic naming, I
mean in fact automatic file name rotation based on different
criterion)<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Here it is:</div>
<div class="moz-cite-prefix">
<ul>
<li>A new command is introduced `<font face="monospace">can
log start vfs-auto</font>` which is exactly like `<font
face="monospace">can log start vfs</font>` but, you
guessed it, with automatic file name rotation<br>
</li>
<ul>
<li>The parameter to this command is not a file name any
more, but a `<font face="monospace">prefix</font>` (see
below)<br>
<br>
</li>
</ul>
<li>New configuration items are introduced:</li>
<ul>
<li>`<font face="monospace">[can] log.file.maxsize_kb</font>`
: if the log file size exceeds this number, the file will
be rotated<br>
</li>
<li>`<font face="monospace">[can] log.file.maxduration_s</font>`
: if the log file has logged for more than (this number)
seconds, the file will be rotated</li>
<li>`<font face="monospace">[can] log.file.keep_empty</font>`
: in case a rotation would have created an empty file (no
messages), should we rotate nevertheless (<font
face="monospace">`true`</font>) or not (`<font
face="monospace">false`</font>) ?<br>
</li>
<li>`<font face="monospace">[can] log.file.pattern</font>` :
this describes how to name the file with some variables
(see below)<br>
<br>
</li>
</ul>
<li>A new event `<font face="monospace">can.log.rotate_files</font>`
is introduced to allow one to force a file name rotation<br>
</li>
</ul>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The file name (`<font
face="monospace">log.file.pattern</font>`) can be configured
with (combination of) the following elements:</div>
<div class="moz-cite-prefix">
<ul>
<li>`<font face="monospace">strftime</font>` format conversion
specification (like<font face="monospace"> `%Y` `%H`</font>
...)<br>
</li>
<li>`<font face="monospace">{vehicleid}</font>` will be
replaced by the configured vehicle id</li>
<li>`<font face="monospace">{session}</font>` will be replaced
by the counter of restarts of the module (always
incrementing) (See
<a class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/779"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/779</a>
for a discussion, will certainly evolve)<br>
</li>
<li>`<font face="monospace">{prefix}</font>` is the argument
to the log command `<font face="monospace">can log start
vfs-auto <format> <prefix></font>`, so that
you can configure multiple logs in parallel and have them
log to different files (by varying `<font face="monospace">prefix</font>`)<br>
</li>
<li>`<font face="monospace">{splits}</font>` is a counter of
the number of cycles that occurred to this very log file</li>
<li>`<font face="monospace">{extension}</font>` is the
preferred extension for the chosen log format</li>
</ul>
<p>(Any non-existing directory will be created on the go during
file rotation)<br>
</p>
<p><br>
</p>
</div>
<div class="moz-cite-prefix">The default for these variables are:</div>
<div class="moz-cite-prefix">```</div>
<div class="moz-cite-prefix"><font face="monospace">OVMS# con lis
can<br>
can (readable writeable)<br>
log.file.keep_empty: false<br>
log.file.maxduration_s: 1800<br>
log.file.maxsize_kb: 1024<br>
log.file.pattern:
/sd/log/{vehicleid}/{session}/{prefix}/{splits}-%Y%m%d-%H%M%S{extension}</font><br>
</div>
<div class="moz-cite-prefix">```<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">To illustrate a sequence of file name
rotation, here is an excerpt with smaller values (maxduration:
120s / maxsize : 2kb), started by the command `<font
face="monospace">can log start vfs-auto crtd essai</font>`.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The first two renames occur because
of maxduration (>= 120s) while the third rename occur because
the log file exceeded 2048 bytes.</div>
<div class="moz-cite-prefix">(In addition, SNTP synchro occurred
just before the first rename, so you see that the first file
segment is dated 1970... while the second one has proper date
and time)<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">```</div>
<div class="moz-cite-prefix"><font face="monospace">...<br>
</font></div>
<div class="moz-cite-prefix"><font face="monospace">I (119937)
canlog-vfs: canlog_vfs_autonaming::OutputMsg() size:172, log
duration: 120s<br>
I (119957) canlog-vfs: Closed vfs log
'/sd/log/ovms-vehicle/00000390/essai/00000001-19700101-010008.crtd':
Size:172 <a class="moz-txt-link-freetext" href="Messages:2"
moz-do-not-send="true">Messages:2</a> Dropped:0 Filtered:0
Rate:0.0%<br>
I (119967) canlog-vfs: Changing file name from
'/sd/log/ovms-vehicle/00000390/essai/00000001-19700101-010008.crtd'
to
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd'<br>
I (119977) canlog-vfs: Now logging CAN messages to
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd'<br>
<br>
I (239967) canlog-vfs: canlog_vfs_autonaming::OutputMsg()
size:1943, log duration: 120s<br>
I (239987) canlog-vfs: Closed vfs log
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd':
Size:1.9k <a class="moz-txt-link-freetext" href="Messages:22"
moz-do-not-send="true">Messages:22</a> Dropped:0 Filtered:0
Rate:0.0%<br>
I (239997) canlog-vfs: Changing file name from
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd'
to
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd'<br>
I (240007) canlog-vfs: Now logging CAN messages to
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd'<br>
<br>
I (334967) canlog-vfs: canlog_vfs_autonaming::OutputMsg()
size:2111, log duration: 95s<br>
I (334987) canlog-vfs: Closed vfs log
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd':
Size:2.1k <a class="moz-txt-link-freetext" href="Messages:24"
moz-do-not-send="true">Messages:24</a> Dropped:0 Filtered:0
Rate:0.0%<br>
I (334997) canlog-vfs: Changing file name from
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd'
to
'/sd/log/ovms-vehicle/00000390/essai/00000004-20221126-160327.crtd'<br>
I (335007) canlog-vfs: Now logging CAN messages to
'/sd/log/ovms-vehicle/00000390/essai/00000004-20221126-160327.crtd'<br>
...</font></div>
<div class="moz-cite-prefix"><font face="monospace"><br>
</font></div>
<div class="moz-cite-prefix"><font face="monospace">OVMS# vfs rls
/sd/log/ovms-vehicle/00000390/essai/<br>
172 26-Nov-2022 15:59
/sd/log/ovms-vehicle/00000390/essai/00000001-19700101-010008.crtd<br>
1.9k 26-Nov-2022 16:01
/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd<br>
2.1k 26-Nov-2022 16:03
/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd<br>
0 30-Nov-1979 00:00
/sd/log/ovms-vehicle/00000390/essai/00000004-20221126-160327.crtd</font><br>
<br>
</div>
<div class="moz-cite-prefix">```<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Notes:</div>
<div class="moz-cite-prefix">
<ul>
<li>maxsize comparisons are done after writing, which means
that you will have files slightly bigger than maxsize - not
under maxsize. (for maxduration however it should be OK as
the comparison is done before writing)<br>
<br>
</li>
<li>While the comparison of maxsize is done on the real number
of bytes written, the concept of empty file (for `<font
face="monospace">[can] log.file.keep_empty</font>`) is
based on the number of messages logged to the file.<br>
This, because most of the files will have a header written
even if no messages are written ; which I considered to be
nevertheless an empty file.<br>
Those "empty" files with no messages will not be rotated in
that case (and kept open for logging).</li>
</ul>
<p>Let me know what you think of this, if you have some remarks,
and even if you feel it has no interest whatsoever.</p>
<p>Regards,</p>
<p>Ludovic<br>
</p>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Le 12/11/2022 à 09:02, Michael Balzer
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:e6969d56-6429-a38e-5ffb-4932e9fee9d5@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Ludovic,<br>
<br>
most of your thoughts were also mine on designing the file
logging scheme, I suggest taking that as a reference &
inspiration.<br>
<br>
I suggest controlling this through a similar config scheme,
introducing more can config param instances as needed. There's
currently only the queue size:<br>
<br>
<font face="monospace">OVMS# con lis can<br>
can (readable writeable)<br>
log.queuesize: 300</font><br>
<br>
The file logging sync period feature is what you need to not
lose a complete file on crash, it also enables following a file
by "vfs tail" and transferring the file content written so far
without stopping the logger.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 09.11.22 um 22:52 schrieb
Ludovic LANGE:<br>
</div>
<blockquote type="cite"
cite="mid:3a3a4714-e439-56a9-885f-f8098b21f603@lange.nom.fr">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<p>Hi,<br>
</p>
<p>I'm planning to implement an "automatic naming" of the CAN
log files for the VFS target. It would also take care of the
file name rotation according to some criterion (file size,
and/or log duration).</p>
<p>I've created an issue to follow it here <a
class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/748"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/748</a>
(and some others also) but let me first share what I have in
mind:<br>
</p>
<p><br>
</p>
<p>The use case is the following: I want to setup OVMS as a
CAN bus datalogger. It should be always-on, always and
continuously logging, event after a restart.<br>
</p>
<p>(For that, my current solution is to use event scripts, cf
<a class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/766"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/766</a>
)<br>
</p>
<p>Logging to the same file, in that case, present some
challenges:</p>
<ul>
<li>file size would grow indefinitely, which could be then
make it difficult to copy / network send somewhere<br>
</li>
<li>file would always be opened, which could prevent any
action on it (if we want to send it somewhere else, or
remove it)</li>
<li>in case of crash / corruption the whole file could be
lost</li>
<li>in case of reboot you'd end overwriting the file.<br>
</li>
</ul>
<p>Of course we should change the file name, but it's not easy
to do it in a script.</p>
<p>It may be possible to do this with Duktape and `<code
class="docutils literal notranslate"><span class="pre">OvmsCommand.Exec</span></code>`
but I fear that we may miss events if we stop / start
logging on a highly loaded bus.<br>
</p>
<p><br>
</p>
<p>A proposal to solve this would be to add a new mode in
`canlog_vfs.cpp` that would cater for this automatic naming
(inspiration for this feature coming from another product).</p>
<p><br>
In this mode you don't pass the path as a parameter, but
instead a `<font face="monospace">log prefix</font>` that
will be used to construct the file path. <br>
<br>
Syntax could be something like : `<font face="monospace">can
log start vfs-auto crtd <log prefix> [filter1] ...
[filterN]</font>` (no filename).<br>
<br>
It would automatically generate a log file name, taking into
account the splits / rotations that can be configured:<br>
</p>
<ul>
<li>if asked for by configuration - rotate file name (=
increment file name) after a file size is reached</li>
<li>if asked for by configuration - rotate file name (=
increment file name) after a certain logging duration has
been reached</li>
</ul>
<p><br>
Naming would be 'automatic' with something like : <br>
`<font face="monospace">/sd/log/<vehicleid>/<#power
on>/<log prefix>/<#rotations>.<log
extension></font>`<br>
<br>
where:<br>
* `<font face="monospace">/sd/log/</font>` is arbitrary.
Could be coming from configuration if wanted (esp. for
module without sd card)<br>
* `<font face="monospace"><vehicleid></font>` is there
to help to declutter your sdcard (if you share it with
multiple modules), or to declutter your storage server (when
<a class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/749"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/749</a>
has been completed). We may have to make it a valid
filename.<br>
* `<font face="monospace"><#power on></font>` is the
number of time the module has powered on ; it is an always
incrementing number expressed as an integer on 8 digits.
Depending on the configuration, some modules will be
always-on ; others will stop more frequently - the number
will overflow after 99.999.999 power on, which will
certainly see a change of sd card, so no fear of unwanted
overwriting of files<br>
* `<font face="monospace"><log prefix></font>` is a
way to distinguish between multiple concurrent can logs
active.<br>
* `<font face="monospace"><#rotations></font>` is a
counter of rotations for this file and this power on ; it is
an integer on 8 digits ; reset to `<font face="monospace">00000001</font>`
at each poweroff and that will increment each time there is
a file name rotation.<br>
<br>
some configuration items cater for the file name rotation:<br>
* `<font face="monospace">split_size_mb</font>` : when the
file has reached this size, the file is closed, the element
`<font face="monospace"><#rotations></font>` is
incremented, and a new file with the 'automatic' name is
opened.<br>
* `<font face="monospace">split_duration_s</font>` : when
the file has logged for this duration, the file is closed,
the element `<font face="monospace"><#rotations></font>`
is incremented, and a new file with the 'automatic' name is
opened.</p>
<p><br>
<br>
example of file name: `<font face="monospace">/sd/log/mycar/00000024/can1-25k/00000005.crtd</font>`</p>
<p><br>
</p>
<p>(Of course we need a way to either stop logging once the
VFS is full, or to auto-remove some log files under this
path)<br>
</p>
<p><br>
</p>
<p>Let me know if you have any thoughts / concerns about this.</p>
<p><br>
</p>
<p>Regards,</p>
<p>Ludovic</p>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<p><br>
</p>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</body>
</html>