<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Ludovic,<br>
<br>
<div class="moz-cite-prefix">Am 02.12.22 um 23:31 schrieb Ludovic
LANGE:<br>
</div>
<blockquote type="cite"
cite="mid:aa918479-4abf-a75b-8a33-cfbe27332e21@lange.nom.fr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">Hi Michael,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Thanks for your encouraging feedback
!</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">A few points:</div>
<div class="moz-cite-prefix">
<blockquote type="cite"> I assume "vfs-auto" will set another
config instance to enable continuation after a crash/reboot?</blockquote>
In fact not at the moment :-) Based on the discussion we had in
<a class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/766"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/766</a>
, I settled on the attachment to `<font face="monospace">sd.*</font>`
events to start/stop the log. It's working OK for the moment and
starts as soon as the system is up and SC card mounted.</div>
</blockquote>
<br>
Of course you need to listen to the SD events, but you wouldn't want
the user having to take care of adding & removing scripts to
events to manage the CAN logger, a simple command should be
sufficient. You could provide a scheme to let "vfs-auto" create
& delete event scripts, but that would be risky, as scripts may
be altered by the user as well.<br>
<br>
The system file logger works similary, it starts automatically when
a path is configured. It also listens for config changes, so you can
start it simply by setting the path, which enables very simple web
control. The system file logger can be started without persistence
from the "log file" command, or with persistence by setting the
config. The "log file" command could be extended by e.g. an option
"-p" to make it set the config as well. For consistency, this usage
pattern should apply to the CAN file logger as well.<br>
<br>
To provide this scheme for the CAN logger you'll need to capture the
actual vfs-auto logging configuration in config instances (might
simply be the argument vector or shell full command).<br>
<br>
<blockquote type="cite"
cite="mid:aa918479-4abf-a75b-8a33-cfbe27332e21@lange.nom.fr">
<div class="moz-cite-prefix">
<blockquote type="cite">Please avoid "_" on event names.</blockquote>
Any technical reason for that ? Or is it for consistency ?</div>
<div class="moz-cite-prefix">Should I replace with '<font
face="monospace">-</font>' or just have `<font
face="monospace">can.log.rotatefiles</font>` ?<br>
</div>
</blockquote>
<br>
It's for consistency, we generally only use group/level separators
(dots). See the list of system events for reference, it's e.g.
"system.shuttingdown", not "system.shutting_down".<br>
<br>
<blockquote type="cite"
cite="mid:aa918479-4abf-a75b-8a33-cfbe27332e21@lange.nom.fr">
<div class="moz-cite-prefix"> </div>
<div class="moz-cite-prefix">
<blockquote type="cite">Maybe you should raise e.g.
"can.log.rotate.pre" synchronously before the actual rotation,
so handlers can add some last info to the file, and
"can.log.rotate.post" after.</blockquote>
Interesting. How could we ensure that all handlers have finished
handling the event ? Waiting for some time ? I'm a little bit
concerned of having to introduce delay in the rotation itself,
not wanting to drop events.</div>
</blockquote>
<br>
Synchronous events are implemented using the `event_signal_done_fn
callback` on `SignalEvent()`. The "done" callback executes after all
event handlers have been called.<br>
<br>
A more recent example of this is in my fix for issue #702, see
`OvmsConfig::Restore()`:<br>
<br>
<font face="monospace"> // Signal & wait for components to
shutdown:<br>
{<br>
OvmsSemaphore eventdone;<br>
auto callback = [](const char* event, void* data) {
((OvmsSemaphore*)data)->Give(); };<br>
MyEvents.SignalEvent("config.restore", &eventdone,
callback);<br>
eventdone.Take();<br>
}<br>
</font><br>
This code pattern can be applied if the current task simply needs to
(and can) pause execution until all event handlers have finished
execution. If you need to continue processing in the task emitting
the event, you'll implement a separate callback. See the system
shutdown procedure in `ovms_boot` for this pattern:<br>
<br>
<font face="monospace"> m_shutdown_timer = 60; // Give them 60
seconds to shutdown<br>
MyEvents.SignalEvent("system.shuttingdown", NULL,
boot_shuttingdown_done);<br>
</font><br>
Here, the separate `boot_shuttingdown_done()` function will be
called when all "system.shuttingdown" handlers have been processed.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:aa918479-4abf-a75b-8a33-cfbe27332e21@lange.nom.fr">
<div class="moz-cite-prefix">
<blockquote type="cite">Adding a status dump before closing a
file would be good so you can see if there were drops</blockquote>
Yes, that's useful to close the file with a dump of stats and
counters. (And you're right about clearing them)</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">
<blockquote type="cite">A command to log arbitrary comments
would also make sense, that could be used by users and
scripts.</blockquote>
Indeed. Maybe as a broadcast to all open logs, and/or having the
option to send the comments to a specific log.<br>
</div>
</blockquote>
<br>
As a broadcast, same pattern as for the "can status" log entry, but
possibly generally sending to all buses (i.e. command "can log
comment").<br>
<br>
<br>
<blockquote type="cite"
cite="mid:aa918479-4abf-a75b-8a33-cfbe27332e21@lange.nom.fr">
<div class="moz-cite-prefix"> </div>
<div class="moz-cite-prefix">Do you feel we should have all those
features (pre/post events, status dump, log comment, your
previous suggestion of file logging sync period) ready for a
first PR candidate ? Or would you prefer that we try to polish
the first functionality (some docs are needed, event naming),
create a first PR, and then add incrementally those features as
PRs ?<br>
</div>
</blockquote>
<br>
These are improvement options, the initial PR only needs to be
consistent and functional.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<blockquote type="cite"
cite="mid:aa918479-4abf-a75b-8a33-cfbe27332e21@lange.nom.fr">
<div class="moz-cite-prefix"> </div>
<div class="moz-cite-prefix">Regards,</div>
<div class="moz-cite-prefix">Ludovic<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Le 02/12/2022 à 20:15, Michael Balzer
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:4c1b7e66-5c8f-8946-f02f-b1f4cca30c33@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Ludovic,<br>
<br>
this is far more than I ever needed for my vehicle CAN logging,
but I can see this making sense in vehicle or ECU development,
and in gathering initial data for reverse engineering from users
in the field. With this, users can easily collect data over a
couple of days taking photos & notes, and then send the
collected data to the developer. That's currently hard to do, as
users would need to monitor & manage the continuous logging.<br>
<br>
Your config scheme looks appropriate. I assume "vfs-auto" will
set another config instance to enable continuation after a
crash/reboot?<br>
<br>
Please avoid "_" on event names. Maybe you should raise e.g.
"can.log.rotate.pre" synchronously before the actual rotation,
so handlers can add some last info to the file, and
"can.log.rotate.post" after.<br>
<br>
Adding a status dump before closing a file would be good so you
can see if there were drops. That currently needs a "can canX
status" to be triggered. Maybe clearing the counters on
transition to the next file then also makes sense.<br>
<br>
A command to log arbitrary comments would also make sense, that
could be used by users and scripts.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 26.11.22 um 23:31 schrieb
Ludovic LANGE:<br>
</div>
<blockquote type="cite"
cite="mid:a2e4a4b6-cf7a-bcc4-0c30-f8181050e357@lange.nom.fr">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<div class="moz-cite-prefix">Hi,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">To follow on this previous idea,
I'd like to share a Proof Of Concept of my proposal in the `<font
face="monospace">748-split-can-log-files</font>` branch
for comments - you can see the diff here <a
class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/compare/master...llange:Open-Vehicle-Monitoring-System-3:748-split-can-log-files"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/compare/master...llange:Open-Vehicle-Monitoring-System-3:748-split-can-log-files</a></div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">It does not (yet !) include all
the suggestions received - notably absent is the proposal of
file logging sync period from Michael below - but I'll try
to take them into account.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">There is an open issue <a
class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/748"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/748</a>
to gather public comments regarding the code if you prefer
to do it on GitHub.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">(When talking of automatic
naming, I mean in fact automatic file name rotation based on
different criterion)<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Here it is:</div>
<div class="moz-cite-prefix">
<ul>
<li>A new command is introduced `<font face="monospace">can
log start vfs-auto</font>` which is exactly like `<font
face="monospace">can log start vfs</font>` but, you
guessed it, with automatic file name rotation<br>
</li>
<ul>
<li>The parameter to this command is not a file name any
more, but a `<font face="monospace">prefix</font>`
(see below)<br>
<br>
</li>
</ul>
<li>New configuration items are introduced:</li>
<ul>
<li>`<font face="monospace">[can] log.file.maxsize_kb</font>`
: if the log file size exceeds this number, the file
will be rotated<br>
</li>
<li>`<font face="monospace">[can] log.file.maxduration_s</font>`
: if the log file has logged for more than (this
number) seconds, the file will be rotated</li>
<li>`<font face="monospace">[can] log.file.keep_empty</font>`
: in case a rotation would have created an empty file
(no messages), should we rotate nevertheless (<font
face="monospace">`true`</font>) or not (`<font
face="monospace">false`</font>) ?<br>
</li>
<li>`<font face="monospace">[can] log.file.pattern</font>`
: this describes how to name the file with some
variables (see below)<br>
<br>
</li>
</ul>
<li>A new event `<font face="monospace">can.log.rotate_files</font>`
is introduced to allow one to force a file name rotation<br>
</li>
</ul>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The file name (`<font
face="monospace">log.file.pattern</font>`) can be
configured with (combination of) the following elements:</div>
<div class="moz-cite-prefix">
<ul>
<li>`<font face="monospace">strftime</font>` format
conversion specification (like<font face="monospace">
`%Y` `%H`</font> ...)<br>
</li>
<li>`<font face="monospace">{vehicleid}</font>` will be
replaced by the configured vehicle id</li>
<li>`<font face="monospace">{session}</font>` will be
replaced by the counter of restarts of the module
(always incrementing) (See <a
class="moz-txt-link-freetext"
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/779"
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/779</a>
for a discussion, will certainly evolve)<br>
</li>
<li>`<font face="monospace">{prefix}</font>` is the
argument to the log command `<font face="monospace">can
log start vfs-auto <format> <prefix></font>`,
so that you can configure multiple logs in parallel and
have them log to different files (by varying `<font
face="monospace">prefix</font>`)<br>
</li>
<li>`<font face="monospace">{splits}</font>` is a counter
of the number of cycles that occurred to this very log
file</li>
<li>`<font face="monospace">{extension}</font>` is the
preferred extension for the chosen log format</li>
</ul>
<p>(Any non-existing directory will be created on the go
during file rotation)<br>
</p>
<p><br>
</p>
</div>
<div class="moz-cite-prefix">The default for these variables
are:</div>
<div class="moz-cite-prefix">```</div>
<div class="moz-cite-prefix"><font face="monospace">OVMS# con
lis can<br>
can (readable writeable)<br>
log.file.keep_empty: false<br>
log.file.maxduration_s: 1800<br>
log.file.maxsize_kb: 1024<br>
log.file.pattern:
/sd/log/{vehicleid}/{session}/{prefix}/{splits}-%Y%m%d-%H%M%S{extension}</font><br>
</div>
<div class="moz-cite-prefix">```<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">To illustrate a sequence of file
name rotation, here is an excerpt with smaller values
(maxduration: 120s / maxsize : 2kb), started by the command
`<font face="monospace">can log start vfs-auto crtd essai</font>`.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The first two renames occur
because of maxduration (>= 120s) while the third rename
occur because the log file exceeded 2048 bytes.</div>
<div class="moz-cite-prefix">(In addition, SNTP synchro
occurred just before the first rename, so you see that the
first file segment is dated 1970... while the second one has
proper date and time)<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">```</div>
<div class="moz-cite-prefix"><font face="monospace">...<br>
</font></div>
<div class="moz-cite-prefix"><font face="monospace">I (119937)
canlog-vfs: canlog_vfs_autonaming::OutputMsg() size:172,
log duration: 120s<br>
I (119957) canlog-vfs: Closed vfs log
'/sd/log/ovms-vehicle/00000390/essai/00000001-19700101-010008.crtd':
Size:172 <a class="moz-txt-link-freetext"
href="Messages:2" moz-do-not-send="true">Messages:2</a>
Dropped:0 Filtered:0 Rate:0.0%<br>
I (119967) canlog-vfs: Changing file name from
'/sd/log/ovms-vehicle/00000390/essai/00000001-19700101-010008.crtd'
to
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd'<br>
I (119977) canlog-vfs: Now logging CAN messages to
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd'<br>
<br>
I (239967) canlog-vfs: canlog_vfs_autonaming::OutputMsg()
size:1943, log duration: 120s<br>
I (239987) canlog-vfs: Closed vfs log
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd':
Size:1.9k <a class="moz-txt-link-freetext"
href="Messages:22" moz-do-not-send="true">Messages:22</a>
Dropped:0 Filtered:0 Rate:0.0%<br>
I (239997) canlog-vfs: Changing file name from
'/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd'
to
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd'<br>
I (240007) canlog-vfs: Now logging CAN messages to
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd'<br>
<br>
I (334967) canlog-vfs: canlog_vfs_autonaming::OutputMsg()
size:2111, log duration: 95s<br>
I (334987) canlog-vfs: Closed vfs log
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd':
Size:2.1k <a class="moz-txt-link-freetext"
href="Messages:24" moz-do-not-send="true">Messages:24</a>
Dropped:0 Filtered:0 Rate:0.0%<br>
I (334997) canlog-vfs: Changing file name from
'/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd'
to
'/sd/log/ovms-vehicle/00000390/essai/00000004-20221126-160327.crtd'<br>
I (335007) canlog-vfs: Now logging CAN messages to
'/sd/log/ovms-vehicle/00000390/essai/00000004-20221126-160327.crtd'<br>
...</font></div>
<div class="moz-cite-prefix"><font face="monospace"><br>
</font></div>
<div class="moz-cite-prefix"><font face="monospace">OVMS# vfs
rls /sd/log/ovms-vehicle/00000390/essai/<br>
172 26-Nov-2022 15:59
/sd/log/ovms-vehicle/00000390/essai/00000001-19700101-010008.crtd<br>
1.9k 26-Nov-2022 16:01
/sd/log/ovms-vehicle/00000390/essai/00000002-20221126-155952.crtd<br>
2.1k 26-Nov-2022 16:03
/sd/log/ovms-vehicle/00000390/essai/00000003-20221126-160152.crtd<br>
0 30-Nov-1979 00:00
/sd/log/ovms-vehicle/00000390/essai/00000004-20221126-160327.crtd</font><br>
<br>
</div>
<div class="moz-cite-prefix">```<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Notes:</div>
<div class="moz-cite-prefix">
<ul>
<li>maxsize comparisons are done after writing, which
means that you will have files slightly bigger than
maxsize - not under maxsize. (for maxduration however it
should be OK as the comparison is done before writing)<br>
<br>
</li>
<li>While the comparison of maxsize is done on the real
number of bytes written, the concept of empty file (for
`<font face="monospace">[can] log.file.keep_empty</font>`)
is based on the number of messages logged to the file.<br>
This, because most of the files will have a header
written even if no messages are written ; which I
considered to be nevertheless an empty file.<br>
Those "empty" files with no messages will not be rotated
in that case (and kept open for logging).</li>
</ul>
<p>Let me know what you think of this, if you have some
remarks, and even if you feel it has no interest
whatsoever.</p>
<p>Regards,</p>
<p>Ludovic<br>
</p>
</div>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</body>
</html>