<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Hi Mark,<br>
    <br>
    I presume no changes to CAN_frame_t that would be visible to other
    applications, e.g. OBD2ECU, right?<br>
    <br>
    Also, any chance that the changes to MCP2515 might help diagnose the
    cause of our bus hangs?<br>
    <br>
    Greg<br>
    <br>
    <br>
    <div class="moz-cite-prefix">Mark Webb-Johnson wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:182AA6E6-0E33-41E4-B4BE-B378048B47BA@webb-johnson.net">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <div class=""><br class="">
      </div>
      I’ve completed most of the work of this restructuring, and the new
      arrangement looks good. The new work uses CAN_log_message_t
      (rather than CAN_frame_t) for most things other than basic frame
      tx/rx, and that is based on Michael’s can log CAN_LogMsg_t (almost
      unchanged, apart from timestamp). I’ve added a canfilter class to
      be able to do software filtering (not just in logging, but also in
      general). Then candump* has moved to canformat* to virtualise all
      the different formats (CRTD, PCAP, etc) for can dumps (with a
      modular registration of formats so we can add more formats easily,
      without change to core components).
      <div class=""><br class="">
      </div>
      <div class="">The last change I have to make is to canlog. I want
        it to use the canfilter and canformat systems so all it is doing
        is logging (and not involved in filtering, or formatting - both
        of which are now general purpose facilities). It seems that the
        ‘re serve’ framework can move into this, and perhaps we can log
        to things other than the file system.</div>
      <div class=""><br class="">
      </div>
      <div class="">The changes were quite extensive:</div>
      <div class=""><br class="">
      </div>
      <div class="">
        <ul class="MailOutline">
          <li class="">Everything in the can component</li>
          <li class="">retools component</li>
          <li class="">mcp2515 component</li>
          <li class="">esp32can component</li>
          <li class="">dbc component</li>
        </ul>
        <div><br class="">
        </div>
        <div>I should be done with the changes in the next day or two,
          and then will do some testing in my car. Hopefully should be
          able to merge my commits later this week.</div>
        <div><br class="">
        </div>
        <div>Regards, Mark.</div>
        <div><br class="">
          <blockquote type="cite" class="">
            <div class="">On 12 Jun 2019, at 9:00 PM, Mark Webb-Johnson
              <<a href="mailto:mark@webb-johnson.net" class=""
                moz-do-not-send="true">mark@webb-johnson.net</a>>
              wrote:</div>
            <br class="Apple-interchange-newline">
            <div class="">
              <meta http-equiv="Content-Type" content="text/html;
                charset=UTF-8" class="">
              <div style="word-wrap: break-word; -webkit-nbsp-mode:
                space; line-break: after-white-space;" class="">I also
                see the filter system in canlog could be very useful as
                a general filtering system. I suggest to move that into
                the central CAN class, and allow such software filters
                to be attached to listeners, callbacks, and loggers.
                <div class=""><br class="">
                </div>
                <div class="">By refactoring this, I think we can get
                  some very useful functionality.</div>
                <div class=""><br class="">
                </div>
                <div class="">@Michael, one question, for you: What is
                  the purpose of RegisterCallback (and the callbacks vs
                  listeners, in general). Listeners get their frames
                  delivered via a queue (so must have a task running).
                  Callbacks are simple function callbacks running
                  synchronously with the CAN task. From what I can see,
                  the only user of callbacks is vehicle_twizy where
                  CanResponder is set as the callback on CAN1. Why is
                  that necessary, given that the standard vehicle object
                  has IncomingFrameCan1() to receive frames. Is this
                  required for performance (running synchronously with
                  the CAN receive task), or some other reason why it
                  can’t run from the vehicle task on queue receive of
                  the frame? In other words, why can’t we just call
                  CanResponder() from within IncomingFrameCan1()?</div>
                <div class=""><br class="">
                </div>
                <div class="">Regards, Mark.<br class="">
                  <div class=""><br class="">
                    <blockquote type="cite" class="">
                      <div class="">On 11 Jun 2019, at 8:49 PM, Mark
                        Webb-Johnson <<a
                          href="mailto:mark@webb-johnson.net" class=""
                          moz-do-not-send="true">mark@webb-johnson.net</a>>
                        wrote:</div>
                      <br class="Apple-interchange-newline">
                      <div class="">
                        <meta http-equiv="Content-Type"
                          content="text/html; charset=UTF-8" class="">
                        <div style="word-wrap: break-word;
                          -webkit-nbsp-mode: space; line-break:
                          after-white-space;" class="">It has become a
                          little duplicated:
                          <div class=""><br class="">
                          </div>
                          <blockquote style="margin: 0 0 0 40px; border:
                            none; padding: 0px;" class="">
                            <div class="">
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">struct
                                    CAN_frame_t</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">  {</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">  canbus*
                                        origin;                   //
                                    Origin of the frame</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class=""> 
                                    CAN_FIR_t   FIR;                    
                                     // Frame information record</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class=""> 
                                    uint32_t    MsgID;                  
                                     // Message ID</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">  union</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">    {</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">   
                                    uint8_t   u8[8];                  
                                     // Payload byte access</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">   
                                    uint32_t  u32[2];                  
                                    // Payload u32 access (Att: little
                                    endian!)</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">   
                                    uint64_t  u64;                    
                                     // Payload u64 access (Att: little
                                    endian!)</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">    }
                                    data;</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class=""><br
                                      class="">
                                  </span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class=""> 
                                    esp_err_t Write(canbus* bus=NULL,
                                    TickType_t maxqueuewait=0);  // bus:
                                    NULL=origin</span></font></div>
                              <div class=""><font class="" face="Andale
                                  Mono"><span style="font-style: normal;
                                    font-size: 14px;" class="">  };</span></font></div>
                            </div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-style: normal;
                                  font-size: 14px;" class=""><br
                                    class="">
                                </span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-size: 14px;"
                                  class="">typedef struct<br class="">
                                    {<br class="">
                                    uint32_t timestamp;<br class="">
                                    canbus* bus;<br class="">
                                    CAN_LogEntry_t type;<br class="">
                                    union<br class="">
                                      {<br class="">
                                      CAN_frame_t frame;<br class="">
                                      CAN_status_t status;<br class="">
                                      char* text;<br class="">
                                      };<br class="">
                                    } CAN_LogMsg_t;</span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-size: 14px;"
                                  class=""><br class="">
                                </span></font></div>
                            <div class=""><font class="" face="Andale
                                Mono"><span style="font-size: 14px;"
                                  class="">
                                  <div class="">typedef struct</div>
                                  <div class="">  {</div>
                                  <div class="">  CAN_frame_t last;</div>
                                  <div class="">  uint32_t rxcount;</div>
                                  <div class="">  struct
                                    __attribute__((__packed__))</div>
                                  <div class="">    {</div>
                                  <div class="">    struct {</div>
                                  <div class="">      uint8_t Ignore:1;
                                        // 0x01</div>
                                  <div class="">      uint8_t Changed:1;
                                       // 0x02</div>
                                  <div class="">      uint8_t
                                    Discovered:1; // 0x04</div>
                                  <div class="">      uint8_t :1;      
                                        // 0x08</div>
                                  <div class="">      uint8_t :1;      
                                        // 0x10</div>
                                  <div class="">      uint8_t :1;      
                                        // 0x20</div>
                                  <div class="">      uint8_t :1;      
                                        // 0x40</div>
                                  <div class="">      uint8_t :1;      
                                        // 0x80</div>
                                  <div class="">      } b;</div>
                                  <div class="">    uint8_t dc;        
                                        // Data bytes changed</div>
                                  <div class="">    uint8_t dd;        
                                        // Data bytes discovered</div>
                                  <div class="">    uint8_t spare;</div>
                                  <div class="">    } attr;</div>
                                  <div class="">  } re_record_t;</div>
                                </span></font></div>
                          </blockquote>
                          <div class="">
                            <div class=""><br class="">
                            </div>
                            <div class="">CAN_frame_t has origin for the
                              can bus the message arrived on, while
                              CAN_LogMsg_t has frame.origin and bus
                              (presumably because bus is needed for
                              status and text messages).</div>
                            <div class=""><br class="">
                            </div>
                            <div class="">Then we have candump and
                              canlog virtual interfaces (canlog supports
                              ’trace’ and ‘crtd’, while candump supports
                              ‘crtd’ and ‘pap’).</div>
                            <div class=""><br class="">
                            </div>
                          </div>
                          <div class="">The users of these higher level
                            structures are RETOOLS and CANLOG. I suggest
                            the following changes:</div>
                          <div class=""><br class="">
                          </div>
                          <div class="">
                            <ul class="MailOutline">
                              <li class="">Have a single format
                                conversion virtual class, and then
                                initial implementations for pcap, crtd
                                and trace. That can virtualise the
                                conversion of a CAN message to be
                                logged/transmitted/received to/from the
                                external format. This can include a
                                factory for registration of types (by
                                textual name), as well as support
                                functions for command registration
                                (including whether they have support for
                                reading, writing, or both). This just
                                provides format conversion and does not
                                provide any file handling or
                                transmission functions. Canlog and
                                Candump get merged to form this.<br
                                  class="">
                                <br class="">
                              </li>
                              <li class="">Perhaps base this on
                                CAN_LogMsg_t, but I suggest a more
                                generic name (perhaps CAN_message_t, to
                                differentiate that this is a can message
                                rather than a simple frame). It is
                                CAN_frame_t + additional support for
                                stats and general text messages.<br
                                  class="">
                                <br class="">
                              </li>
                              <li class="">Tidy up the CAN_LogMsg_t bus
                                vs frame.origin issue.<br class="">
                              </li>
                            </ul>
                          </div>
                          <div class=""><br class="">
                          </div>
                          <div class="">Simply put, separate the
                            formatting into a standalone virtual
                            implementations, then leave retools and
                            canlog to handle the actual RE stuff or file
                            logging, or whatever else is required with
                            the data that gets converted.</div>
                          <div class=""><br class="">
                          </div>
                          <div class="">Does that make sense?</div>
                          <div class=""><br class="">
                          </div>
                          <div class="">Regards, Mark.</div>
                          <div class=""><br class="">
                          </div>
                          <div class="">
                            <div class="">
                              <blockquote type="cite" class="">
                                <div class="">On 11 Jun 2019, at 3:01
                                  PM, Michael Balzer <<a
                                    href="mailto:dexter@expeedo.de"
                                    class="" moz-do-not-send="true">dexter@expeedo.de</a>>
                                  wrote:</div>
                                <br class="Apple-interchange-newline">
                                <div class="">
                                  <div class="">Mark,<br class="">
                                    <br class="">
                                    when implementing the can log
                                    framework I introduced the
                                    CAN_LogMsg_t specifically to capture
                                    the original timestamp of the frame,
                                    so it wouldn't get lost if<br
                                      class="">
                                    the logging task gets behind, and to
                                    transport other log data types than
                                    just frames to the loggers.<br
                                      class="">
                                    <br class="">
                                    I would also expect the RE tools
                                    stream to produce event info &
                                    statistics entries like can log
                                    does, as that is very helpful in
                                    analysing. Maybe rebasing<br
                                      class="">
                                    candump on can log would be better?<br
                                      class="">
                                    <br class="">
                                    Regards,<br class="">
                                    Michael<br class="">
                                    <br class="">
                                    <br class="">
                                    Am 11.06.19 um 08:37 schrieb Mark
                                    Webb-Johnson:<br class="">
                                    <blockquote type="cite" class="">This
                                      really should be unified, and
                                      probably not too hard to do. Crazy
                                      to have two so similar frameworks.
                                      I think we can just have one
                                      virtual class for converting CAN
                                      frames to/from other formats, and
                                      then implementations for CRTD,
                                      PCAP, etc.<br class="">
                                      <br class="">
                                      I do remember that last time I
                                      looked at this, canlog used
                                      CAN_LogMsg_t, and candump used
                                      CAN_frame_t (with CAN_LogMsg_t
                                      having some other fields for
                                      logging statistics, etc), and that
                                      made it non-trivial. Perhaps
                                      canlog should be built on top of
                                      candump?<br class="">
                                      <br class="">
                                      I will have a look at it.<br
                                        class="">
                                      <br class="">
                                      Regards, Mark.<br class="">
                                      <br class="">
                                      <blockquote type="cite" class="">On
                                        6 Jun 2019, at 4:06 PM, Michael
                                        Balzer <<a
                                          href="mailto:dexter@expeedo.de"
                                          class=""
                                          moz-do-not-send="true">dexter@expeedo.de</a>>
                                        wrote:<br class="">
                                        <br class="">
                                        Steve,<br class="">
                                        <br class="">
                                        Mark quoted from the RE tools
                                        candump_crtd class, which has
                                        its own crtd formatter.<br
                                          class="">
                                        <br class="">
                                        For the canlog framework I
                                        currently get the timestamp from
                                        esp_log_timestamp(). Should be
                                        easy to change that to
                                        gettimeofday().<br class="">
                                        <br class="">
                                        Regards,<br class="">
                                        Michael<br class="">
                                        <br class="">
                                        <br class="">
                                        Am 05.06.19 um 18:39 schrieb
                                        Stephen Casner:<br class="">
                                        <blockquote type="cite" class="">Mark,<br
                                            class="">
                                          <br class="">
                                          If by "unix julian time" you
                                          mean a timestamp like
                                          1559751698 (the<br class="">
                                          time as I write this), than
                                          that is not what Greg and I
                                          have observed<br class="">
                                          in the CRTD logs.  Here are
                                          the first lines in our files:<br
                                            class="">
                                          <br class="">
                                          151766.574 CXX Info Type:crtd;
                                          Path:'/sd/gps.crtd';
                                          Filter:1:100-100;
                                          Vehicle:TR2N;<br class="">
                                          <br class="">
                                          383603.293 CXX Info Type:crtd;
                                          Path:'/sd/location.crtd';
                                          Filter:1:100-100;
                                          Vehicle:TR1N;<br class="">
                                          <br class="">
                                          Yet OVMS does have the current
                                          time from NTP:<br class="">
                                          <br class="">
                                          OVMS# time<br class="">
                                          Time Zone:<br class="">
                                          UTC Time:   2019-06-05
                                          16:35:45 UTC<br class="">
                                          Local Time: 2019-06-05
                                          16:35:45 GMT<br class="">
                                          Provider:   ntp<br class="">
                                          <br class="">
                                          PROVIDER             STRATUM
                                           UPDATE TIME<br class="">
                                          *ntp                       1
                                               57 Wed Jun  5 16:35:44
                                          2019<br class="">
                                          <br class="">
                                          Who calls candump_crtd::get()
                                          with the timeval that goes
                                          into the<br class="">
                                          sprintf() you mention?<br
                                            class="">
                                          <br class="">
                                                      -- Steve<br
                                            class="">
                                          <br class="">
                                          On Wed, 5 Jun 2019, Mark
                                          Webb-Johnson wrote:<br
                                            class="">
                                          <br class="">
                                          <blockquote type="cite"
                                            class="">The CRTD timestamps
                                            should be unix julian time
                                            (not uptime). Produced by
                                            this:<br class="">
                                            <br class="">
                                            sprintf(m_buf,"%ld.%06ld
                                            %sR%s %0*X",<br class="">
                                              time->tv_sec,
                                            time->tv_usec,<br
                                              class="">
                                              busnumber,<br class="">
                                              (frame->FIR.B.FF ==
                                            CAN_frame_std) ? "11":"29",<br
                                              class="">
                                              (frame->FIR.B.FF ==
                                            CAN_frame_std) ? 3 : 8,<br
                                              class="">
                                              frame->MsgID);<br
                                              class="">
                                            <br class="">
                                            Once module gets time from
                                            the vehicle (or gps), it
                                            should show the correct
                                            timestamps.<br class="">
                                            <br class="">
                                            Regards, Mark.<br class="">
                                          </blockquote>
_______________________________________________<br class="">
                                          OvmsDev mailing list<br
                                            class="">
                                          <a
                                            href="mailto:OvmsDev@lists.openvehicles.com"
                                            class=""
                                            moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                            class="">
                                          <a
                                            href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                            class=""
                                            moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                            class="">
                                        </blockquote>
                                        -- <br class="">
                                        Michael Balzer * Helkenberger
                                        Weg 9 * D-58256 Ennepetal<br
                                          class="">
                                        Fon 02333 / 833 5735 * Handy
                                        0176 / 206 989 26<br class="">
                                        <br class="">
                                        <br class="">
_______________________________________________<br class="">
                                        OvmsDev mailing list<br class="">
                                        <a
                                          href="mailto:OvmsDev@lists.openvehicles.com"
                                          class=""
                                          moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                          class="">
                                        <a
                                          href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                          class=""
                                          moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                          class="">
                                      </blockquote>
_______________________________________________<br class="">
                                      OvmsDev mailing list<br class="">
                                      <a
                                        href="mailto:OvmsDev@lists.openvehicles.com"
                                        class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                        class="">
                                      <a
                                        href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                        class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                        class="">
                                    </blockquote>
                                    <br class="">
                                    -- <br class="">
                                    Michael Balzer * Helkenberger Weg 9
                                    * D-58256 Ennepetal<br class="">
                                    Fon 02333 / 833 5735 * Handy 0176 /
                                    206 989 26<br class="">
                                    <br class="">
                                    <br class="">
_______________________________________________<br class="">
                                    OvmsDev mailing list<br class="">
                                    <a
                                      href="mailto:OvmsDev@lists.openvehicles.com"
                                      class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                                      class="">
                                    <a
                                      href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
                                      class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                                      class="">
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                            <br class="">
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <br class="">
                </div>
              </div>
              _______________________________________________<br
                class="">
              OvmsDev mailing list<br class="">
              <a href="mailto:OvmsDev@lists.openvehicles.com" class=""
                moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
                class="">
              <a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
                class="">
            </div>
          </blockquote>
        </div>
        <br class="">
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>