[Ovmsdev] MQTT and Ovms Server v3

Tue Jul 17 09:37:23 HKT 2018

Jacob,

> Is there a reason why OVMS bundles three different SSL libraries?

I think wolfSSL is there (but WOLFCRYPT_ONLY) because wolfSSH needs it.

The ESP-IDF offers mbedtls and openssl libraries, but we don’t currently use either.

> Also why is there a custom HTTP implementation when mongoose already has one?

Primarily because we wrote those before we included mongoose into the project. But also because the mongoose http_client code sucks. It reads the entire message body into RAM, before passing it on to the client; that just won’t work for something like a firmware image.

To complicate the choice further, ESP now includes a esp_http_client in their latest 3.x IDF (which wasn’t there before).

I did try to convert our TCP and HTTP client libraries to mongoose a while ago, but failed. Those libraries are blocking (with i/o done in the context of the thread calling them), but the mongoose library is event-based non-blocking (with i/o output in the context of the calling thread, and i/o input in the context of the mongoose thread). Mongoose didn’t use to be thread safe, and that caused us all sorts of issues (now solved, by the work Michael and others did to make it thread safe).

> IMO the best way would be to rewrite ovms_net and
> ovms_http to use mongoose and configure mongoose to compile with SSL
> support (probably mbedtls as the other two don't work).

I agree. In general, I prefer the mongoose approach, and we are trying to standardise on that (for good or bad).

Probably the best would be to fix the http client in mongoose to work properly (or at least have an option to deliver the body block by block as they arrive), then convert our stuff that uses ovms tcp/http libraries, then drop the tcp and http libraries. Not trivial given the different models (blocking vs non-blocking events).

Between openssl vs mbedtls, I don’t really care. I think we’re going to have to manage certificate loading ourselves anyway (as we need to load from a combination of statically defined (in flash using COMPONENT_EMBED_TXTFILES and asm(“_binary_* start/end) and dynamic (in /store/tls/trustedca/* or somewhere like that).

Regards, Mark.

> On 17 Jul 2018, at 1:02 AM, Jakob Löw <ovms at m4gnus.de> wrote:
> 
> Hey,
> 
> I've been playing around a bit with implementing SSL/TLS support. For
> starters I wanted to implement an OvmsNetTlsConnection class which
> could then be used for https requests. At first I tried using wolfSSL
> but it turns out wolfSSL is compiled with WOLFCRYPT_ONLY. OpenSSL also
> didn't work as SSL_CTX_load_verify_locations is missing in ESP-IDF's
> OpenSSL. I didn't try mbedtls yet. Is there a reason why OVMS bundles
> three different SSL libraries? Also why is there a custom HTTP
> implementation when mongoose already has one? Speaking of mongoose, it
> seems to have an abstraction layer which allows to use one of the three
> SSL/TLS libraries. IMO the best way would be to rewrite ovms_net and
> ovms_http to use mongoose and configure mongoose to compile with SSL
> support (probably mbedtls as the other two don't work).
> 
> - Jakob
> 
> On Mon, 2018-07-16 at 12:47 +0800, Mark Webb-Johnson wrote:
>> Drupal authentication is done, and working against mosquitto now on
>> the live api.openvehicles.com site. Plain text MQTT only at the
>> moment (I still need to get mosquitto and OVMS module firmware
>> supporting SSL).
>> 
>> To configure OVMS v3 to use Server V3 protocol:
>> 
>> config set:
>> vehicle id <vehicleid>
>> server.v3 server api.openvehicles.com
>> server.v3 port 1883
>> server.v3 user <openvehicles.com username>
>> password server.v3 <openvehicles.com password>
>> 
>> server v3 start
>> 
>> Can also 'config set auto server.v3 yes’ to auto-start at boot
>> 
>> If you have an MQTT client, you can connect to the same server to
>> query the metrics, issue commands, etc. The default topic hierarchy
>> is ovms/<username>/<vehicleid>.
>> 
>> I think we are close now. Just need SSL support. And then Apps…
>> 
>> Feedback appreciated.
>> 
>> Regards, Mark.
>> 
>>> On 10 Jul 2018, at 2:42 PM, Mark Webb-Johnson <mark at webb-johnson.ne
>>> t> wrote:
>>> 
>>> I’ve just pushed the support for notifications in OVMS Server v3.
>>> 
>>> With that, the server v3 is functionally complete (at least on the
>>> car side). We can:
>>> 
>>> Connect/disconnect
>>> Handle lists of apps connecting/disconnecting
>>> Send metrics
>>> Send events
>>> Send notifications (including info, error, alert, and historical
>>> data)
>>> Receive commands, run them, and return the results
>>> 
>>> I’m still struggling against drupal authentication at the server
>>> side; once that is done we can open this up to wider testing.
>>> 
>>> Overall, I’m pretty happy with it. We need SSL/TLS support, but at
>>> least functionally now it works and is complete.
>>> 
>>> Regards, Mark
>>> 
>>>> On 6 Jul 2018, at 4:16 PM, Mark Webb-Johnson <mark at webb-johnson.n
>>>> et> wrote:
>>>> 
>>>>>>> I suppose the following topic names:
>>>>>>> metrics:       <prefix>/metric/#
>>>>>>> events:        <prefix>/event/#
>>>>>>> notifications: <prefix>/notify/#
>>>>>>> config:        <prefix>/config/#
>>>>>>> logs:          <prefix>/log/<tag>
>>>>>>> active:        <prefix>/client/<clientid>/active
>>>>>>> requests:      <prefix>/client/<clientid>/request/#
>>>>>>> commands:      <prefix>/client/<clientid>/command/<command
>>>>>>> id>
>>>>>>> cmd responses: <prefix>/client/<clientid>/response/<command
>>>>>>> id>
>>>>>> 
>>>>>> All ok. I am fine with this, and looks clean. I will make the
>>>>>> changes today, as I want to get this into some cars asap so
>>>>>> we can get a feel for how it behaves.
>>>> 
>>>> Ok. Now implemented, and tested:
>>>> 
>>>> <prefix>/metric/#
>>>> <prefix>/client/<clientid>/active
>>>> <prefix>/client/<clientid>/command/<command id>
>>>> <prefix>/client/<clientid>/response/<command id>
>>>> 
>>>> I don’t think config, logs, and requests are critical or urgent.
>>>> So, I will try to finish events tonight (as that is relatively
>>>> simple). Notifications over the weekend (more tricky, especially
>>>> for historical data). It can go in my car with tonight’s nightly
>>>> ota, for real world testing.
>>>> 
>>>> I’m still trying to get the authentication working for drupal vs
>>>> mosquitto. Once that is done, I can open up
>>>> api.openvehicles.com MQTT for public use. If I can’t get it done
>>>> within the next couple of days, I’ll try another broker (lua
>>>> scripting, anyone?).
>>>> 
>>>> P.S. Commands over MQTT are pretty cool:
>>>> 
>>>> <PastedGraphic-2.tiff>
>>>> <PastedGraphic-3.tiff>
>>>> 
>>>> Regards, Mark.
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>> 
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>> 
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev