[Ovmsdev] OVMS Server v3
Michael Balzer
dexter at expeedo.de
Thu Mar 12 19:12:43 HKT 2020
Mark,
awesome, I'll have a look and try running the new version at the weekend.
Regarding config restructuring see OvmsConfig::upgrade().
Regards,
Michael
Am 11.03.20 um 08:00 schrieb Mark Webb-Johnson:
>
> This seems stable to me now. Running well, with no outstanding issues (at least in the configuration I am running).
>
> The code is all committed to github, and documentation in the usual place:
>
> https://docs.openvehicles.com/en/latest/server/index.html
>
>
> I’m now going to look at the OVMS module firmware to see if it is possible to support this scheme 0x31 in a neat way. The config ’server.v2’
> directory is a little messy at the moment (password stored there, rather than in password/server.v2, so the whole config section needs to be
> read-only).
>
> Regards, Mark.
>
>> On 6 Mar 2020, at 12:54 PM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>
>>
>> Wow, that was a lot more complex than I anticipated (in particular, making vehicleid’s unique per owner, not globally unique). 3,000
>> monolithic lines to 4,000 structured lines. Code has been committed to GitHub, and is running live on api.openvehicles.com
>> <http://api.openvehicles.com/> now.
>>
>> Warning: There may (will) be bugs. If killed all the obvious ones, but running live is still showing up some edge cases. I don’t recommend
>> anybody else runs this in production yet (particularly as rolling back would be non-trivial, as a result of the database changes).
>>
>> Major changes include:
>>
>> 1. Complete restructuring, to be plugin based. This allows optional functionality to be installed (for example, if you don’t require drupal
>> integration don’t enable that plugin), as well as new plugins to be developed. The list of plugins I am using on my production server
>> are: VECE, DbDBI, AuthDrupal, ApiV2, Push, PushAPNS, PushGCM, PushMAIL, ApiHttp, ApiHttpCore, and ApiHttpMqapi.
>>
>> 2. The system is designed to be installed and run from a github clone. Just clone the server repository, change directory to v3/server,
>> configure and run appropriately. The .gitignore file allows changes to be made to the configuration without affecting the master github.
>>
>> 3. Database format upgrades. See server/ovms_server_v2_to_v2.sql for the schema changes. If upgrading, you must deploy these database
>> changes (just ’source’ the script in the mysql console). The upgrade may take a while (in particular, the last stage for historical data).
>>
>> 4. The database no longer requires vehicle IDs to be globally unique. However, there are some caveats:
>>
>> * The v2 crypto 0x30 scheme only sends the vehicle ID, not the owner’s username (so if there are two vehicles in the database with the
>> same vehicle ID, we don’t know which one is the correct one). If you use that scheme (as everything does nowadays), vehicle ID still
>> needs to be globally unique for you. For this reason, my Drupal vehicle plugin still checks and enforces unique IDs. The 0x30 login
>> system won’t allow you to login to a vehicle who’s ID has more than one owner. Going forward, as we move away from 0x30 and ’server
>> password’, this will become less of an issue.
>>
>> * The new v3 crypto 0x31 scheme sends username as well as vehicle ID, so supports per-user vehicle IDs.
>>
>> * The HTTP API sends username, so supports per-user vehicle IDs.
>>
>> * Perhaps we should move to ‘VIN’ as vehicle ID?
>>
>> 5. The new crypto scheme 0x31 is much simpler to use, and script. Preferable, IMHO - but requires SSL/TLS for protection.
>>
>> 6. I’ve added support for API tokens, and love how they have turned out. I recommend this as the best approach going forward. Each token
>> belongs to a particular owner, and has privilege rights associated with it. Tokens can be created and revoked by the owner. It is much
>> easier to see which application (or car) has access to what. For example, the car module can ask for username+password, use that to
>> obtain a token, store it locally and use that for authentication going forward (with no need to store the server password at all).
>>
>> 7. Plugin authentication works well. Extend with new mechanisms as required. The ‘Authenticate’ plugin callback is merely passed
>> username+password and returns the permissions granted.
>>
>> 8. Push notifications are nicely modular and extendable. Already supports APNS, GCM, and MAIL.
>>
>> 9. I’ve called it v3, because this architecture will be able to cooperate with MQTT (for things like HTTP API, authentication, historical
>> data, push notifications, etc).
>>
>>
>> Still todo:
>>
>> * An option for automatic vehicle registration (easy).
>>
>> * Improvements to the token allocation API (in particular to retrieve existing tokens for specific application usage such as car modules).
>>
>> * A few miscellaneous functions not often used.
>>
>> * Documentation. I’ve documented the API changes (both v2 protocol for 0x31, and HTTP API extensions for authentication options and api
>> tokens), but still to document installation and configuration instructions for the server itself.
>>
>> * Administrator access. Still deciding the best way to handle this. Perhaps ’administrative api tokens’.
>>
>> * Permissions and Rights. The core is there, but need to extend to everything and document what rights are used for what. This is only
>> relevant for api tokens anyway (as the other two authentication methods grant ‘*’ rights anyway).
>>
>>
>> Final warning: Please don’t deploy this in productions systems yet. Changes are massive.
>>
>> Regards, Mark.
>>
>>> On 28 Feb 2020, at 3:47 PM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>>
>>>
>>> The new framework is running now on my development bench, and seems good. It was a major re-structuring, so I’ve upped the version to 3.0
>>> (particularly as this is going to end up connecting to MQTT for HTTP API, database logging, push notifications, authentication, etc). I
>>> should finish my testing this weekend, then bring it up on api.openvehicles.com <http://api.openvehicles.com/>. Others should migrate to it
>>> with care.
>>>
>>> Regarding the authentication, I have almost finished implementing:
>>>
>>> * An Auth Token facility (with authtokens stored against an associated owner ID).
>>> o The HTTP API (using username+password authentication) can be used to issue new tokens, as well as enquire on the tokens already issued.
>>> o The HTTP API can be used to enumerate registered vehicles, and otherwise maintain them.
>>> o The drupal website extension will be extended to also allow viewing and maintenance of tokens.
>>> o An authentication token is:
>>> + Owner ID (so zero or more tokens belong to this owner)
>>> + Token issued by the server on request (the token itself, unique, and the primary key)
>>> + Usage identifier (identifying the car module, app id, etc)
>>> + Usage description (a textual description of the usage)
>>> + Permissions (a permission string identifying what the token can be used for)
>>> + Created date+time
>>> + Updated date+time
>>> o Requests for a new token are passed the usage identifier and description as parameters, and authenticated by username+password.
>>> + If a token already exists for that usage identifier, then the description is simply updated (along with updated date+time) and
>>> the token returned.
>>> + If no token exists, a random one is created and returned (along with usage identifier, description, created and updated date+times).
>>>
>>> * An extension to MQTT authentication API to allow authentication either by username+password, or username+authtoken.
>>>
>>> * An extension to the HTTP API to allow authentication by username+token, in addition to the existing username+password.
>>>
>>> * An extension to the V2 API to allow authentication by username+password or username+authtoken, in addition to the existing
>>> vehicled+serverpassword (crypto scheme 0x31).
>>>
>>> * An extension to the V2 API to allow for optional automatic registration of new vehicles (a successful login with a non-existent vehicle
>>> ID will simply create it with a random server password).
>>>
>>> * Removed the restriction that vehicle ID should be unique on the server (now just unique for each user).
>>>
>>> * The preferred approach for a new app/car connection will then be:
>>> o The user is interactively asked to select a server and provide his username+password
>>> o A usage identifier and description is generated programatically
>>> o The HTTP API is used to obtain a token (or recall the token if previously registered)
>>> o The username, usage identifier, and token are stored persistently
>>> o The username, token, and vehicle ID is used to login using v2 or v3 protocols
>>>
>>> * So, three complementary authentication mechanisms are provided:
>>> o The v2 vehicleid+serverpassword mechanism (with full permissions to access that particular vehicle)
>>> o Username+Password mechanism (with full permission to maintain tokens, access all vehicles, and do everything)
>>> o Username+Token mechanism (with permissions specified on the token)
>>>
>>>
>>> Comments welcome, and we can refine the above if necessary, but it is at least a starting point. I am trying to maintain as much flexibility
>>> as possible, but at the same time make things easier for the user. I’ve had four support requests so far this week for people messing up
>>> either the server they are using (app api.openvehicles, car dexters-web), the vehicle ID, or confusion between all the passwords.
>>>
>>> Regards, Mark.
>>>
>>>> On 23 Feb 2020, at 8:54 PM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>>>
>>>> This is turning into a bigger job than I imagined. "Give a mouse a cookie", and all that.
>>>>
>>>> The ovms_server.pl has gotten horrendous over the years. Almost 3,000 monolithic lines of code, 4 server listeners, three different types
>>>> of server, push notifications, database synchronisation, etc. I tried turning on ‘use strict’ and it showed up a bunch of bugs and errors.
>>>>
>>>> So, I am refactoring it to a plugin architecture. That should make it more maintainable, and also provide a foundation for it to work
>>>> better with the v3 MQTT. I suggest people hold off from making any changes to the server code in the next few days.
>>>>
>>>> Regards, Mark
>>>>
>>>>> On 21 Feb 2020, at 9:48 PM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>>>>
>>>>> Ok. I will rework a modular approach. Should be able to get this done over the weekend.
>>>>>
>>>>> Yes, strict and warn would help.
>>>>>
>>>>> Mark
>>>>>
>>>>> P.S. Explains why nobody used the http api on my server :-)
>>>>>
>>>>>> On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>>>
>>>>>> Thanks Mark,
>>>>>>
>>>>>> I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere
>>>>>> else. Maybe "strict" mode would have told me about that.
>>>>>>
>>>>>> And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my
>>>>>> password hashing…
>>>>>>
>>>>>> A modular solution seems to be best, easy to add custom implementations and to provide some standard modules.
>>>>>>
>>>>>> Regards,
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson:
>>>>>>>
>>>>>>> An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup:
>>>>>>>
>>>>>>>
>>>>>>> require $config->val('db',’pw_module’,’auth_none.pl’);
>>>>>>>
>>>>>>>
>>>>>>> …
>>>>>>>
>>>>>>>
>>>>>>> If (&auth_password_check($passwordhash, $password))
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>> Provide a ‘auto_none.pl’:
>>>>>>>
>>>>>>> #!/usr/bin/perl
>>>>>>>
>>>>>>> sub auth_password_check
>>>>>>> {
>>>>>>> my ($hash,$password) = @_;
>>>>>>>
>>>>>>> return 0;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc.
>>>>>>>
>>>>>>> This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the
>>>>>>> old-style require.
>>>>>>>
>>>>>>> That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if
>>>>>>> we have a separate module that drupal-dependant code could be removed from ovms_server.pl.
>>>>>>>
>>>>>>> Is that a better solution?
>>>>>>>
>>>>>>> Regards, Mark.
>>>>>>>
>>>>>>>> On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>>>>>>>
>>>>>>>> Michael,
>>>>>>>>
>>>>>>>> Just before your commit, the server code was:
>>>>>>>>
>>>>>>>> my $passwordhash = $row->{'pass'};
>>>>>>>> if (&drupal_password_check($passwordhash, $password))
>>>>>>>>
>>>>>>>> …
>>>>>>>>
>>>>>>>> sub drupal_password_check
>>>>>>>> {
>>>>>>>> my ($ph,$password) = @_;
>>>>>>>>
>>>>>>>> my $iter_log2 = index($itoa64,substr($ph,3,1));
>>>>>>>> my $iter_count = 1 << $iter_log2;
>>>>>>>>
>>>>>>>> my $phash = substr($ph,0,12);
>>>>>>>> my $salt = substr($ph,4,8);
>>>>>>>>
>>>>>>>> my $hash = sha512($salt.$password);
>>>>>>>> do
>>>>>>>> {
>>>>>>>> $hash = sha512($hash.$password);
>>>>>>>> $iter_count--;
>>>>>>>> } while ($iter_count > 0);
>>>>>>>>
>>>>>>>> my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
>>>>>>>>
>>>>>>>> return ($encoded eq $ph);
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Your change was:
>>>>>>>>
>>>>>>>> # User password encoding function:
>>>>>>>> my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’);
>>>>>>>>
>>>>>>>> …
>>>>>>>>
>>>>>>>> my $passwordhash = $row->{'pass'};
>>>>>>>> my $encoded = eval $pw_encode;
>>>>>>>> if ($encoded eq $passwordhash)
>>>>>>>>
>>>>>>>> …
>>>>>>>>
>>>>>>>> sub drupal_password
>>>>>>>> {
>>>>>>>> my ($password) = @_;
>>>>>>>>
>>>>>>>> my $iter_log2 = index($itoa64,substr($ph,3,1));
>>>>>>>> my $iter_count = 1 << $iter_log2;
>>>>>>>>
>>>>>>>> my $phash = substr($ph,0,12);
>>>>>>>> my $salt = substr($ph,4,8);
>>>>>>>>
>>>>>>>> my $hash = sha512($salt.$password);
>>>>>>>> do
>>>>>>>> {
>>>>>>>> $hash = sha512($hash.$password);
>>>>>>>> $iter_count--;
>>>>>>>> } while ($iter_count > 0);
>>>>>>>>
>>>>>>>> my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
>>>>>>>>
>>>>>>>> return $encoded;
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash)
>>>>>>>> to extract meta data to set the encoding parameters.
>>>>>>>>
>>>>>>>> The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the
>>>>>>>> password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the
>>>>>>>> hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables
>>>>>>>> to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables
>>>>>>>> and make brute force approaches computationally unfeasible.
>>>>>>>>
>>>>>>>> For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous
>>>>>>>> encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new
>>>>>>>> password in the same way.
>>>>>>>>
>>>>>>>> This won’t just affect drupal, but any system with a non-trivial password hashing function.
>>>>>>>>
>>>>>>>> So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make
>>>>>>>> it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will
>>>>>>>> allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup
>>>>>>>> (ldap, etc).
>>>>>>>>
>>>>>>>> Regards, Mark.
>>>>>>>>
>>>>>>>>> On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>>>>>>
>>>>>>>>> Mark,
>>>>>>>>>
>>>>>>>>> I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the
>>>>>>>>> "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example).
>>>>>>>>>
>>>>>>>>> That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function
>>>>>>>>> for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you
>>>>>>>>> please elaborate? I'd like to understand what was going wrong.
>>>>>>>>>
>>>>>>>>> With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check"
>>>>>>>>> hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that
>>>>>>>>> simply reuses the existing "pw_encode" hook.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Michael
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson:
>>>>>>>>>> Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication
>>>>>>>>>> didn’t work.
>>>>>>>>>>
>>>>>>>>>> It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also
>>>>>>>>>> weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and
>>>>>>>>>> the other by MQ authentication).
>>>>>>>>>>
>>>>>>>>>> I standardised to use a new pw_check (overridable in the config) parameter, which defaults to:
>>>>>>>>>>
>>>>>>>>>> drupal_password_check($passwordhash,$password)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both
>>>>>>>>>> authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the
>>>>>>>>>> pw_check parameter in the config.
>>>>>>>>>>
>>>>>>>>>> I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach.
>>>>>>>>>>
>>>>>>>>>> Regards, Mark
>>>>>>>>>>
>>>>>>>>>>> On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark at webb-johnson.net <mailto:mark at webb-johnson.net>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Strange. I have zero using mine. Must be a EU thing?
>>>>>>>>>>>
>>>>>>>>>>> I’ll keep it in mind and try not to break anything.
>>>>>>>>>>>
>>>>>>>>>>> Regards, Mark.
>>>>>>>>>>>
>>>>>>>>>>>> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Mark,
>>>>>>>>>>>>
>>>>>>>>>>>> grep "main: http" in the log: yes, I've got some users accessing the API frequently.
>>>>>>>>>>>>
>>>>>>>>>>>> Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Michael
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson:
>>>>>>>>>>>>> Is anyone here using the HTTP API at all?
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems so tied to the v2 protocol, as to not be much use.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards, Mark.
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>> _______________________________________________
>>>>>>>>> OvmsDev mailing list
>>>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OvmsDev mailing list
>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20200312/85d07667/attachment.htm>
More information about the OvmsDev
mailing list