Anyone using HTTP API?
Is anyone here using the HTTP API at all? It seems so tied to the v2 protocol, as to not be much use. Regards, Mark.
Mark, grep "main: http" in the log: yes, I've got some users accessing the API frequently. Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. Regards, Michael Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson:
Is anyone here using the HTTP API at all?
It seems so tied to the v2 protocol, as to not be much use.
Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Strange. I have zero using mine. Must be a EU thing? I’ll keep it in mind and try not to break anything. Regards, Mark.
On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de> wrote:
Mark,
grep "main: http" in the log: yes, I've got some users accessing the API frequently.
Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days.
Regards, Michael
Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson:
Is anyone here using the HTTP API at all?
It seems so tied to the v2 protocol, as to not be much use.
Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work. It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication). I standardised to use a new pw_check (overridable in the config) parameter, which defaults to: drupal_password_check($passwordhash,$password) and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config. I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach. Regards, Mark
On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Strange. I have zero using mine. Must be a EU thing?
I’ll keep it in mind and try not to break anything.
Regards, Mark.
On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de> wrote:
Mark,
grep "main: http" in the log: yes, I've got some users accessing the API frequently.
Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days.
Regards, Michael
Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson:
Is anyone here using the HTTP API at all?
It seems so tied to the v2 protocol, as to not be much use.
Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Mark, I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example). That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong. With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook. Regards, Michael Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson:
Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work.
It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication).
I standardised to use a new pw_check (overridable in the config) parameter, which defaults to:
drupal_password_check($passwordhash,$password)
and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config.
I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach.
Regards, Mark
On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Strange. I have zero using mine. Must be a EU thing?
I’ll keep it in mind and try not to break anything.
Regards, Mark.
On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
grep "main: http" in the log: yes, I've got some users accessing the API frequently.
Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days.
Regards, Michael
Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson:
Is anyone here using the HTTP API at all?
It seems so tied to the v2 protocol, as to not be much use.
Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Michael, Just before your commit, the server code was: my $passwordhash = $row->{'pass'}; if (&drupal_password_check($passwordhash, $password)) … sub drupal_password_check { my ($ph,$password) = @_; my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2; my $phash = substr($ph,0,12); my $salt = substr($ph,4,8); my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0); my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); return ($encoded eq $ph); } Your change was: # User password encoding function: my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’); … my $passwordhash = $row->{'pass'}; my $encoded = eval $pw_encode; if ($encoded eq $passwordhash) … sub drupal_password { my ($password) = @_; my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2; my $phash = substr($ph,0,12); my $salt = substr($ph,4,8); my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0); my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); return $encoded; } You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters. The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible. For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way. This won’t just affect drupal, but any system with a non-trivial password hashing function. So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc). Regards, Mark.
On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de> wrote:
Mark,
I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example).
That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong.
With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook.
Regards, Michael
Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson:
Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work.
It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication).
I standardised to use a new pw_check (overridable in the config) parameter, which defaults to:
drupal_password_check($passwordhash,$password)
and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config.
I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach.
Regards, Mark
On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Strange. I have zero using mine. Must be a EU thing?
I’ll keep it in mind and try not to break anything.
Regards, Mark.
On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
grep "main: http" in the log: yes, I've got some users accessing the API frequently.
Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days.
Regards, Michael
Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson:
Is anyone here using the HTTP API at all?
It seems so tied to the v2 protocol, as to not be much use.
Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup: require $config->val('db',’pw_module’,’auth_none.pl’); … If (&auth_password_check($passwordhash, $password)) ... Provide a ‘auto_none.pl’: #!/usr/bin/perl sub auth_password_check { my ($hash,$password) = @_; return 0; } Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc. This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require. That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl. Is that a better solution? Regards, Mark.
On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Michael,
Just before your commit, the server code was:
my $passwordhash = $row->{'pass'}; if (&drupal_password_check($passwordhash, $password))
…
sub drupal_password_check { my ($ph,$password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return ($encoded eq $ph); }
Your change was:
# User password encoding function: my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’);
…
my $passwordhash = $row->{'pass'}; my $encoded = eval $pw_encode; if ($encoded eq $passwordhash)
…
sub drupal_password { my ($password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return $encoded; }
You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters.
The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible.
For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way.
This won’t just affect drupal, but any system with a non-trivial password hashing function.
So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc).
Regards, Mark.
On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example).
That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong.
With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook.
Regards, Michael
Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson:
Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work.
It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication).
I standardised to use a new pw_check (overridable in the config) parameter, which defaults to:
drupal_password_check($passwordhash,$password)
and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config.
I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach.
Regards, Mark
On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Strange. I have zero using mine. Must be a EU thing?
I’ll keep it in mind and try not to break anything.
Regards, Mark.
On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
grep "main: http" in the log: yes, I've got some users accessing the API frequently.
Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days.
Regards, Michael
Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson:
Is anyone here using the HTTP API at all?
It seems so tied to the v2 protocol, as to not be much use.
Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Thanks Mark, I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that. And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing… A modular solution seems to be best, easy to add custom implementations and to provide some standard modules. Regards, Michael Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson:
An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup:
require $config->val('db',’pw_module’,’auth_none.pl’);
…
If (&auth_password_check($passwordhash, $password))
...
Provide a ‘auto_none.pl’:
#!/usr/bin/perl
sub auth_password_check { my ($hash,$password) = @_;
return 0; }
Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc.
This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require.
That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl.
Is that a better solution?
Regards, Mark.
On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Michael,
Just before your commit, the server code was:
my $passwordhash = $row->{'pass'}; if (&drupal_password_check($passwordhash, $password))
…
sub drupal_password_check { my ($ph,$password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return ($encoded eq $ph); }
Your change was:
# User password encoding function: my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’);
…
my $passwordhash = $row->{'pass'}; my $encoded = eval $pw_encode; if ($encoded eq $passwordhash)
…
sub drupal_password { my ($password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return $encoded; }
You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters.
The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible.
For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way.
This won’t just affect drupal, but any system with a non-trivial password hashing function.
So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc).
Regards, Mark.
On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example).
That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong.
With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook.
Regards, Michael
Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson:
Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work.
It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication).
I standardised to use a new pw_check (overridable in the config) parameter, which defaults to:
drupal_password_check($passwordhash,$password)
and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config.
I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach.
Regards, Mark
On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Strange. I have zero using mine. Must be a EU thing?
I’ll keep it in mind and try not to break anything.
Regards, Mark.
On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
grep "main: http" in the log: yes, I've got some users accessing the API frequently.
Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days.
Regards, Michael
Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: > Is anyone here using the HTTP API at all? > > It seems so tied to the v2 protocol, as to not be much use. > > Regards, Mark. > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Ok. I will rework a modular approach. Should be able to get this done over the weekend. Yes, strict and warn would help. Mark P.S. Explains why nobody used the http api on my server :-)
On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter@expeedo.de> wrote:
Thanks Mark,
I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that.
And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing…
A modular solution seems to be best, easy to add custom implementations and to provide some standard modules.
Regards, Michael
Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson:
An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup:
require $config->val('db',’pw_module’,’auth_none.pl’);
…
If (&auth_password_check($passwordhash, $password)) ...
Provide a ‘auto_none.pl’:
#!/usr/bin/perl
sub auth_password_check { my ($hash,$password) = @_;
return 0; }
Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc.
This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require.
That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl.
Is that a better solution?
Regards, Mark.
On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Michael,
Just before your commit, the server code was:
my $passwordhash = $row->{'pass'}; if (&drupal_password_check($passwordhash, $password))
…
sub drupal_password_check { my ($ph,$password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return ($encoded eq $ph); }
Your change was:
# User password encoding function: my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’);
…
my $passwordhash = $row->{'pass'}; my $encoded = eval $pw_encode; if ($encoded eq $passwordhash)
…
sub drupal_password { my ($password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return $encoded; }
You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters.
The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible.
For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way.
This won’t just affect drupal, but any system with a non-trivial password hashing function.
So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc).
Regards, Mark.
On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de> wrote:
Mark,
I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example).
That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong.
With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook.
Regards, Michael
Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson:
Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work.
It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication).
I standardised to use a new pw_check (overridable in the config) parameter, which defaults to:
drupal_password_check($passwordhash,$password)
and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config.
I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach.
Regards, Mark
On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Strange. I have zero using mine. Must be a EU thing?
I’ll keep it in mind and try not to break anything.
Regards, Mark.
> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de> wrote: > > Mark, > > grep "main: http" in the log: yes, I've got some users accessing the API frequently. > > Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. > > Regards, > Michael > > > Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: >> Is anyone here using the HTTP API at all? >> >> It seems so tied to the v2 protocol, as to not be much use. >> >> Regards, Mark. >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev > > > -- > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 > > > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
This is turning into a bigger job than I imagined. "Give a mouse a cookie", and all that. The ovms_server.pl has gotten horrendous over the years. Almost 3,000 monolithic lines of code, 4 server listeners, three different types of server, push notifications, database synchronisation, etc. I tried turning on ‘use strict’ and it showed up a bunch of bugs and errors. So, I am refactoring it to a plugin architecture. That should make it more maintainable, and also provide a foundation for it to work better with the v3 MQTT. I suggest people hold off from making any changes to the server code in the next few days. Regards, Mark
On 21 Feb 2020, at 9:48 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Ok. I will rework a modular approach. Should be able to get this done over the weekend.
Yes, strict and warn would help.
Mark
P.S. Explains why nobody used the http api on my server :-)
On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter@expeedo.de> wrote:
Thanks Mark,
I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that.
And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing…
A modular solution seems to be best, easy to add custom implementations and to provide some standard modules.
Regards, Michael
Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson:
An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup:
require $config->val('db',’pw_module’,’auth_none.pl’);
…
If (&auth_password_check($passwordhash, $password)) ...
Provide a ‘auto_none.pl’:
#!/usr/bin/perl
sub auth_password_check { my ($hash,$password) = @_;
return 0; }
Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc.
This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require.
That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl.
Is that a better solution?
Regards, Mark.
On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Michael,
Just before your commit, the server code was:
my $passwordhash = $row->{'pass'}; if (&drupal_password_check($passwordhash, $password))
…
sub drupal_password_check { my ($ph,$password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return ($encoded eq $ph); }
Your change was:
# User password encoding function: my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’);
…
my $passwordhash = $row->{'pass'}; my $encoded = eval $pw_encode; if ($encoded eq $passwordhash)
…
sub drupal_password { my ($password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return $encoded; }
You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters.
The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible.
For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way.
This won’t just affect drupal, but any system with a non-trivial password hashing function.
So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc).
Regards, Mark.
On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example).
That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong.
With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook.
Regards, Michael
Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson:
Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work.
It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication).
I standardised to use a new pw_check (overridable in the config) parameter, which defaults to:
drupal_password_check($passwordhash,$password)
and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config.
I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach.
Regards, Mark
> On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: > > Strange. I have zero using mine. Must be a EU thing? > > I’ll keep it in mind and try not to break anything. > > Regards, Mark. > >> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >> >> Mark, >> >> grep "main: http" in the log: yes, I've got some users accessing the API frequently. >> >> Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. >> >> Regards, >> Michael >> >> >> Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: >>> Is anyone here using the HTTP API at all? >>> >>> It seems so tied to the v2 protocol, as to not be much use. >>> >>> Regards, Mark. >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >> >> >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> > > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
The new framework is running now on my development bench, and seems good. It was a major re-structuring, so I’ve upped the version to 3.0 (particularly as this is going to end up connecting to MQTT for HTTP API, database logging, push notifications, authentication, etc). I should finish my testing this weekend, then bring it up on api.openvehicles.com <http://api.openvehicles.com/>. Others should migrate to it with care. Regarding the authentication, I have almost finished implementing: An Auth Token facility (with authtokens stored against an associated owner ID). The HTTP API (using username+password authentication) can be used to issue new tokens, as well as enquire on the tokens already issued. The HTTP API can be used to enumerate registered vehicles, and otherwise maintain them. The drupal website extension will be extended to also allow viewing and maintenance of tokens. An authentication token is: Owner ID (so zero or more tokens belong to this owner) Token issued by the server on request (the token itself, unique, and the primary key) Usage identifier (identifying the car module, app id, etc) Usage description (a textual description of the usage) Permissions (a permission string identifying what the token can be used for) Created date+time Updated date+time Requests for a new token are passed the usage identifier and description as parameters, and authenticated by username+password. If a token already exists for that usage identifier, then the description is simply updated (along with updated date+time) and the token returned. If no token exists, a random one is created and returned (along with usage identifier, description, created and updated date+times). An extension to MQTT authentication API to allow authentication either by username+password, or username+authtoken. An extension to the HTTP API to allow authentication by username+token, in addition to the existing username+password. An extension to the V2 API to allow authentication by username+password or username+authtoken, in addition to the existing vehicled+serverpassword (crypto scheme 0x31). An extension to the V2 API to allow for optional automatic registration of new vehicles (a successful login with a non-existent vehicle ID will simply create it with a random server password). Removed the restriction that vehicle ID should be unique on the server (now just unique for each user). The preferred approach for a new app/car connection will then be: The user is interactively asked to select a server and provide his username+password A usage identifier and description is generated programatically The HTTP API is used to obtain a token (or recall the token if previously registered) The username, usage identifier, and token are stored persistently The username, token, and vehicle ID is used to login using v2 or v3 protocols So, three complementary authentication mechanisms are provided: The v2 vehicleid+serverpassword mechanism (with full permissions to access that particular vehicle) Username+Password mechanism (with full permission to maintain tokens, access all vehicles, and do everything) Username+Token mechanism (with permissions specified on the token) Comments welcome, and we can refine the above if necessary, but it is at least a starting point. I am trying to maintain as much flexibility as possible, but at the same time make things easier for the user. I’ve had four support requests so far this week for people messing up either the server they are using (app api.openvehicles, car dexters-web), the vehicle ID, or confusion between all the passwords. Regards, Mark.
On 23 Feb 2020, at 8:54 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
This is turning into a bigger job than I imagined. "Give a mouse a cookie", and all that.
The ovms_server.pl has gotten horrendous over the years. Almost 3,000 monolithic lines of code, 4 server listeners, three different types of server, push notifications, database synchronisation, etc. I tried turning on ‘use strict’ and it showed up a bunch of bugs and errors.
So, I am refactoring it to a plugin architecture. That should make it more maintainable, and also provide a foundation for it to work better with the v3 MQTT. I suggest people hold off from making any changes to the server code in the next few days.
Regards, Mark
On 21 Feb 2020, at 9:48 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Ok. I will rework a modular approach. Should be able to get this done over the weekend.
Yes, strict and warn would help.
Mark
P.S. Explains why nobody used the http api on my server :-)
On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Thanks Mark,
I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that.
And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing…
A modular solution seems to be best, easy to add custom implementations and to provide some standard modules.
Regards, Michael
Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson:
An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup:
require $config->val('db',’pw_module’,’auth_none.pl’);
…
If (&auth_password_check($passwordhash, $password)) ...
Provide a ‘auto_none.pl’:
#!/usr/bin/perl
sub auth_password_check { my ($hash,$password) = @_;
return 0; }
Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc.
This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require.
That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl.
Is that a better solution?
Regards, Mark.
On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Michael,
Just before your commit, the server code was:
my $passwordhash = $row->{'pass'}; if (&drupal_password_check($passwordhash, $password))
…
sub drupal_password_check { my ($ph,$password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return ($encoded eq $ph); }
Your change was:
# User password encoding function: my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’);
…
my $passwordhash = $row->{'pass'}; my $encoded = eval $pw_encode; if ($encoded eq $passwordhash)
…
sub drupal_password { my ($password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return $encoded; }
You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters.
The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible.
For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way.
This won’t just affect drupal, but any system with a non-trivial password hashing function.
So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc).
Regards, Mark.
On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Mark,
I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example).
That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong.
With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook.
Regards, Michael
Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson: > Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work. > > It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication). > > I standardised to use a new pw_check (overridable in the config) parameter, which defaults to: > > drupal_password_check($passwordhash,$password) > > and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config. > > I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach. > > Regards, Mark > >> On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: >> >> Strange. I have zero using mine. Must be a EU thing? >> >> I’ll keep it in mind and try not to break anything. >> >> Regards, Mark. >> >>> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >>> >>> Mark, >>> >>> grep "main: http" in the log: yes, I've got some users accessing the API frequently. >>> >>> Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. >>> >>> Regards, >>> Michael >>> >>> >>> Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: >>>> Is anyone here using the HTTP API at all? >>>> >>>> It seems so tied to the v2 protocol, as to not be much use. >>>> >>>> Regards, Mark. >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>> >>> >>> -- >>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>> >>> >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> > > > > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Wow, that was a lot more complex than I anticipated (in particular, making vehicleid’s unique per owner, not globally unique). 3,000 monolithic lines to 4,000 structured lines. Code has been committed to GitHub, and is running live on api.openvehicles.com <http://api.openvehicles.com/> now. Warning: There may (will) be bugs. If killed all the obvious ones, but running live is still showing up some edge cases. I don’t recommend anybody else runs this in production yet (particularly as rolling back would be non-trivial, as a result of the database changes). Major changes include: Complete restructuring, to be plugin based. This allows optional functionality to be installed (for example, if you don’t require drupal integration don’t enable that plugin), as well as new plugins to be developed. The list of plugins I am using on my production server are: VECE, DbDBI, AuthDrupal, ApiV2, Push, PushAPNS, PushGCM, PushMAIL, ApiHttp, ApiHttpCore, and ApiHttpMqapi. The system is designed to be installed and run from a github clone. Just clone the server repository, change directory to v3/server, configure and run appropriately. The .gitignore file allows changes to be made to the configuration without affecting the master github. Database format upgrades. See server/ovms_server_v2_to_v2.sql for the schema changes. If upgrading, you must deploy these database changes (just ’source’ the script in the mysql console). The upgrade may take a while (in particular, the last stage for historical data). The database no longer requires vehicle IDs to be globally unique. However, there are some caveats: The v2 crypto 0x30 scheme only sends the vehicle ID, not the owner’s username (so if there are two vehicles in the database with the same vehicle ID, we don’t know which one is the correct one). If you use that scheme (as everything does nowadays), vehicle ID still needs to be globally unique for you. For this reason, my Drupal vehicle plugin still checks and enforces unique IDs. The 0x30 login system won’t allow you to login to a vehicle who’s ID has more than one owner. Going forward, as we move away from 0x30 and ’server password’, this will become less of an issue. The new v3 crypto 0x31 scheme sends username as well as vehicle ID, so supports per-user vehicle IDs. The HTTP API sends username, so supports per-user vehicle IDs. Perhaps we should move to ‘VIN’ as vehicle ID? The new crypto scheme 0x31 is much simpler to use, and script. Preferable, IMHO - but requires SSL/TLS for protection. I’ve added support for API tokens, and love how they have turned out. I recommend this as the best approach going forward. Each token belongs to a particular owner, and has privilege rights associated with it. Tokens can be created and revoked by the owner. It is much easier to see which application (or car) has access to what. For example, the car module can ask for username+password, use that to obtain a token, store it locally and use that for authentication going forward (with no need to store the server password at all). Plugin authentication works well. Extend with new mechanisms as required. The ‘Authenticate’ plugin callback is merely passed username+password and returns the permissions granted. Push notifications are nicely modular and extendable. Already supports APNS, GCM, and MAIL. I’ve called it v3, because this architecture will be able to cooperate with MQTT (for things like HTTP API, authentication, historical data, push notifications, etc). Still todo: An option for automatic vehicle registration (easy). Improvements to the token allocation API (in particular to retrieve existing tokens for specific application usage such as car modules). A few miscellaneous functions not often used. Documentation. I’ve documented the API changes (both v2 protocol for 0x31, and HTTP API extensions for authentication options and api tokens), but still to document installation and configuration instructions for the server itself. Administrator access. Still deciding the best way to handle this. Perhaps ’administrative api tokens’. Permissions and Rights. The core is there, but need to extend to everything and document what rights are used for what. This is only relevant for api tokens anyway (as the other two authentication methods grant ‘*’ rights anyway). Final warning: Please don’t deploy this in productions systems yet. Changes are massive. Regards, Mark.
On 28 Feb 2020, at 3:47 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
The new framework is running now on my development bench, and seems good. It was a major re-structuring, so I’ve upped the version to 3.0 (particularly as this is going to end up connecting to MQTT for HTTP API, database logging, push notifications, authentication, etc). I should finish my testing this weekend, then bring it up on api.openvehicles.com <http://api.openvehicles.com/>. Others should migrate to it with care.
Regarding the authentication, I have almost finished implementing:
An Auth Token facility (with authtokens stored against an associated owner ID). The HTTP API (using username+password authentication) can be used to issue new tokens, as well as enquire on the tokens already issued. The HTTP API can be used to enumerate registered vehicles, and otherwise maintain them. The drupal website extension will be extended to also allow viewing and maintenance of tokens. An authentication token is: Owner ID (so zero or more tokens belong to this owner) Token issued by the server on request (the token itself, unique, and the primary key) Usage identifier (identifying the car module, app id, etc) Usage description (a textual description of the usage) Permissions (a permission string identifying what the token can be used for) Created date+time Updated date+time Requests for a new token are passed the usage identifier and description as parameters, and authenticated by username+password. If a token already exists for that usage identifier, then the description is simply updated (along with updated date+time) and the token returned. If no token exists, a random one is created and returned (along with usage identifier, description, created and updated date+times).
An extension to MQTT authentication API to allow authentication either by username+password, or username+authtoken.
An extension to the HTTP API to allow authentication by username+token, in addition to the existing username+password.
An extension to the V2 API to allow authentication by username+password or username+authtoken, in addition to the existing vehicled+serverpassword (crypto scheme 0x31).
An extension to the V2 API to allow for optional automatic registration of new vehicles (a successful login with a non-existent vehicle ID will simply create it with a random server password).
Removed the restriction that vehicle ID should be unique on the server (now just unique for each user).
The preferred approach for a new app/car connection will then be: The user is interactively asked to select a server and provide his username+password A usage identifier and description is generated programatically The HTTP API is used to obtain a token (or recall the token if previously registered) The username, usage identifier, and token are stored persistently The username, token, and vehicle ID is used to login using v2 or v3 protocols
So, three complementary authentication mechanisms are provided: The v2 vehicleid+serverpassword mechanism (with full permissions to access that particular vehicle) Username+Password mechanism (with full permission to maintain tokens, access all vehicles, and do everything) Username+Token mechanism (with permissions specified on the token)
Comments welcome, and we can refine the above if necessary, but it is at least a starting point. I am trying to maintain as much flexibility as possible, but at the same time make things easier for the user. I’ve had four support requests so far this week for people messing up either the server they are using (app api.openvehicles, car dexters-web), the vehicle ID, or confusion between all the passwords.
Regards, Mark.
On 23 Feb 2020, at 8:54 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
This is turning into a bigger job than I imagined. "Give a mouse a cookie", and all that.
The ovms_server.pl has gotten horrendous over the years. Almost 3,000 monolithic lines of code, 4 server listeners, three different types of server, push notifications, database synchronisation, etc. I tried turning on ‘use strict’ and it showed up a bunch of bugs and errors.
So, I am refactoring it to a plugin architecture. That should make it more maintainable, and also provide a foundation for it to work better with the v3 MQTT. I suggest people hold off from making any changes to the server code in the next few days.
Regards, Mark
On 21 Feb 2020, at 9:48 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Ok. I will rework a modular approach. Should be able to get this done over the weekend.
Yes, strict and warn would help.
Mark
P.S. Explains why nobody used the http api on my server :-)
On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Thanks Mark,
I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that.
And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing…
A modular solution seems to be best, easy to add custom implementations and to provide some standard modules.
Regards, Michael
Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson:
An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup:
require $config->val('db',’pw_module’,’auth_none.pl’);
…
If (&auth_password_check($passwordhash, $password)) ...
Provide a ‘auto_none.pl’:
#!/usr/bin/perl
sub auth_password_check { my ($hash,$password) = @_;
return 0; }
Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc.
This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require.
That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl.
Is that a better solution?
Regards, Mark.
On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Michael,
Just before your commit, the server code was:
my $passwordhash = $row->{'pass'}; if (&drupal_password_check($passwordhash, $password))
…
sub drupal_password_check { my ($ph,$password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return ($encoded eq $ph); }
Your change was:
# User password encoding function: my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’);
…
my $passwordhash = $row->{'pass'}; my $encoded = eval $pw_encode; if ($encoded eq $passwordhash)
…
sub drupal_password { my ($password) = @_;
my $iter_log2 = index($itoa64,substr($ph,3,1)); my $iter_count = 1 << $iter_log2;
my $phash = substr($ph,0,12); my $salt = substr($ph,4,8);
my $hash = sha512($salt.$password); do { $hash = sha512($hash.$password); $iter_count--; } while ($iter_count > 0);
my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55);
return $encoded; }
You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters.
The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible.
For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way.
This won’t just affect drupal, but any system with a non-trivial password hashing function.
So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc).
Regards, Mark.
> On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: > > Mark, > > I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example). > > That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong. > > With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook. > > Regards, > Michael > > > Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson: >> Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work. >> >> It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication). >> >> I standardised to use a new pw_check (overridable in the config) parameter, which defaults to: >> >> drupal_password_check($passwordhash,$password) >> >> and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config. >> >> I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach. >> >> Regards, Mark >> >>> On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: >>> >>> Strange. I have zero using mine. Must be a EU thing? >>> >>> I’ll keep it in mind and try not to break anything. >>> >>> Regards, Mark. >>> >>>> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >>>> >>>> Mark, >>>> >>>> grep "main: http" in the log: yes, I've got some users accessing the API frequently. >>>> >>>> Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. >>>> >>>> Regards, >>>> Michael >>>> >>>> >>>> Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: >>>>> Is anyone here using the HTTP API at all? >>>>> >>>>> It seems so tied to the v2 protocol, as to not be much use. >>>>> >>>>> Regards, Mark. >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>>> >>>> >>>> -- >>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>> >>>> >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>> >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >> >> >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> > > -- > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
I'm not sure if this is related or not but I just noticed that both of my cars were offline WRT the iphone app. The status page shows V2 connected but there were events shown that it was disconnecting/reconnecting. I've been running with V2 TLS enabled ever since it was possible. But I see that unchecking that box allows me to connect. I'm using 3.2.010-30 (Since Sep 29th). I appended some log monitor output (with V2 TLS enabled). It sort of looks like there might be some interaction between V2 and V3 (I've also had my modules connection to a mosquito server with TLS enabled). Craig W (269308) gsm-mux: Frame overflow (2048 bytes) I (270038) webserver: HTTP POST /api/execute I (270048) webcommand: HttpCommandStream[0x3fa400b8]: 1850360 bytes free, executing: server v2 start I (270058) ovms-server-v2: Status: Server has been started I (270068) ovms-server-v3: Tx event server.v2.waitnetwork I (270078) ovms-server-v2: OVMS Server v2 running I (270098) ovms-server-v2: Connection is api.openvehicles.com:6870 ???? I (270108) ovms-server-v2: Status: Connecting... I (270128) ovms-server-v3: Tx event server.v2.connecting W (270398) ovms-server-v2: Connection failed E (270398) ovms-server-v2: Status: Error: Connection failed I (270418) ovms-server-v3: Tx event server.v2.waitreconnect I (270628) ovms-server-v2: Status: Disconnected I (270648) ovms-server-v3: Tx event server.v2.waitreconnect I (290128) ovms-server-v2: Connection is api.openvehicles.com:6870 ???? I (290128) ovms-server-v2: Status: Connecting... I (290168) ovms-server-v3: Tx event server.v2.connecting W (290608) ovms-server-v2: Connection failed E (290608) ovms-server-v2: Status: Error: Connection failed E (290618) mongoose: mg_ssl_mbed_log 0x3fa577a4 mbedtls_ssl_flush_output() returned -1 (-0x0001) E (290618) mongoose: mg_ssl_mbed_log 0x3fa577a4 ssl_write_record() returned -1 (-0x0001) E (290618) mongoose: mg_ssl_mbed_log 0x3fa577a4 mbedtls_ssl_write_handshake_msg() returned -1 (-0x0001) E (290618) mongoose: mg_ssl_if_mbed_err 0x3fa577a4 SSL error: -1 W (290618) ovms-server-v2: Connection failed E (290618) ovms-server-v2: Status: Error: Connection failed I (290678) ovms-server-v3: Tx event server.v2.waitreconnect I (290748) ovms-server-v3: Tx event server.v2.waitreconnect I (290978) ovms-server-v2: Status: Disconnected I (290988) ovms-server-v3: Tx event server.v2.waitreconnect
Craig, The message 'ovms-server-v3: Tx event server.v2.connecting’ is confusing. It means that ovms-server-v3 is transmitting it’s received event ’server.v2.connecting’ to the MQTT server. The transmitter of the event was presumable server-v2. The big issue I see is ‘W (269308) gsm-mux: Frame overflow (2048 bytes)’. That would imply the GSM comms is messed up. Surprised anything works after that. Regards, Mark.
On 9 Mar 2020, at 1:48 PM, Craig Leres <leres@xse.com> wrote:
I'm not sure if this is related or not but I just noticed that both of my cars were offline WRT the iphone app. The status page shows V2 connected but there were events shown that it was disconnecting/reconnecting. I've been running with V2 TLS enabled ever since it was possible. But I see that unchecking that box allows me to connect.
I'm using 3.2.010-30 (Since Sep 29th). I appended some log monitor output (with V2 TLS enabled). It sort of looks like there might be some interaction between V2 and V3 (I've also had my modules connection to a mosquito server with TLS enabled).
Craig
W (269308) gsm-mux: Frame overflow (2048 bytes) I (270038) webserver: HTTP POST /api/execute I (270048) webcommand: HttpCommandStream[0x3fa400b8]: 1850360 bytes free, executing: server v2 start I (270058) ovms-server-v2: Status: Server has been started I (270068) ovms-server-v3: Tx event server.v2.waitnetwork I (270078) ovms-server-v2: OVMS Server v2 running I (270098) ovms-server-v2: Connection is api.openvehicles.com:6870 ???? I (270108) ovms-server-v2: Status: Connecting... I (270128) ovms-server-v3: Tx event server.v2.connecting W (270398) ovms-server-v2: Connection failed E (270398) ovms-server-v2: Status: Error: Connection failed I (270418) ovms-server-v3: Tx event server.v2.waitreconnect I (270628) ovms-server-v2: Status: Disconnected I (270648) ovms-server-v3: Tx event server.v2.waitreconnect I (290128) ovms-server-v2: Connection is api.openvehicles.com:6870 ???? I (290128) ovms-server-v2: Status: Connecting... I (290168) ovms-server-v3: Tx event server.v2.connecting W (290608) ovms-server-v2: Connection failed E (290608) ovms-server-v2: Status: Error: Connection failed E (290618) mongoose: mg_ssl_mbed_log 0x3fa577a4 mbedtls_ssl_flush_output() returned -1 (-0x0001) E (290618) mongoose: mg_ssl_mbed_log 0x3fa577a4 ssl_write_record() returned -1 (-0x0001) E (290618) mongoose: mg_ssl_mbed_log 0x3fa577a4 mbedtls_ssl_write_handshake_msg() returned -1 (-0x0001) E (290618) mongoose: mg_ssl_if_mbed_err 0x3fa577a4 SSL error: -1 W (290618) ovms-server-v2: Connection failed E (290618) ovms-server-v2: Status: Error: Connection failed I (290678) ovms-server-v3: Tx event server.v2.waitreconnect I (290748) ovms-server-v3: Tx event server.v2.waitreconnect I (290978) ovms-server-v2: Status: Disconnected I (290988) ovms-server-v3: Tx event server.v2.waitreconnect
On 2020-03-08 23:20, Mark Webb-Johnson wrote:
The message 'ovms-server-v3: Tx event server.v2.connecting’ is confusing. It means that ovms-server-v3 is transmitting it’s received event ’server.v2.connecting’ to the MQTT server. The transmitter of the event was presumable server-v2.
Are you saying it's the V3 server that is failing to connect when I enable TLS for the V2 server?
The big issue I see is ‘W (269308) gsm-mux: Frame overflow (2048 bytes)’. That would imply the GSM comms is messed up. Surprised anything works after that.
Would that caused by poor signal strength? When in the garage my cars usually see anywhere from -97 to -101 dBm. Anyway I don't understand why I can't see the cars on my iphone when I enable TLS for the V2 server. Craig
On 9 Mar 2020, at 2:39 PM, Craig Leres <leres@xse.com> wrote:
On 2020-03-08 23:20, Mark Webb-Johnson wrote:
The message 'ovms-server-v3: Tx event server.v2.connecting’ is confusing. It means that ovms-server-v3 is transmitting it’s received event ’server.v2.connecting’ to the MQTT server. The transmitter of the event was presumable server-v2.
Are you saying it's the V3 server that is failing to connect when I enable TLS for the V2 server?
No, just that the message shown appears confusing and may be why you suspect v3 server is interfering with v2 server.
The big issue I see is ‘W (269308) gsm-mux: Frame overflow (2048 bytes)’. That would imply the GSM comms is messed up. Surprised anything works after that.
Would that caused by poor signal strength? When in the garage my cars usually see anywhere from -97 to -101 dBm.
Anyway I don't understand why I can't see the cars on my iphone when I enable TLS for the V2 server.
I think most likely some buffer overflow in the modem communications. Which modem version is this with? Regards, Mark.
This seems stable to me now. Running well, with no outstanding issues (at least in the configuration I am running). The code is all committed to github, and documentation in the usual place: https://docs.openvehicles.com/en/latest/server/index.html <https://docs.openvehicles.com/en/latest/server/index.html> I’m now going to look at the OVMS module firmware to see if it is possible to support this scheme 0x31 in a neat way. The config ’server.v2’ directory is a little messy at the moment (password stored there, rather than in password/server.v2, so the whole config section needs to be read-only). Regards, Mark.
On 6 Mar 2020, at 12:54 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Wow, that was a lot more complex than I anticipated (in particular, making vehicleid’s unique per owner, not globally unique). 3,000 monolithic lines to 4,000 structured lines. Code has been committed to GitHub, and is running live on api.openvehicles.com <http://api.openvehicles.com/> now.
Warning: There may (will) be bugs. If killed all the obvious ones, but running live is still showing up some edge cases. I don’t recommend anybody else runs this in production yet (particularly as rolling back would be non-trivial, as a result of the database changes).
Major changes include:
Complete restructuring, to be plugin based. This allows optional functionality to be installed (for example, if you don’t require drupal integration don’t enable that plugin), as well as new plugins to be developed. The list of plugins I am using on my production server are: VECE, DbDBI, AuthDrupal, ApiV2, Push, PushAPNS, PushGCM, PushMAIL, ApiHttp, ApiHttpCore, and ApiHttpMqapi.
The system is designed to be installed and run from a github clone. Just clone the server repository, change directory to v3/server, configure and run appropriately. The .gitignore file allows changes to be made to the configuration without affecting the master github.
Database format upgrades. See server/ovms_server_v2_to_v2.sql for the schema changes. If upgrading, you must deploy these database changes (just ’source’ the script in the mysql console). The upgrade may take a while (in particular, the last stage for historical data).
The database no longer requires vehicle IDs to be globally unique. However, there are some caveats:
The v2 crypto 0x30 scheme only sends the vehicle ID, not the owner’s username (so if there are two vehicles in the database with the same vehicle ID, we don’t know which one is the correct one). If you use that scheme (as everything does nowadays), vehicle ID still needs to be globally unique for you. For this reason, my Drupal vehicle plugin still checks and enforces unique IDs. The 0x30 login system won’t allow you to login to a vehicle who’s ID has more than one owner. Going forward, as we move away from 0x30 and ’server password’, this will become less of an issue.
The new v3 crypto 0x31 scheme sends username as well as vehicle ID, so supports per-user vehicle IDs.
The HTTP API sends username, so supports per-user vehicle IDs.
Perhaps we should move to ‘VIN’ as vehicle ID?
The new crypto scheme 0x31 is much simpler to use, and script. Preferable, IMHO - but requires SSL/TLS for protection.
I’ve added support for API tokens, and love how they have turned out. I recommend this as the best approach going forward. Each token belongs to a particular owner, and has privilege rights associated with it. Tokens can be created and revoked by the owner. It is much easier to see which application (or car) has access to what. For example, the car module can ask for username+password, use that to obtain a token, store it locally and use that for authentication going forward (with no need to store the server password at all).
Plugin authentication works well. Extend with new mechanisms as required. The ‘Authenticate’ plugin callback is merely passed username+password and returns the permissions granted.
Push notifications are nicely modular and extendable. Already supports APNS, GCM, and MAIL.
I’ve called it v3, because this architecture will be able to cooperate with MQTT (for things like HTTP API, authentication, historical data, push notifications, etc).
Still todo:
An option for automatic vehicle registration (easy).
Improvements to the token allocation API (in particular to retrieve existing tokens for specific application usage such as car modules).
A few miscellaneous functions not often used.
Documentation. I’ve documented the API changes (both v2 protocol for 0x31, and HTTP API extensions for authentication options and api tokens), but still to document installation and configuration instructions for the server itself.
Administrator access. Still deciding the best way to handle this. Perhaps ’administrative api tokens’.
Permissions and Rights. The core is there, but need to extend to everything and document what rights are used for what. This is only relevant for api tokens anyway (as the other two authentication methods grant ‘*’ rights anyway).
Final warning: Please don’t deploy this in productions systems yet. Changes are massive.
Regards, Mark.
On 28 Feb 2020, at 3:47 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
The new framework is running now on my development bench, and seems good. It was a major re-structuring, so I’ve upped the version to 3.0 (particularly as this is going to end up connecting to MQTT for HTTP API, database logging, push notifications, authentication, etc). I should finish my testing this weekend, then bring it up on api.openvehicles.com <http://api.openvehicles.com/>. Others should migrate to it with care.
Regarding the authentication, I have almost finished implementing:
An Auth Token facility (with authtokens stored against an associated owner ID). The HTTP API (using username+password authentication) can be used to issue new tokens, as well as enquire on the tokens already issued. The HTTP API can be used to enumerate registered vehicles, and otherwise maintain them. The drupal website extension will be extended to also allow viewing and maintenance of tokens. An authentication token is: Owner ID (so zero or more tokens belong to this owner) Token issued by the server on request (the token itself, unique, and the primary key) Usage identifier (identifying the car module, app id, etc) Usage description (a textual description of the usage) Permissions (a permission string identifying what the token can be used for) Created date+time Updated date+time Requests for a new token are passed the usage identifier and description as parameters, and authenticated by username+password. If a token already exists for that usage identifier, then the description is simply updated (along with updated date+time) and the token returned. If no token exists, a random one is created and returned (along with usage identifier, description, created and updated date+times).
An extension to MQTT authentication API to allow authentication either by username+password, or username+authtoken.
An extension to the HTTP API to allow authentication by username+token, in addition to the existing username+password.
An extension to the V2 API to allow authentication by username+password or username+authtoken, in addition to the existing vehicled+serverpassword (crypto scheme 0x31).
An extension to the V2 API to allow for optional automatic registration of new vehicles (a successful login with a non-existent vehicle ID will simply create it with a random server password).
Removed the restriction that vehicle ID should be unique on the server (now just unique for each user).
The preferred approach for a new app/car connection will then be: The user is interactively asked to select a server and provide his username+password A usage identifier and description is generated programatically The HTTP API is used to obtain a token (or recall the token if previously registered) The username, usage identifier, and token are stored persistently The username, token, and vehicle ID is used to login using v2 or v3 protocols
So, three complementary authentication mechanisms are provided: The v2 vehicleid+serverpassword mechanism (with full permissions to access that particular vehicle) Username+Password mechanism (with full permission to maintain tokens, access all vehicles, and do everything) Username+Token mechanism (with permissions specified on the token)
Comments welcome, and we can refine the above if necessary, but it is at least a starting point. I am trying to maintain as much flexibility as possible, but at the same time make things easier for the user. I’ve had four support requests so far this week for people messing up either the server they are using (app api.openvehicles, car dexters-web), the vehicle ID, or confusion between all the passwords.
Regards, Mark.
On 23 Feb 2020, at 8:54 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
This is turning into a bigger job than I imagined. "Give a mouse a cookie", and all that.
The ovms_server.pl has gotten horrendous over the years. Almost 3,000 monolithic lines of code, 4 server listeners, three different types of server, push notifications, database synchronisation, etc. I tried turning on ‘use strict’ and it showed up a bunch of bugs and errors.
So, I am refactoring it to a plugin architecture. That should make it more maintainable, and also provide a foundation for it to work better with the v3 MQTT. I suggest people hold off from making any changes to the server code in the next few days.
Regards, Mark
On 21 Feb 2020, at 9:48 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Ok. I will rework a modular approach. Should be able to get this done over the weekend.
Yes, strict and warn would help.
Mark
P.S. Explains why nobody used the http api on my server :-)
On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Thanks Mark,
I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that.
And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing…
A modular solution seems to be best, easy to add custom implementations and to provide some standard modules.
Regards, Michael
Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson:
An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup:
require $config->val('db',’pw_module’,’auth_none.pl’);
…
If (&auth_password_check($passwordhash, $password)) ...
Provide a ‘auto_none.pl’:
#!/usr/bin/perl
sub auth_password_check { my ($hash,$password) = @_;
return 0; }
Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc.
This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require.
That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl.
Is that a better solution?
Regards, Mark.
> On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: > > Michael, > > Just before your commit, the server code was: > > my $passwordhash = $row->{'pass'}; > if (&drupal_password_check($passwordhash, $password)) > > … > > sub drupal_password_check > { > my ($ph,$password) = @_; > > my $iter_log2 = index($itoa64,substr($ph,3,1)); > my $iter_count = 1 << $iter_log2; > > my $phash = substr($ph,0,12); > my $salt = substr($ph,4,8); > > my $hash = sha512($salt.$password); > do > { > $hash = sha512($hash.$password); > $iter_count--; > } while ($iter_count > 0); > > my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); > > return ($encoded eq $ph); > } > > Your change was: > > # User password encoding function: > my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’); > > … > > my $passwordhash = $row->{'pass'}; > my $encoded = eval $pw_encode; > if ($encoded eq $passwordhash) > > … > > sub drupal_password > { > my ($password) = @_; > > my $iter_log2 = index($itoa64,substr($ph,3,1)); > my $iter_count = 1 << $iter_log2; > > my $phash = substr($ph,0,12); > my $salt = substr($ph,4,8); > > my $hash = sha512($salt.$password); > do > { > $hash = sha512($hash.$password); > $iter_count--; > } while ($iter_count > 0); > > my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); > > return $encoded; > } > > You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters. > > The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible. > > For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way. > > This won’t just affect drupal, but any system with a non-trivial password hashing function. > > So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc). > > Regards, Mark. > >> On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >> >> Mark, >> >> I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example). >> >> That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong. >> >> With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook. >> >> Regards, >> Michael >> >> >> Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson: >>> Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work. >>> >>> It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication). >>> >>> I standardised to use a new pw_check (overridable in the config) parameter, which defaults to: >>> >>> drupal_password_check($passwordhash,$password) >>> >>> and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config. >>> >>> I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach. >>> >>> Regards, Mark >>> >>>> On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: >>>> >>>> Strange. I have zero using mine. Must be a EU thing? >>>> >>>> I’ll keep it in mind and try not to break anything. >>>> >>>> Regards, Mark. >>>> >>>>> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >>>>> >>>>> Mark, >>>>> >>>>> grep "main: http" in the log: yes, I've got some users accessing the API frequently. >>>>> >>>>> Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> >>>>> Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: >>>>>> Is anyone here using the HTTP API at all? >>>>>> >>>>>> It seems so tied to the v2 protocol, as to not be much use. >>>>>> >>>>>> Regards, Mark. >>>>>> _______________________________________________ >>>>>> OvmsDev mailing list >>>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>>>> >>>>> >>>>> -- >>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>> >>>>> >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>>> >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>> >>> >>> >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >> >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Mark, awesome, I'll have a look and try running the new version at the weekend. Regarding config restructuring see OvmsConfig::upgrade(). Regards, Michael Am 11.03.20 um 08:00 schrieb Mark Webb-Johnson:
This seems stable to me now. Running well, with no outstanding issues (at least in the configuration I am running).
The code is all committed to github, and documentation in the usual place:
https://docs.openvehicles.com/en/latest/server/index.html
I’m now going to look at the OVMS module firmware to see if it is possible to support this scheme 0x31 in a neat way. The config ’server.v2’ directory is a little messy at the moment (password stored there, rather than in password/server.v2, so the whole config section needs to be read-only).
Regards, Mark.
On 6 Mar 2020, at 12:54 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Wow, that was a lot more complex than I anticipated (in particular, making vehicleid’s unique per owner, not globally unique). 3,000 monolithic lines to 4,000 structured lines. Code has been committed to GitHub, and is running live on api.openvehicles.com <http://api.openvehicles.com/> now.
Warning: There may (will) be bugs. If killed all the obvious ones, but running live is still showing up some edge cases. I don’t recommend anybody else runs this in production yet (particularly as rolling back would be non-trivial, as a result of the database changes).
Major changes include:
1. Complete restructuring, to be plugin based. This allows optional functionality to be installed (for example, if you don’t require drupal integration don’t enable that plugin), as well as new plugins to be developed. The list of plugins I am using on my production server are: VECE, DbDBI, AuthDrupal, ApiV2, Push, PushAPNS, PushGCM, PushMAIL, ApiHttp, ApiHttpCore, and ApiHttpMqapi.
2. The system is designed to be installed and run from a github clone. Just clone the server repository, change directory to v3/server, configure and run appropriately. The .gitignore file allows changes to be made to the configuration without affecting the master github.
3. Database format upgrades. See server/ovms_server_v2_to_v2.sql for the schema changes. If upgrading, you must deploy these database changes (just ’source’ the script in the mysql console). The upgrade may take a while (in particular, the last stage for historical data).
4. The database no longer requires vehicle IDs to be globally unique. However, there are some caveats:
* The v2 crypto 0x30 scheme only sends the vehicle ID, not the owner’s username (so if there are two vehicles in the database with the same vehicle ID, we don’t know which one is the correct one). If you use that scheme (as everything does nowadays), vehicle ID still needs to be globally unique for you. For this reason, my Drupal vehicle plugin still checks and enforces unique IDs. The 0x30 login system won’t allow you to login to a vehicle who’s ID has more than one owner. Going forward, as we move away from 0x30 and ’server password’, this will become less of an issue.
* The new v3 crypto 0x31 scheme sends username as well as vehicle ID, so supports per-user vehicle IDs.
* The HTTP API sends username, so supports per-user vehicle IDs.
* Perhaps we should move to ‘VIN’ as vehicle ID?
5. The new crypto scheme 0x31 is much simpler to use, and script. Preferable, IMHO - but requires SSL/TLS for protection.
6. I’ve added support for API tokens, and love how they have turned out. I recommend this as the best approach going forward. Each token belongs to a particular owner, and has privilege rights associated with it. Tokens can be created and revoked by the owner. It is much easier to see which application (or car) has access to what. For example, the car module can ask for username+password, use that to obtain a token, store it locally and use that for authentication going forward (with no need to store the server password at all).
7. Plugin authentication works well. Extend with new mechanisms as required. The ‘Authenticate’ plugin callback is merely passed username+password and returns the permissions granted.
8. Push notifications are nicely modular and extendable. Already supports APNS, GCM, and MAIL.
9. I’ve called it v3, because this architecture will be able to cooperate with MQTT (for things like HTTP API, authentication, historical data, push notifications, etc).
Still todo:
* An option for automatic vehicle registration (easy).
* Improvements to the token allocation API (in particular to retrieve existing tokens for specific application usage such as car modules).
* A few miscellaneous functions not often used.
* Documentation. I’ve documented the API changes (both v2 protocol for 0x31, and HTTP API extensions for authentication options and api tokens), but still to document installation and configuration instructions for the server itself.
* Administrator access. Still deciding the best way to handle this. Perhaps ’administrative api tokens’.
* Permissions and Rights. The core is there, but need to extend to everything and document what rights are used for what. This is only relevant for api tokens anyway (as the other two authentication methods grant ‘*’ rights anyway).
Final warning: Please don’t deploy this in productions systems yet. Changes are massive.
Regards, Mark.
On 28 Feb 2020, at 3:47 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
The new framework is running now on my development bench, and seems good. It was a major re-structuring, so I’ve upped the version to 3.0 (particularly as this is going to end up connecting to MQTT for HTTP API, database logging, push notifications, authentication, etc). I should finish my testing this weekend, then bring it up on api.openvehicles.com <http://api.openvehicles.com/>. Others should migrate to it with care.
Regarding the authentication, I have almost finished implementing:
* An Auth Token facility (with authtokens stored against an associated owner ID). o The HTTP API (using username+password authentication) can be used to issue new tokens, as well as enquire on the tokens already issued. o The HTTP API can be used to enumerate registered vehicles, and otherwise maintain them. o The drupal website extension will be extended to also allow viewing and maintenance of tokens. o An authentication token is: + Owner ID (so zero or more tokens belong to this owner) + Token issued by the server on request (the token itself, unique, and the primary key) + Usage identifier (identifying the car module, app id, etc) + Usage description (a textual description of the usage) + Permissions (a permission string identifying what the token can be used for) + Created date+time + Updated date+time o Requests for a new token are passed the usage identifier and description as parameters, and authenticated by username+password. + If a token already exists for that usage identifier, then the description is simply updated (along with updated date+time) and the token returned. + If no token exists, a random one is created and returned (along with usage identifier, description, created and updated date+times).
* An extension to MQTT authentication API to allow authentication either by username+password, or username+authtoken.
* An extension to the HTTP API to allow authentication by username+token, in addition to the existing username+password.
* An extension to the V2 API to allow authentication by username+password or username+authtoken, in addition to the existing vehicled+serverpassword (crypto scheme 0x31).
* An extension to the V2 API to allow for optional automatic registration of new vehicles (a successful login with a non-existent vehicle ID will simply create it with a random server password).
* Removed the restriction that vehicle ID should be unique on the server (now just unique for each user).
* The preferred approach for a new app/car connection will then be: o The user is interactively asked to select a server and provide his username+password o A usage identifier and description is generated programatically o The HTTP API is used to obtain a token (or recall the token if previously registered) o The username, usage identifier, and token are stored persistently o The username, token, and vehicle ID is used to login using v2 or v3 protocols
* So, three complementary authentication mechanisms are provided: o The v2 vehicleid+serverpassword mechanism (with full permissions to access that particular vehicle) o Username+Password mechanism (with full permission to maintain tokens, access all vehicles, and do everything) o Username+Token mechanism (with permissions specified on the token)
Comments welcome, and we can refine the above if necessary, but it is at least a starting point. I am trying to maintain as much flexibility as possible, but at the same time make things easier for the user. I’ve had four support requests so far this week for people messing up either the server they are using (app api.openvehicles, car dexters-web), the vehicle ID, or confusion between all the passwords.
Regards, Mark.
On 23 Feb 2020, at 8:54 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
This is turning into a bigger job than I imagined. "Give a mouse a cookie", and all that.
The ovms_server.pl has gotten horrendous over the years. Almost 3,000 monolithic lines of code, 4 server listeners, three different types of server, push notifications, database synchronisation, etc. I tried turning on ‘use strict’ and it showed up a bunch of bugs and errors.
So, I am refactoring it to a plugin architecture. That should make it more maintainable, and also provide a foundation for it to work better with the v3 MQTT. I suggest people hold off from making any changes to the server code in the next few days.
Regards, Mark
On 21 Feb 2020, at 9:48 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Ok. I will rework a modular approach. Should be able to get this done over the weekend.
Yes, strict and warn would help.
Mark
P.S. Explains why nobody used the http api on my server :-)
On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:
Thanks Mark,
I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that.
And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing…
A modular solution seems to be best, easy to add custom implementations and to provide some standard modules.
Regards, Michael
Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson: > > An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup: > > > require $config->val('db',’pw_module’,’auth_none.pl’); > > > … > > > If (&auth_password_check($passwordhash, $password)) > > ... > > > Provide a ‘auto_none.pl’: > > #!/usr/bin/perl > > sub auth_password_check > { > my ($hash,$password) = @_; > > return 0; > } > > > Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc. > > This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the > old-style require. > > That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if > we have a separate module that drupal-dependant code could be removed from ovms_server.pl. > > Is that a better solution? > > Regards, Mark. > >> On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: >> >> Michael, >> >> Just before your commit, the server code was: >> >> my $passwordhash = $row->{'pass'}; >> if (&drupal_password_check($passwordhash, $password)) >> >> … >> >> sub drupal_password_check >> { >> my ($ph,$password) = @_; >> >> my $iter_log2 = index($itoa64,substr($ph,3,1)); >> my $iter_count = 1 << $iter_log2; >> >> my $phash = substr($ph,0,12); >> my $salt = substr($ph,4,8); >> >> my $hash = sha512($salt.$password); >> do >> { >> $hash = sha512($hash.$password); >> $iter_count--; >> } while ($iter_count > 0); >> >> my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); >> >> return ($encoded eq $ph); >> } >> >> >> Your change was: >> >> # User password encoding function: >> my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’); >> >> … >> >> my $passwordhash = $row->{'pass'}; >> my $encoded = eval $pw_encode; >> if ($encoded eq $passwordhash) >> >> … >> >> sub drupal_password >> { >> my ($password) = @_; >> >> my $iter_log2 = index($itoa64,substr($ph,3,1)); >> my $iter_count = 1 << $iter_log2; >> >> my $phash = substr($ph,0,12); >> my $salt = substr($ph,4,8); >> >> my $hash = sha512($salt.$password); >> do >> { >> $hash = sha512($hash.$password); >> $iter_count--; >> } while ($iter_count > 0); >> >> my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); >> >> return $encoded; >> } >> >> >> You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) >> to extract meta data to set the encoding parameters. >> >> The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the >> password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the >> hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables >> to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables >> and make brute force approaches computationally unfeasible. >> >> For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous >> encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new >> password in the same way. >> >> This won’t just affect drupal, but any system with a non-trivial password hashing function. >> >> So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make >> it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will >> allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup >> (ldap, etc). >> >> Regards, Mark. >> >>> On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >>> >>> Mark, >>> >>> I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the >>> "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example). >>> >>> That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function >>> for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you >>> please elaborate? I'd like to understand what was going wrong. >>> >>> With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" >>> hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that >>> simply reuses the existing "pw_encode" hook. >>> >>> Regards, >>> Michael >>> >>> >>> Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson: >>>> Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication >>>> didn’t work. >>>> >>>> It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also >>>> weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and >>>> the other by MQ authentication). >>>> >>>> I standardised to use a new pw_check (overridable in the config) parameter, which defaults to: >>>> >>>> drupal_password_check($passwordhash,$password) >>>> >>>> >>>> and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both >>>> authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the >>>> pw_check parameter in the config. >>>> >>>> I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach. >>>> >>>> Regards, Mark >>>> >>>>> On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: >>>>> >>>>> Strange. I have zero using mine. Must be a EU thing? >>>>> >>>>> I’ll keep it in mind and try not to break anything. >>>>> >>>>> Regards, Mark. >>>>> >>>>>> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >>>>>> >>>>>> Mark, >>>>>> >>>>>> grep "main: http" in the log: yes, I've got some users accessing the API frequently. >>>>>> >>>>>> Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> >>>>>> Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: >>>>>>> Is anyone here using the HTTP API at all? >>>>>>> >>>>>>> It seems so tied to the v2 protocol, as to not be much use. >>>>>>> >>>>>>> Regards, Mark. >>>>>>> _______________________________________________ >>>>>>> OvmsDev mailing list >>>>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>> >>>>>> >>>>>> -- >>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> OvmsDev mailing list >>>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>> >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>> >>>> >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>> >>> -- >>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >> > > > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Regarding config restructuring see OvmsConfig::upgrade().
Looks perfect. I hadn’t seen that before. Concerning how this project is now so big that I can’t keep track of it all 🤔 Regards, Mark.
On 12 Mar 2020, at 7:12 PM, Michael Balzer <dexter@expeedo.de> wrote:
Mark,
awesome, I'll have a look and try running the new version at the weekend.
Regarding config restructuring see OvmsConfig::upgrade().
Regards, Michael
Am 11.03.20 um 08:00 schrieb Mark Webb-Johnson:
This seems stable to me now. Running well, with no outstanding issues (at least in the configuration I am running).
The code is all committed to github, and documentation in the usual place:
https://docs.openvehicles.com/en/latest/server/index.html <https://docs.openvehicles.com/en/latest/server/index.html>
I’m now going to look at the OVMS module firmware to see if it is possible to support this scheme 0x31 in a neat way. The config ’server.v2’ directory is a little messy at the moment (password stored there, rather than in password/server.v2, so the whole config section needs to be read-only).
Regards, Mark.
On 6 Mar 2020, at 12:54 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Wow, that was a lot more complex than I anticipated (in particular, making vehicleid’s unique per owner, not globally unique). 3,000 monolithic lines to 4,000 structured lines. Code has been committed to GitHub, and is running live on api.openvehicles.com <http://api.openvehicles.com/> now.
Warning: There may (will) be bugs. If killed all the obvious ones, but running live is still showing up some edge cases. I don’t recommend anybody else runs this in production yet (particularly as rolling back would be non-trivial, as a result of the database changes).
Major changes include:
Complete restructuring, to be plugin based. This allows optional functionality to be installed (for example, if you don’t require drupal integration don’t enable that plugin), as well as new plugins to be developed. The list of plugins I am using on my production server are: VECE, DbDBI, AuthDrupal, ApiV2, Push, PushAPNS, PushGCM, PushMAIL, ApiHttp, ApiHttpCore, and ApiHttpMqapi.
The system is designed to be installed and run from a github clone. Just clone the server repository, change directory to v3/server, configure and run appropriately. The .gitignore file allows changes to be made to the configuration without affecting the master github.
Database format upgrades. See server/ovms_server_v2_to_v2.sql for the schema changes. If upgrading, you must deploy these database changes (just ’source’ the script in the mysql console). The upgrade may take a while (in particular, the last stage for historical data).
The database no longer requires vehicle IDs to be globally unique. However, there are some caveats:
The v2 crypto 0x30 scheme only sends the vehicle ID, not the owner’s username (so if there are two vehicles in the database with the same vehicle ID, we don’t know which one is the correct one). If you use that scheme (as everything does nowadays), vehicle ID still needs to be globally unique for you. For this reason, my Drupal vehicle plugin still checks and enforces unique IDs. The 0x30 login system won’t allow you to login to a vehicle who’s ID has more than one owner. Going forward, as we move away from 0x30 and ’server password’, this will become less of an issue.
The new v3 crypto 0x31 scheme sends username as well as vehicle ID, so supports per-user vehicle IDs.
The HTTP API sends username, so supports per-user vehicle IDs.
Perhaps we should move to ‘VIN’ as vehicle ID?
The new crypto scheme 0x31 is much simpler to use, and script. Preferable, IMHO - but requires SSL/TLS for protection.
I’ve added support for API tokens, and love how they have turned out. I recommend this as the best approach going forward. Each token belongs to a particular owner, and has privilege rights associated with it. Tokens can be created and revoked by the owner. It is much easier to see which application (or car) has access to what. For example, the car module can ask for username+password, use that to obtain a token, store it locally and use that for authentication going forward (with no need to store the server password at all).
Plugin authentication works well. Extend with new mechanisms as required. The ‘Authenticate’ plugin callback is merely passed username+password and returns the permissions granted.
Push notifications are nicely modular and extendable. Already supports APNS, GCM, and MAIL.
I’ve called it v3, because this architecture will be able to cooperate with MQTT (for things like HTTP API, authentication, historical data, push notifications, etc).
Still todo:
An option for automatic vehicle registration (easy).
Improvements to the token allocation API (in particular to retrieve existing tokens for specific application usage such as car modules).
A few miscellaneous functions not often used.
Documentation. I’ve documented the API changes (both v2 protocol for 0x31, and HTTP API extensions for authentication options and api tokens), but still to document installation and configuration instructions for the server itself.
Administrator access. Still deciding the best way to handle this. Perhaps ’administrative api tokens’.
Permissions and Rights. The core is there, but need to extend to everything and document what rights are used for what. This is only relevant for api tokens anyway (as the other two authentication methods grant ‘*’ rights anyway).
Final warning: Please don’t deploy this in productions systems yet. Changes are massive.
Regards, Mark.
On 28 Feb 2020, at 3:47 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
The new framework is running now on my development bench, and seems good. It was a major re-structuring, so I’ve upped the version to 3.0 (particularly as this is going to end up connecting to MQTT for HTTP API, database logging, push notifications, authentication, etc). I should finish my testing this weekend, then bring it up on api.openvehicles.com <http://api.openvehicles.com/>. Others should migrate to it with care.
Regarding the authentication, I have almost finished implementing:
An Auth Token facility (with authtokens stored against an associated owner ID). The HTTP API (using username+password authentication) can be used to issue new tokens, as well as enquire on the tokens already issued. The HTTP API can be used to enumerate registered vehicles, and otherwise maintain them. The drupal website extension will be extended to also allow viewing and maintenance of tokens. An authentication token is: Owner ID (so zero or more tokens belong to this owner) Token issued by the server on request (the token itself, unique, and the primary key) Usage identifier (identifying the car module, app id, etc) Usage description (a textual description of the usage) Permissions (a permission string identifying what the token can be used for) Created date+time Updated date+time Requests for a new token are passed the usage identifier and description as parameters, and authenticated by username+password. If a token already exists for that usage identifier, then the description is simply updated (along with updated date+time) and the token returned. If no token exists, a random one is created and returned (along with usage identifier, description, created and updated date+times).
An extension to MQTT authentication API to allow authentication either by username+password, or username+authtoken.
An extension to the HTTP API to allow authentication by username+token, in addition to the existing username+password.
An extension to the V2 API to allow authentication by username+password or username+authtoken, in addition to the existing vehicled+serverpassword (crypto scheme 0x31).
An extension to the V2 API to allow for optional automatic registration of new vehicles (a successful login with a non-existent vehicle ID will simply create it with a random server password).
Removed the restriction that vehicle ID should be unique on the server (now just unique for each user).
The preferred approach for a new app/car connection will then be: The user is interactively asked to select a server and provide his username+password A usage identifier and description is generated programatically The HTTP API is used to obtain a token (or recall the token if previously registered) The username, usage identifier, and token are stored persistently The username, token, and vehicle ID is used to login using v2 or v3 protocols
So, three complementary authentication mechanisms are provided: The v2 vehicleid+serverpassword mechanism (with full permissions to access that particular vehicle) Username+Password mechanism (with full permission to maintain tokens, access all vehicles, and do everything) Username+Token mechanism (with permissions specified on the token)
Comments welcome, and we can refine the above if necessary, but it is at least a starting point. I am trying to maintain as much flexibility as possible, but at the same time make things easier for the user. I’ve had four support requests so far this week for people messing up either the server they are using (app api.openvehicles, car dexters-web), the vehicle ID, or confusion between all the passwords.
Regards, Mark.
On 23 Feb 2020, at 8:54 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
This is turning into a bigger job than I imagined. "Give a mouse a cookie", and all that.
The ovms_server.pl has gotten horrendous over the years. Almost 3,000 monolithic lines of code, 4 server listeners, three different types of server, push notifications, database synchronisation, etc. I tried turning on ‘use strict’ and it showed up a bunch of bugs and errors.
So, I am refactoring it to a plugin architecture. That should make it more maintainable, and also provide a foundation for it to work better with the v3 MQTT. I suggest people hold off from making any changes to the server code in the next few days.
Regards, Mark
On 21 Feb 2020, at 9:48 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
Ok. I will rework a modular approach. Should be able to get this done over the weekend.
Yes, strict and warn would help.
Mark
P.S. Explains why nobody used the http api on my server :-)
> On 21 Feb 2020, at 9:35 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: > > Thanks Mark, > > I must have been blind… but perl also never fails to amaze me in terms of "compiles fine, but won't run" -- $ph isn't defined anywhere else. Maybe "strict" mode would have told me about that. > > And I didn't think about meta data in the hash. You're right, we need to pass both values to the function. And I need to rework my password hashing… > > A modular solution seems to be best, easy to add custom implementations and to provide some standard modules. > > Regards, > Michael > > > Am 21.02.20 um 12:15 schrieb Mark Webb-Johnson: >> >> An alternative would be to implement a server authentication module, and to ‘require’ that into the system at startup: >> >> require $config->val('db',’pw_module’,’auth_none.pl’); >> >> … >> >> If (&auth_password_check($passwordhash, $password)) >> ... >> >> Provide a ‘auto_none.pl’: >> >> #!/usr/bin/perl >> >> sub auth_password_check >> { >> my ($hash,$password) = @_; >> >> return 0; >> } >> >> Then a ‘auth_drupal7.pl’, ‘auth_sha1.pl’, etc. >> >> This could also be done in a perl modular fashion by having the module provided as an object (using ‘use …’). Probably cleaner than the old-style require. >> >> That is much more extendable and standardised. In particular, there is also code that syncs Drupal users to ovms_owners (svr_tim) and if we have a separate module that drupal-dependant code could be removed from ovms_server.pl. >> >> Is that a better solution? >> >> Regards, Mark. >> >>> On 21 Feb 2020, at 11:37 AM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: >>> >>> Michael, >>> >>> Just before your commit, the server code was: >>> >>> my $passwordhash = $row->{'pass'}; >>> if (&drupal_password_check($passwordhash, $password)) >>> >>> … >>> >>> sub drupal_password_check >>> { >>> my ($ph,$password) = @_; >>> >>> my $iter_log2 = index($itoa64,substr($ph,3,1)); >>> my $iter_count = 1 << $iter_log2; >>> >>> my $phash = substr($ph,0,12); >>> my $salt = substr($ph,4,8); >>> >>> my $hash = sha512($salt.$password); >>> do >>> { >>> $hash = sha512($hash.$password); >>> $iter_count--; >>> } while ($iter_count > 0); >>> >>> my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); >>> >>> return ($encoded eq $ph); >>> } >>> >>> Your change was: >>> >>> # User password encoding function: >>> my $pw_encode = $config->val('db','pw_encode','drupal_password($password)’); >>> >>> … >>> >>> my $passwordhash = $row->{'pass'}; >>> my $encoded = eval $pw_encode; >>> if ($encoded eq $passwordhash) >>> >>> … >>> >>> sub drupal_password >>> { >>> my ($password) = @_; >>> >>> my $iter_log2 = index($itoa64,substr($ph,3,1)); >>> my $iter_count = 1 << $iter_log2; >>> >>> my $phash = substr($ph,0,12); >>> my $salt = substr($ph,4,8); >>> >>> my $hash = sha512($salt.$password); >>> do >>> { >>> $hash = sha512($hash.$password); >>> $iter_count--; >>> } while ($iter_count > 0); >>> >>> my $encoded = substr($phash . &drupal_password_base64_encode($hash,length($hash)),0,55); >>> >>> return $encoded; >>> } >>> >>> You changed the parameters from ($ph,$password) to just ($password), but the drupal_password function still needs to use $ph (the hash) to extract meta data to set the encoding parameters. >>> >>> The problem is that Drupal (and others) has a strong hashing function with multiple iterations. The meta data for that is stored in the password hash itself. The unix crypt library does something similar (with the encoding method and salt stored as meta data in the hash). Just storing passwords as straight hashes (md5, sha1, etc) is fundamentally not secure, as it is trivial to use rainbow tables to break the hashes - so most modern systems use iterations, salts, or other techniques to limit the effectiveness of rainbow tables and make brute force approaches computationally unfeasible. >>> >>> For many systems, we can only encode a password in the same way as a previous encoding if we know the meta data of the previous encoding (and that is stored in the hash). Hence we need the hash as a parameter, to extract the meta data to be able to encode the new password in the same way. >>> >>> This won’t just affect drupal, but any system with a non-trivial password hashing function. >>> >>> So, pw_encode() needs both the old hash as well as the plaintext password to encode. At which point, I think it becomes easier to make it simply pw_check() returning a boolean. It also seems easier to me to do that as a plugin function (pw_check vs pw_encode) as it will allow other non-trivial hashing comparisons if required. For example, say you needed to check the password against an external lookup (ldap, etc). >>> >>> Regards, Mark. >>> >>>> On 21 Feb 2020, at 1:12 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >>>> >>>> Mark, >>>> >>>> I did 1b73a7f8 to split the "create & compare password" function into separate "create" & "compare" steps, and introduced the "pw_encode" config hook to be able to supply just a custom "create" operation. That simplifies the config (see example). >>>> >>>> That change has been working since 2016 on my server. I see you reintroduced the "create & compare" function as a separate function for the MQTT auth, but don't see why that was needed. I also don't see why the separated function was broken on your server. Can you please elaborate? I'd like to understand what was going wrong. >>>> >>>> With reverting to the "create & compare", this breaks the configuration of servers not using Drupal. Essentially, the new "pw_check" hook does just the previous "pw_encode" and adds the comparison to that, so I'd rather opt for adding a default function here that simply reuses the existing "pw_encode" hook. >>>> >>>> Regards, >>>> Michael >>>> >>>> >>>> Am 20.02.20 um 04:09 schrieb Mark Webb-Johnson: >>>>> Even stranger. This conversation obviously triggered someone to try it and then raise a support ticket that HTTP API authentication didn’t work. >>>>> >>>>> It seems a change was made back in 2016-02-01 23:59:22 (1b73a7f8) that broke the pw_encode function (drupal_password). It was also weird because we had drupal_password and drupal_password_check functions, doing pretty much the same thing (one used by HTTP API and the other by MQ authentication). >>>>> >>>>> I standardised to use a new pw_check (overridable in the config) parameter, which defaults to: >>>>> >>>>> drupal_password_check($passwordhash,$password) >>>>> >>>>> and stopped using the pw_encode config value. I also changed the MQ authentication stuff to use the same pw_check parameter (so both authentication uses are now able to be changed in the same config). If using something other than drupal, just need to change the pw_check parameter in the config. >>>>> >>>>> I realise that this may break other users of the server, but it doesn’t seem a difficult fix to make, and is a much better approach. >>>>> >>>>> Regards, Mark >>>>> >>>>>> On 19 Feb 2020, at 1:53 PM, Mark Webb-Johnson <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote: >>>>>> >>>>>> Strange. I have zero using mine. Must be a EU thing? >>>>>> >>>>>> I’ll keep it in mind and try not to break anything. >>>>>> >>>>>> Regards, Mark. >>>>>> >>>>>>> On 18 Feb 2020, at 8:41 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote: >>>>>>> >>>>>>> Mark, >>>>>>> >>>>>>> grep "main: http" in the log: yes, I've got some users accessing the API frequently. >>>>>>> >>>>>>> Usage is mostly /api/charge followed by /api/status & /api/historical, but almost all calls have been used during the last days. >>>>>>> >>>>>>> Regards, >>>>>>> Michael >>>>>>> >>>>>>> >>>>>>> Am 18.02.20 um 04:28 schrieb Mark Webb-Johnson: >>>>>>>> Is anyone here using the HTTP API at all? >>>>>>>> >>>>>>>> It seems so tied to the v2 protocol, as to not be much use. >>>>>>>> >>>>>>>> Regards, Mark. >>>>>>>> _______________________________________________ >>>>>>>> OvmsDev mailing list >>>>>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OvmsDev mailing list >>>>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>>>>> >>>>>> _______________________________________________ >>>>>> OvmsDev mailing list >>>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>>> >>>> -- >>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >>> >> >> >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> > > -- > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
participants (3)
-
Craig Leres -
Mark Webb-Johnson -
Michael Balzer