<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr"><br></div><div dir="ltr">Can you send me the patch to enable debugging, and I will try with 3.3 branch. I agree that rolling back 3.2 to mbedtls seems sensible for the short term.</div><div dir="ltr"><br></div><div dir="ltr">Regards, Mark</div><div dir="ltr"><br></div><div dir="ltr">P.S. I managed to fix this on another project running an old OpenSSL by removing the X3 trusted ca (it had both x1 and x3 trusted). Behavior is highly dependent on the library treatment of multiple verification paths and trusted ca vs cert chain conflicts.</div><div dir="ltr"><br><div dir="ltr"></div><blockquote type="cite">On 29 Sep 2021, at 8:11 PM, Michael Balzer <dexter@expeedo.de> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
I managed to get wolfSSL debug logs, but don't know how to interpret
them. See files attached.<br>
<br>
The final error seems to be…<br>
<br>
<font face="monospace">I (597167) wolfssl: wolfSSL Leaving
DoCertificate, return -188<br>
I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return
-188<br>
I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188<br>
E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818
<a class="moz-txt-link-freetext" href="file:/home/balzer/esp/Open-Vehic">file:/home/balzer/esp/Open-Vehic</a><br>
E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507
<a class="moz-txt-link-freetext" href="file:/home/balzer/esp/Open-Vehic">file:/home/balzer/esp/Open-Vehic</a></font><br>
<br>
which is<br>
<font face="monospace"> ASN_NO_SIGNER_E = -188, /* ASN no
signer to confirm failure */</font><br>
<br>
I've had a look at the code, I don't see a chance to find the bug
within the remaining time.<br>
<br>
We're 24 hours from expiration. I'll try reverting the
Mongoose-WolfSSL commits now. Steve, if you already prepared
something in that direction, tell me.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 29.09.21 um 09:56 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite" cite="mid:FA798F70-8B49-4CAD-9833-1128FEF31871@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class=""><br class="">
</div>
The cross-signing bit seems to be on the X1 cert provided in the
certificate chain by the server. The one provided there is signed
by X3. The problem comes because a different X1 certificate may be
provided as a trusted CA (and that X1 is self-signed with a long
expiration date - 2035). So the verification process has two
possible paths to verify the cert, and the result depends on which
one it chooses (and how it handles a trusted CA cert supplementing
/ replacing one provided by the server).
<div class=""><br class="">
</div>
<div class="">Overall, a mess. I’m fighting this same issue now
with a bunch of systems in my day job, and likely a large part
of the Internet is going to start to fail tomorrow. Particularly
a lot of small embedded devices.</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 29 Sep 2021, at 2:51 PM, Michael Balzer
<<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="content-isolator__container">
<div class="protected-part">
<div class="protected-title">Signed PGP part</div>
<div class="protected-content">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div class=""> SNI was pretty much intact already, I
took care of a remaining Mongoose SNI bug in
commit 9e4f31249856a67317f99920e47efa34f6734c73 /
mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f
(April 2020).<br class="">
<br class="">
I understand the WolfSSL chain management uses a
supplied root cert over one delivered by the
server:<br class="">
<a class="moz-txt-link-freetext" href="https://www.wolfssl.com/docs/wolfssl-manual/ch7/" moz-do-not-send="true">https://www.wolfssl.com/docs/wolfssl-manual/ch7/</a>
→ section 7.3<br class="">
<br class="">
<blockquote type="cite" class="">
<div class="">Our problem could be either the
wolfssl not recognising the X1 cert, or not
supporting the cross-signing arrangement
(where there are two paths to verify - the
expired X3 and the provided X1). I suspect the
latter. In an ideal world, having both X1 and
X3 trusted shouldn’t be a problem.</div>
</blockquote>
<br class="">
Is cross signing supported at all by WolfSSL? The
only place I've found it mentioned on wolfssl is
this page: <a class="moz-txt-link-freetext" href="https://www.wolfssl.com/certificate-chain-chain-trust/" moz-do-not-send="true">https://www.wolfssl.com/certificate-chain-chain-trust/</a><br class="">
…where they say "(We’ll cover cross-signing
below)" … but I don't see them doing that.<br class="">
<br class="">
There is also no commit in their git history
regarding cross-signed certificates or multiple
signatures, so I don't think updating to 4.8.1
would help.<br class="">
<br class="">
The intermediate "R3" has issuer "C=US, O=Internet
Security Research Group, CN=ISRG Root X1", which
should match the new ISRG certificate, but not the
DST certificate. Maybe that's identified by a
fingerprint in the signature though (?), and maybe
the DST signature comes first and WolfSSL just
doesn't check further signatures if the first
fails?<br class="">
<br class="">
I'll look into the code later on, but need to take
care of my daily job duties first.<br class="">
<br class="">
If we can't fix this today, I think our best
option is to switch back to mbedTLS for the final
3.2 release and try to fix this for the initial
3.3 release.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 29.09.21 um 03:40
schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:F09F30F9-DBC9-4896-B83D-0050749787E0@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div class=""><br class="">
</div>
<div class="">It seems the dexter server returns
different certificates depending on the
presence of SNI extension in the TSL
negotiation. That is pretty normal (where a
single IP is serving multiple different
websites), but I am not sure if our client
library supports that?</div>
<div class=""><br class="">
</div>
<div class="">Without SNI, we get:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class="">Serial Number:</div>
<div class="">
04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54</div>
</div>
<div class="">Subject: CN=<a href="http://ns34.expeedo.de/" class="" moz-do-not-send="true">ns34.expeedo.de</a></div>
<div class="">
<div class=""> X509v3 Subject Alternative
Name:</div>
<div class=""> DNS:<a href="http://ns34.expeedo.de/" class="" moz-do-not-send="true">ns34.expeedo.de</a></div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">With SNI <a href="http://dexters-web.de/" class="" moz-do-not-send="true">dexters-web.de</a>,
we get:</div>
<div class=""><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class="">
<div class="">Serial Number:</div>
<div class="">
03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0</div>
<div class="">Subject: CN=<a href="http://dexter.shopdriver.de/" class="" moz-do-not-send="true">dexter.shopdriver.de</a></div>
<div class="">
<div class="">X509v3 Subject Alternative
Name:</div>
<div class=""> DNS:<a href="http://dexter.shopdriver.de/" class="" moz-do-not-send="true">dexter.shopdriver.de</a>,
DNS:<a href="http://dexters-web.de/" class="" moz-do-not-send="true">dexters-web.de</a>,
DNS:<a href="http://ovms.dexters-web.de/" class="" moz-do-not-send="true">ovms.dexters-web.de</a>,
DNS:<a href="http://www.dexter.shopdriver.de/" class="" moz-do-not-send="true">www.dexter.shopdriver.de</a>,
DNS:<a href="http://www.dexters-web.de/" class="" moz-do-not-send="true">www.dexters-web.de</a></div>
</div>
</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">Assuming the SNI is set
correctly by our library, looking at the <a href="http://dexter.shopdriver.de/" class="" moz-do-not-send="true">dexter.shopdriver.de</a> chain,
we have:</div>
<div class=""><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class="">Subject: CN=<a href="http://dexter.shopdriver.de/" class="" moz-do-not-send="true">dexter.shopdriver.de</a></div>
</div>
<div class="">Issuer: C=US, O=Let's Encrypt,
CN=R3</div>
<div class=""><br class="">
</div>
<div class="">Subject: C=US, O=Let's Encrypt,
CN=R3</div>
<div class="">Issuer: C=US, O=Internet
Security Research Group, CN=ISRG Root X1</div>
<div class=""><br class="">
</div>
<div class="">Subject: C=US, O=Internet
Security Research Group, CN=ISRG Root X1</div>
<div class="">Issuer: O=Digital Signature
Trust Co., CN=DST Root CA X3</div>
<div class=""><br class="">
</div>
<div class="">and
the components/ovms_tls/trustedca/dst.crt we
have is:</div>
<div class=""><br class="">
</div>
<div class="">Subject: O=Digital Signature
Trust Co., CN=DST Root CA X3</div>
<div class="">Issuer: O=Digital Signature
Trust Co., CN=DST Root CA X3</div>
<div class="">(Which expires Sep 30 14:01:15
2021 GMT and was issued just a year earlier
- ridiculous for a root CA)</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">The components/ovms_tls/trustedca/isrg_x1.crt
is:</div>
<div class=""><br class="">
</div>
</div>
<blockquote style="margin: 0 0 0 40px; border:
none; padding: 0px;" class="">
<div class="">
<div class="">Subject: C=US, O=Internet
Security Research Group, CN=ISRG Root X1</div>
</div>
<div class="">Issuer: C=US, O=Internet
Security Research Group, CN=ISRG Root X1</div>
<div class="">
<div class="">Validity</div>
<div class=""> Not Before: Jun 4
11:04:38 2015 GMT</div>
<div class=""> Not After : Jun 4
11:04:38 2035 GMT</div>
<div class="">sha256WithRSAEncryption 4096
bit</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">Our problem could be either the
wolfssl not recognising the X1 cert, or not
supporting the cross-signing arrangement
(where there are two paths to verify - the
expired X3 and the provided X1). I suspect
the latter. In an ideal world, having both
X1 and X3 trusted shouldn’t be a problem.</div>
<div class=""><br class="">
</div>
<div class="">There is a later 4.8.1 version
of wolfssl (we seem to use 4.7.0 released
February 2021). Perhaps we can try to update
to that?</div>
<div class=""><br class="">
</div>
<div class="">Mark</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 29 Sep 2021, at 4:11 AM,
Michael Balzer <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="content-isolator__container">
<div class="protected-part">
<div class="protected-title">Signed
PGP part</div>
<div class="protected-content">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div class=""> I can now confirm
it's a WolfSSL issue :-(<br class="">
<br class="">
I've switched back to release
3.2.016, i.e. before changing to
WolfSSL, and the ISRG Root X1
certificate works perfectly,
just as it should.<br class="">
<br class="">
Steve, I remember you included a
config option to enable using
WolfSSL, but cannot find it now.
Can you give me a pointer, or
did you remove that option later
on?<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am
28.09.21 um 21:53 schrieb
Stephen Casner:<br class="">
</div>
<blockquote type="cite" cite="mid:alpine.OSX.2.21.9999.2109281244430.18922@auge.attlocal.net" class="">
<pre class="moz-quote-pre" wrap="">I wonder if we are hitting a key size limit in the WolfSSL code. I
know there are configuration parameters in user_settings.h for
different SHA sizes, but I don't recall one for RSA keys.
-- Steve
On Tue, 28 Sep 2021, Michael Balzer wrote:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">Still no luck, and getting clueless... :-/
Let's Encrypt have a live test server:
<a class="moz-txt-link-freetext" href="https://valid-isrgrootx1.letsencrypt.org/" moz-do-not-send="true">https://valid-isrgrootx1.letsencrypt.org/</a>
Here's a simple test script to access that server:
(function(){
HTTP.request({
url: <a class="moz-txt-link-rfc2396E" href="https://valid-isrgrootx1.letsencrypt.org/" moz-do-not-send="true">"https://valid-isrgrootx1.letsencrypt.org/"</a>,
always: function() { JSON.print(this); }
});
})();
Just copy this into the editor and evaluate.
After commenting out the DST certificate from the TLS source, the http request
terminates with an SSL error (-3) with installed...
* self-signed ISRG certificate
* cross-signed ISRG / DST root certificate
* LE R3 certificate
* any combination of these three.
But the access works immediately after reinstalling the old DST root
certificate. There is no DST reference in the chain of that server...
I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT
lib --; apparently all they needed to do was to add the very same self-signed
ISRG X1 certificate I added:
*
<a class="moz-txt-link-freetext" href="https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5" moz-do-not-send="true">https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5</a>
So: why would WolfSSL not use the supplied ISRG certificate? Why would it use
the DST cert without DST being present as an issuer?
The ISRG certificate is the only one having...
signed using : RSA with SHA-256
RSA key size : 4096 bits
But that would surely be supported by WolfSSL, wouldn't it?
I'm running out of ideas, and this really isn't my primary area. Any help is
appreciated.
Regards,
Michael
Am 28.09.21 um 19:15 schrieb Stephen Casner:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">Michael,
Certainly within our code as it stands there is not any mechanism to
tell the WolfSSL code to adjust its certificate validation procedure.
The browser certificate substitution that you hypothesize is not clear
to me. I would expect the validation to simply follow the chain.
-- Steve
On Tue, 28 Sep 2021, Michael Balzer wrote:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">More info:
All my browsers already have a builtin ISRG X1 certificate signed by ISRG
only, that's the new version:
<a class="moz-txt-link-freetext" href="https://crt.sh/?id=9314791" moz-do-not-send="true">https://crt.sh/?id=9314791</a>
My server still sends the ISRG X1 certificate cross signed / issued by DST
Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on
the
last renewal (last month!):
<a class="moz-txt-link-freetext" href="https://crt.sh/?id=3958242236" moz-do-not-send="true">https://crt.sh/?id=3958242236</a>
Without the DST root cert, WolfSSL then fails validating the DST signed X1
root certificate (I assume):
<a class="moz-txt-link-freetext" href="https://www.wolfssl.com/docs/wolfssl-manual/ch7/" moz-do-not-send="true">https://www.wolfssl.com/docs/wolfssl-manual/ch7/</a>
My servers will continue sending that chain including the outdated root
cert
probably until the next renewal, so it's possible having added the new X1
root
certificate didn't solve the issue.
The browsers seem to know how to substitute the DST signed certificate by
the
builtin self-signed (?). Is there a similar option in WolfSSL, and do we
need
to enable that?
Steve, can you confirm this, do you know a solution?
Regards,
Michael
Am 28.09.21 um 15:34 schrieb Michael Balzer:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">I've tried adding the intermediate cert ("R3") and then also my site
certificate, that didn't help.
Only adding the DST cert again fixes the connection.
Any ideas?
OVMS# tls trust list
...
ISRG Root X1 length 1939 bytes
1939 byte certificate: ISRG Root X1
cert. version : 3
serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
issuer name : C=US, O=Internet Security Research Group, CN=ISRG
Root
X1
subject name : C=US, O=Internet Security Research Group, CN=ISRG
Root
X1
issued on : 2015-06-04 11:04:38
expires on : 2035-06-04 11:04:38
signed using : RSA with SHA-256
RSA key size : 4096 bits
basic constraints : CA=true
key usage : Key Cert Sign, CRL Sign
...
dexter length 1972 bytes
1972 byte certificate: dexter
cert. version : 3
serial number :
04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
issuer name : C=US, O=Let's Encrypt, CN=R3
subject name : CN=<a href="http://dexter.shopdriver.de/" class="" moz-do-not-send="true">dexter.shopdriver.de</a>
issued on : 2021-08-07 05:47:57
expires on : 2021-11-05 05:47:55
signed using : RSA with SHA-256
RSA key size : 2048 bits
basic constraints : CA=false
subject alt name : <a href="http://dexter.shopdriver.de/" class="" moz-do-not-send="true">dexter.shopdriver.de</a>, <a href="http://dexters-web.de/" class="" moz-do-not-send="true">dexters-web.de</a>,
<a href="http://ovms.dexters-web.de/" class="" moz-do-not-send="true">ovms.dexters-web.de</a>, <a class="moz-txt-link-abbreviated" href="http://www.dexter.shopdriver.de/" moz-do-not-send="true">www.dexter.shopdriver.de</a>, <a class="moz-txt-link-abbreviated" href="http://www.dexters-web.de/" moz-do-not-send="true">www.dexters-web.de</a>
key usage : Digital Signature, Key Encipherment
ext key usage : TLS Web Server Authentication, TLS Web Client
Authentication
...
r3 length 1826 bytes
1826 byte certificate: r3
cert. version : 3
serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
issuer name : C=US, O=Internet Security Research Group, CN=ISRG
Root
X1
subject name : C=US, O=Let's Encrypt, CN=R3
issued on : 2020-09-04 00:00:00
expires on : 2025-09-15 16:00:00
signed using : RSA with SHA-256
RSA key size : 2048 bits
basic constraints : CA=true, max_pathlen=0
key usage : Digital Signature, Key Cert Sign, CRL Sign
ext key usage : TLS Web Client Authentication, TLS Web Server
Authentication
Am 28.09.21 um 15:07 schrieb Michael Balzer:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">We would need to bypass / shortcut the "eap" test phase.
But I agree, "master" is stable, I haven't had any issues or reports,
so I
think we could do that. The FreeRTOS timer issue I'm working on only
affects very specific conditions, so not necessary to wait for that.
Should we remove the expiring DST certificate in that release then?
...uh oh: just tried removing the DST certificate: the module cannot
connect to my server anymore...!?
I (490213) ovms-server-v2: Connection is <a href="http://ovms.dexters-web.de:6870/" class="" moz-do-not-send="true">ovms.dexters-web.de:6870</a>
TEST1
I (490213) ovms-server-v2: Status: Connecting...
V (490723) ovms-server-v2:
OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
W (490723) ovms-server-v2: Connection failed
E (490723) ovms-server-v2: Status: Error: Connection failed
V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE)
I (490723) ovms-server-v2: Status: Disconnected
Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">Shall we release a full update? The last 3.2?
What we have now in master seems stable.
Mark
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">On 28 Sep 2021, at 5:39 PM, Michael Balzer <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><dexter@expeedo.de></a>
wrote:
Everyone,
the DST root certificate we include (DST Root CA X3) expires on
September 30, i.e. in two days.
OVMS# tls trust list
DST Root CA X3 length 1200 bytes
1200 byte certificate: DST Root CA X3
cert. version : 3
serial number :
44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
issuer name : O=Digital Signature Trust Co., CN=DST Root
CA X3
subject name : O=Digital Signature Trust Co., CN=DST Root
CA X3
issued on : 2000-09-30 21:12:19
* expires on : 2021-09-30 14:01:15*
signed using : RSA with SHA1
RSA key size : 2048 bits
basic constraints : CA=true
key usage : Key Cert Sign, CRL Sign
AFAICT, this root certificate is currently used by the OVMS to
validate Let's Encrypt certificates.
*
<a class="moz-txt-link-freetext" href="https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/" moz-do-not-send="true">https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/</a>
* <a class="moz-txt-link-freetext" href="https://letsencrypt.org/docs/certificate-compatibility/" moz-do-not-send="true">https://letsencrypt.org/docs/certificate-compatibility/</a>
Unfortunately, we missed adding the followup LE root certificate
"ISRG
Root X1" in time.
I've just added that certificate to our builtin certificate
repository, but it's too late now to roll out a "main" update in
time
(isn't it?).
So, to prevent losing TLS connectivity with LE servers, users need
to
manually add the ISRG Root X1 certificate to their TLS
repositories.
I've added a section on this to our user manual:
* <a class="moz-txt-link-freetext" href="https://docs.openvehicles.com/en/latest/userguide/ssltls.html" moz-do-not-send="true">https://docs.openvehicles.com/en/latest/userguide/ssltls.html</a>
If users contact you, point them to that page.
We probably should also remove the expired DST root certificate
after
September 30.
Regards,
Michael
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div>
</div>
</div>
<br class="">
<iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div>
</div>
</div>
<br class="">
<iframe class="content-isolator__isolated-content" sandbox="allow-scripts" scrolling="auto" style="border:none;display:block;overflow:auto;" data-src="data:text/html;charset=UTF-8;base64,PGlmcmFtZS1jb250ZW50IGRhdGEtaWZyYW1lLWhlaWdodD0idHJ1ZSI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188QlI+T3Ztc0RldiBtYWlsaW5nIGxpc3Q8QlI+T3Ztc0RldkBsaXN0cy5vcGVudmVoaWNsZXMuY29tPEJSPmh0dHA6Ly9saXN0cy5vcGVudmVoaWNsZXMuY29tL21haWxtYW4vbGlzdGluZm8vb3Ztc2RldjxCUj48L2lmcmFtZS1jb250ZW50Pg==" width="200" height="10"></iframe></div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26</pre>
</div></blockquote></body></html>