[Ovmsdev] Urgent TLS root certificate issue (Let's Encrypt)
Stephen Casner
casner at acm.org
Wed Sep 29 22:56:35 HKT 2021
Michael,
No, I was sleeping at that hour.
I can send a question about cross-signing support to WolfSSL support
if you like.
-- Steve
On Wed, 29 Sep 2021, Michael Balzer wrote:
> I managed to get wolfSSL debug logs, but don't know how to interpret them. See
> files attached.
>
> The final error seems to be...
>
> I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188
> I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188
> I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188
> E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818
> file:/home/balzer/esp/Open-Vehic
> E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507
> file:/home/balzer/esp/Open-Vehic
>
> which is
> ASN_NO_SIGNER_E = -188, /* ASN no signer to confirm failure */
>
> I've had a look at the code, I don't see a chance to find the bug within the
> remaining time.
>
> We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL
> commits now. Steve, if you already prepared something in that direction, tell
> me.
>
> Regards,
> Michael
>
>
> Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:
> >
> > The cross-signing bit seems to be on the X1 cert provided in the certificate
> > chain by the server. The one provided there is signed by X3. The problem
> > comes because a different X1 certificate may be provided as a trusted CA
> > (and that X1 is self-signed with a long expiration date - 2035). So the
> > verification process has two possible paths to verify the cert, and the
> > result depends on which one it chooses (and how it handles a trusted CA cert
> > supplementing / replacing one provided by the server).
> >
> > Overall, a mess. I'm fighting this same issue now with a bunch of systems in
> > my day job, and likely a large part of the Internet is going to start to
> > fail tomorrow. Particularly a lot of small embedded devices.
> >
> > Regards, Mark.
> >
> > > On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter at expeedo.de
> > > <mailto:dexter at expeedo.de>> wrote:
> > >
> > > Signed PGP part
> > > SNI was pretty much intact already, I took care of a remaining Mongoose
> > > SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose
> > > 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).
> > >
> > > I understand the WolfSSL chain management uses a supplied root cert over
> > > one delivered by the server:
> > > https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3
> > >
> > > > Our problem could be either the wolfssl not recognising the X1 cert, or
> > > > not supporting the cross-signing arrangement (where there are two paths
> > > > to verify - the expired X3 and the provided X1). I suspect the latter.
> > > > In an ideal world, having both X1 and X3 trusted shouldn't be a problem.
> > >
> > > Is cross signing supported at all by WolfSSL? The only place I've found it
> > > mentioned on wolfssl is this page:
> > > https://www.wolfssl.com/certificate-chain-chain-trust/
> > > ...where they say "(We'll cover cross-signing below)" ... but I don't see
> > > them doing that.
> > >
> > > There is also no commit in their git history regarding cross-signed
> > > certificates or multiple signatures, so I don't think updating to 4.8.1
> > > would help.
> > >
> > > The intermediate "R3" has issuer "C=US, O=Internet Security Research
> > > Group, CN=ISRG Root X1", which should match the new ISRG certificate, but
> > > not the DST certificate. Maybe that's identified by a fingerprint in the
> > > signature though (?), and maybe the DST signature comes first and WolfSSL
> > > just doesn't check further signatures if the first fails?
> > >
> > > I'll look into the code later on, but need to take care of my daily job
> > > duties first.
> > >
> > > If we can't fix this today, I think our best option is to switch back to
> > > mbedTLS for the final 3.2 release and try to fix this for the initial 3.3
> > > release.
> > >
> > > Regards,
> > > Michael
> > >
> > >
> > > Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:
> > > >
> > > > It seems the dexter server returns different certificates depending on
> > > > the presence of SNI extension in the TSL negotiation. That is pretty
> > > > normal (where a single IP is serving multiple different websites), but I
> > > > am not sure if our client library supports that?
> > > >
> > > > Without SNI, we get:
> > > >
> > > > Serial Number:
> > > > 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
> > > > Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
> > > > X509v3 Subject Alternative Name:
> > > > DNS:ns34.expeedo.de <http://ns34.expeedo.de/>
> > > >
> > > >
> > > > With SNI dexters-web.de <http://dexters-web.de/>, we get:
> > > >
> > > > Serial Number:
> > > > 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
> > > > Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
> > > > X509v3 Subject Alternative Name:
> > > > DNS:dexter.shopdriver.de
> > > > <http://dexter.shopdriver.de/>, DNS:dexters-web.de
> > > > <http://dexters-web.de/>, DNS:ovms.dexters-web.de
> > > > <http://ovms.dexters-web.de/>, DNS:www.dexter.shopdriver.de
> > > > <http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de
> > > > <http://www.dexters-web.de/>
> > > >
> > > >
> > > > Assuming the SNI is set correctly by our library, looking at the
> > > > dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:
> > > >
> > > > Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
> > > > Issuer: C=US, O=Let's Encrypt, CN=R3
> > > >
> > > > Subject: C=US, O=Let's Encrypt, CN=R3
> > > > Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
> > > >
> > > > Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
> > > > Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
> > > >
> > > > and the components/ovms_tls/trustedca/dst.crt we have is:
> > > >
> > > > Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
> > > > Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
> > > > (Which expires Sep 30 14:01:15 2021 GMT and was issued just a
> > > > year earlier - ridiculous for a root CA)
> > > >
> > > >
> > > > The components/ovms_tls/trustedca/isrg_x1.crt is:
> > > >
> > > > Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
> > > > Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
> > > > Validity
> > > > Not Before: Jun 4 11:04:38 2015 GMT
> > > > Not After : Jun 4 11:04:38 2035 GMT
> > > > sha256WithRSAEncryption 4096 bit
> > > >
> > > >
> > > > Our problem could be either the wolfssl not recognising the X1 cert, or
> > > > not supporting the cross-signing arrangement (where there are two paths
> > > > to verify - the expired X3 and the provided X1). I suspect the latter.
> > > > In an ideal world, having both X1 and X3 trusted shouldn't be a problem.
> > > >
> > > > There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released
> > > > February 2021). Perhaps we can try to update to that?
> > > >
> > > > Mark
> > > >
> > > > > On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter at expeedo.de
> > > > > <mailto:dexter at expeedo.de>> wrote:
> > > > >
> > > > > Signed PGP part
> > > > > I can now confirm it's a WolfSSL issue :-(
> > > > >
> > > > > I've switched back to release 3.2.016, i.e. before changing to
> > > > > WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it
> > > > > should.
> > > > >
> > > > > Steve, I remember you included a config option to enable using
> > > > > WolfSSL, but cannot find it now. Can you give me a pointer, or did you
> > > > > remove that option later on?
> > > > >
> > > > >
> > > > >
> > > > > Am 28.09.21 um 21:53 schrieb Stephen Casner:
> > > > > > I wonder if we are hitting a key size limit in the WolfSSL code. I
> > > > > > know there are configuration parameters in user_settings.h for
> > > > > > different SHA sizes, but I don't recall one for RSA keys.
> > > > > >
> > > > > > -- Steve
> > > > > >
> > > > > > On Tue, 28 Sep 2021, Michael Balzer wrote:
> > > > > >
> > > > > > > Still no luck, and getting clueless... :-/
> > > > > > >
> > > > > > > Let's Encrypt have a live test server:
> > > > > > > https://valid-isrgrootx1.letsencrypt.org/
> > > > > > >
> > > > > > > Here's a simple test script to access that server:
> > > > > > >
> > > > > > > (function(){
> > > > > > > HTTP.request({
> > > > > > > url:"https://valid-isrgrootx1.letsencrypt.org/",
> > > > > > > always: function() { JSON.print(this); }
> > > > > > > });
> > > > > > > })();
> > > > > > >
> > > > > > > Just copy this into the editor and evaluate.
> > > > > > >
> > > > > > > After commenting out the DST certificate from the TLS source, the
> > > > > > > http request
> > > > > > > terminates with an SSL error (-3) with installed...
> > > > > > >
> > > > > > > * self-signed ISRG certificate
> > > > > > > * cross-signed ISRG / DST root certificate
> > > > > > > * LE R3 certificate
> > > > > > > * any combination of these three.
> > > > > > >
> > > > > > > But the access works immediately after reinstalling the old DST
> > > > > > > root
> > > > > > > certificate. There is no DST reference in the chain of that
> > > > > > > server...
> > > > > > >
> > > > > > >
> > > > > > > I found another ESP32 project -- not using WolfSSL but the esp-idf
> > > > > > > SSL & MQTT
> > > > > > > lib --; apparently all they needed to do was to add the very same
> > > > > > > self-signed
> > > > > > > ISRG X1 certificate I added:
> > > > > > >
> > > > > > > *
> > > > > > > https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5
> > > > > > >
> > > > > > >
> > > > > > > So: why would WolfSSL not use the supplied ISRG certificate? Why
> > > > > > > would it use
> > > > > > > the DST cert without DST being present as an issuer?
> > > > > > >
> > > > > > > The ISRG certificate is the only one having...
> > > > > > >
> > > > > > > signed using : RSA with SHA-256
> > > > > > > RSA key size : 4096 bits
> > > > > > >
> > > > > > > But that would surely be supported by WolfSSL, wouldn't it?
> > > > > > >
> > > > > > > I'm running out of ideas, and this really isn't my primary area.
> > > > > > > Any help is
> > > > > > > appreciated.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Michael
> > > > > > >
> > > > > > >
> > > > > > > Am 28.09.21 um 19:15 schrieb Stephen Casner:
> > > > > > > > Michael,
> > > > > > > >
> > > > > > > > Certainly within our code as it stands there is not any
> > > > > > > > mechanism to
> > > > > > > > tell the WolfSSL code to adjust its certificate validation
> > > > > > > > procedure.
> > > > > > > > The browser certificate substitution that you hypothesize is not
> > > > > > > > clear
> > > > > > > > to me. I would expect the validation to simply follow the
> > > > > > > > chain.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Steve
> > > > > > > >
> > > > > > > > On Tue, 28 Sep 2021, Michael Balzer wrote:
> > > > > > > >
> > > > > > > > > More info:
> > > > > > > > >
> > > > > > > > > All my browsers already have a builtin ISRG X1 certificate
> > > > > > > > > signed by ISRG
> > > > > > > > > only, that's the new version:
> > > > > > > > >
> > > > > > > > > https://crt.sh/?id=9314791
> > > > > > > > >
> > > > > > > > > My server still sends the ISRG X1 certificate cross signed /
> > > > > > > > > issued by DST
> > > > > > > > > Root CA X3. That's the chain it got from Let's Encrypt (via
> > > > > > > > > certbot) on
> > > > > > > > > the
> > > > > > > > > last renewal (last month!):
> > > > > > > > >
> > > > > > > > > https://crt.sh/?id=3958242236
> > > > > > > > >
> > > > > > > > > Without the DST root cert, WolfSSL then fails validating the
> > > > > > > > > DST signed X1
> > > > > > > > > root certificate (I assume):
> > > > > > > > > https://www.wolfssl.com/docs/wolfssl-manual/ch7/
> > > > > > > > >
> > > > > > > > > My servers will continue sending that chain including the
> > > > > > > > > outdated root
> > > > > > > > > cert
> > > > > > > > > probably until the next renewal, so it's possible having added
> > > > > > > > > the new X1
> > > > > > > > > root
> > > > > > > > > certificate didn't solve the issue.
> > > > > > > > >
> > > > > > > > > The browsers seem to know how to substitute the DST signed
> > > > > > > > > certificate by
> > > > > > > > > the
> > > > > > > > > builtin self-signed (?). Is there a similar option in WolfSSL,
> > > > > > > > > and do we
> > > > > > > > > need
> > > > > > > > > to enable that?
> > > > > > > > >
> > > > > > > > > Steve, can you confirm this, do you know a solution?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Michael
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Am 28.09.21 um 15:34 schrieb Michael Balzer:
> > > > > > > > > > I've tried adding the intermediate cert ("R3") and then also
> > > > > > > > > > my site
> > > > > > > > > > certificate, that didn't help.
> > > > > > > > > >
> > > > > > > > > > Only adding the DST cert again fixes the connection.
> > > > > > > > > >
> > > > > > > > > > Any ideas?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > OVMS# tls trust list
> > > > > > > > > > ...
> > > > > > > > > > ISRG Root X1 length 1939 bytes
> > > > > > > > > > 1939 byte certificate: ISRG Root X1
> > > > > > > > > > cert. version : 3
> > > > > > > > > > serial number :
> > > > > > > > > > 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
> > > > > > > > > > issuer name : C=US, O=Internet Security Research
> > > > > > > > > > Group, CN=ISRG
> > > > > > > > > > Root
> > > > > > > > > > X1
> > > > > > > > > > subject name : C=US, O=Internet Security Research
> > > > > > > > > > Group, CN=ISRG
> > > > > > > > > > Root
> > > > > > > > > > X1
> > > > > > > > > > issued on : 2015-06-04 11:04:38
> > > > > > > > > > expires on : 2035-06-04 11:04:38
> > > > > > > > > > signed using : RSA with SHA-256
> > > > > > > > > > RSA key size : 4096 bits
> > > > > > > > > > basic constraints : CA=true
> > > > > > > > > > key usage : Key Cert Sign, CRL Sign
> > > > > > > > > > ...
> > > > > > > > > > dexter length 1972 bytes
> > > > > > > > > > 1972 byte certificate: dexter
> > > > > > > > > > cert. version : 3
> > > > > > > > > > serial number :
> > > > > > > > > > 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
> > > > > > > > > > issuer name : C=US, O=Let's Encrypt, CN=R3
> > > > > > > > > > subject name : CN=dexter.shopdriver.de
> > > > > > > > > > <http://dexter.shopdriver.de/>
> > > > > > > > > > issued on : 2021-08-07 05:47:57
> > > > > > > > > > expires on : 2021-11-05 05:47:55
> > > > > > > > > > signed using : RSA with SHA-256
> > > > > > > > > > RSA key size : 2048 bits
> > > > > > > > > > basic constraints : CA=false
> > > > > > > > > > subject alt name :dexter.shopdriver.de
> > > > > > > > > > <http://dexter.shopdriver.de/>,dexters-web.de
> > > > > > > > > > <http://dexters-web.de/>,
> > > > > > > > > > ovms.dexters-web.de
> > > > > > > > > > <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de
> > > > > > > > > > key usage : Digital Signature, Key Encipherment
> > > > > > > > > > ext key usage : TLS Web Server Authentication, TLS
> > > > > > > > > > Web Client
> > > > > > > > > > Authentication
> > > > > > > > > > ...
> > > > > > > > > > r3 length 1826 bytes
> > > > > > > > > > 1826 byte certificate: r3
> > > > > > > > > > cert. version : 3
> > > > > > > > > > serial number :
> > > > > > > > > > 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
> > > > > > > > > > issuer name : C=US, O=Internet Security Research
> > > > > > > > > > Group, CN=ISRG
> > > > > > > > > > Root
> > > > > > > > > > X1
> > > > > > > > > > subject name : C=US, O=Let's Encrypt, CN=R3
> > > > > > > > > > issued on : 2020-09-04 00:00:00
> > > > > > > > > > expires on : 2025-09-15 16:00:00
> > > > > > > > > > signed using : RSA with SHA-256
> > > > > > > > > > RSA key size : 2048 bits
> > > > > > > > > > basic constraints : CA=true, max_pathlen=0
> > > > > > > > > > key usage : Digital Signature, Key Cert Sign,
> > > > > > > > > > CRL Sign
> > > > > > > > > > ext key usage : TLS Web Client Authentication, TLS
> > > > > > > > > > Web Server
> > > > > > > > > > Authentication
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Am 28.09.21 um 15:07 schrieb Michael Balzer:
> > > > > > > > > > > We would need to bypass / shortcut the "eap" test phase.
> > > > > > > > > > >
> > > > > > > > > > > But I agree, "master" is stable, I haven't had any issues
> > > > > > > > > > > or reports,
> > > > > > > > > > > so I
> > > > > > > > > > > think we could do that. The FreeRTOS timer issue I'm
> > > > > > > > > > > working on only
> > > > > > > > > > > affects very specific conditions, so not necessary to wait
> > > > > > > > > > > for that.
> > > > > > > > > > >
> > > > > > > > > > > Should we remove the expiring DST certificate in that
> > > > > > > > > > > release then?
> > > > > > > > > > >
> > > > > > > > > > > ...uh oh: just tried removing the DST certificate: the
> > > > > > > > > > > module cannot
> > > > > > > > > > > connect to my server anymore...!?
> > > > > > > > > > >
> > > > > > > > > > > I (490213) ovms-server-v2: Connection
> > > > > > > > > > > isovms.dexters-web.de:6870
> > > > > > > > > > > <http://ovms.dexters-web.de:6870/>
> > > > > > > > > > > TEST1
> > > > > > > > > > > I (490213) ovms-server-v2: Status: Connecting...
> > > > > > > > > > > V (490723) ovms-server-v2:
> > > > > > > > > > > OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
> > > > > > > > > > > W (490723) ovms-server-v2: Connection failed
> > > > > > > > > > > E (490723) ovms-server-v2: Status: Error: Connection
> > > > > > > > > > > failed
> > > > > > > > > > > V (490723) ovms-server-v2:
> > > > > > > > > > > OvmsServerV2MongooseCallback(MG_EV_CLOSE)
> > > > > > > > > > > I (490723) ovms-server-v2: Status: Disconnected
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:
> > > > > > > > > > > > Shall we release a full update? The last 3.2?
> > > > > > > > > > > >
> > > > > > > > > > > > What we have now in master seems stable.
> > > > > > > > > > > >
> > > > > > > > > > > > Mark
> > > > > > > > > > > >
> > > > > > > > > > > > > On 28 Sep 2021, at 5:39 PM, Michael
> > > > > > > > > > > > > Balzer<dexter at expeedo.de>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Everyone,
> > > > > > > > > > > > >
> > > > > > > > > > > > > the DST root certificate we include (DST Root CA X3)
> > > > > > > > > > > > > expires on
> > > > > > > > > > > > > September 30, i.e. in two days.
> > > > > > > > > > > > >
> > > > > > > > > > > > > OVMS# tls trust list
> > > > > > > > > > > > > DST Root CA X3 length 1200 bytes
> > > > > > > > > > > > > 1200 byte certificate: DST Root CA X3
> > > > > > > > > > > > > cert. version : 3
> > > > > > > > > > > > > serial number :
> > > > > > > > > > > > > 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
> > > > > > > > > > > > > issuer name : O=Digital Signature Trust Co.,
> > > > > > > > > > > > > CN=DST Root
> > > > > > > > > > > > > CA X3
> > > > > > > > > > > > > subject name : O=Digital Signature Trust Co.,
> > > > > > > > > > > > > CN=DST Root
> > > > > > > > > > > > > CA X3
> > > > > > > > > > > > > issued on : 2000-09-30 21:12:19
> > > > > > > > > > > > > * expires on : 2021-09-30 14:01:15*
> > > > > > > > > > > > > signed using : RSA with SHA1
> > > > > > > > > > > > > RSA key size : 2048 bits
> > > > > > > > > > > > > basic constraints : CA=true
> > > > > > > > > > > > > key usage : Key Cert Sign, CRL Sign
> > > > > > > > > > > > >
> > > > > > > > > > > > > AFAICT, this root certificate is currently used by the
> > > > > > > > > > > > > OVMS to
> > > > > > > > > > > > > validate Let's Encrypt certificates.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *
> > > > > > > > > > > > > https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
> > > > > > > > > > > > > *https://letsencrypt.org/docs/certificate-compatibility/
> > > > > > > > > > > > >
> > > > > > > > > > > > > Unfortunately, we missed adding the followup LE root
> > > > > > > > > > > > > certificate
> > > > > > > > > > > > > "ISRG
> > > > > > > > > > > > > Root X1" in time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've just added that certificate to our builtin
> > > > > > > > > > > > > certificate
> > > > > > > > > > > > > repository, but it's too late now to roll out a "main"
> > > > > > > > > > > > > update in
> > > > > > > > > > > > > time
> > > > > > > > > > > > > (isn't it?).
> > > > > > > > > > > > >
> > > > > > > > > > > > > So, to prevent losing TLS connectivity with LE
> > > > > > > > > > > > > servers, users need
> > > > > > > > > > > > > to
> > > > > > > > > > > > > manually add the ISRG Root X1 certificate to their TLS
> > > > > > > > > > > > > repositories.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've added a section on this to our user manual:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *https://docs.openvehicles.com/en/latest/userguide/ssltls.html
> > > > > > > > > > > > >
> > > > > > > > > > > > > If users contact you, point them to that page.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We probably should also remove the expired DST root
> > > > > > > > > > > > > certificate
> > > > > > > > > > > > > after
> > > > > > > > > > > > > September 30.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Michael
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Michael Balzer * Helkenberger Weg 9 * D-58256
> > > > > > > > > > > > > Ennepetal
> > > > > > > > > > > > > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> > > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > > OvmsDev mailing list
> > > > > > > > > > > > > OvmsDev at lists.openvehicles.com
> > > > > > > > > > > > > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > OvmsDev mailing list
> > > > > > > > > > > > OvmsDev at lists.openvehicles.com
> > > > > > > > > > > > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> > > > > > > > > > > --
> > > > > > > > > > > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> > > > > > > > > > > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > OvmsDev mailing list
> > > > > > > > > > > OvmsDev at lists.openvehicles.com
> > > > > > > > > > > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> > > > > > > > > > --
> > > > > > > > > > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> > > > > > > > > > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > OvmsDev mailing list
> > > > > > > > > > OvmsDev at lists.openvehicles.com
> > > > > > > > > > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> > > > > > > > > --
> > > > > > > > > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> > > > > > > > > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > OvmsDev mailing list
> > > > > > > > > OvmsDev at lists.openvehicles.com
> > > > > > > > > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> > > > > > > --
> > > > > > > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> > > > > > > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > OvmsDev mailing list
> > > > > > > OvmsDev at lists.openvehicles.com
> > > > > > > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> > > > >
> > > > > --
> > > > > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> > > > > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> > > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > OvmsDev mailing list
> > > > OvmsDev at lists.openvehicles.com
> > > > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
> > >
> > > --
> > > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> > > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> > >
> >
> >
> > _______________________________________________
> > OvmsDev mailing list
> > OvmsDev at lists.openvehicles.com
> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
>
More information about the OvmsDev
mailing list