[Ovmsdev] Urgent TLS root certificate issue (Let's Encrypt)
Michael Balzer
dexter at expeedo.de
Wed Sep 29 20:09:38 HKT 2021
I managed to get wolfSSL debug logs, but don't know how to interpret
them. See files attached.
The final error seems to be…
I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188
I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188
I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188
E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818
file:/home/balzer/esp/Open-Vehic
E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507
file:/home/balzer/esp/Open-Vehic
which is
ASN_NO_SIGNER_E = -188, /* ASN no signer to confirm failure */
I've had a look at the code, I don't see a chance to find the bug within
the remaining time.
We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL
commits now. Steve, if you already prepared something in that direction,
tell me.
Regards,
Michael
Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:
>
> The cross-signing bit seems to be on the X1 cert provided in the
> certificate chain by the server. The one provided there is signed by
> X3. The problem comes because a different X1 certificate may be
> provided as a trusted CA (and that X1 is self-signed with a long
> expiration date - 2035). So the verification process has two possible
> paths to verify the cert, and the result depends on which one it
> chooses (and how it handles a trusted CA cert supplementing /
> replacing one provided by the server).
>
> Overall, a mess. I’m fighting this same issue now with a bunch of
> systems in my day job, and likely a large part of the Internet is
> going to start to fail tomorrow. Particularly a lot of small embedded
> devices.
>
> Regards, Mark.
>
>> On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter at expeedo.de
>> <mailto:dexter at expeedo.de>> wrote:
>>
>> Signed PGP part
>> SNI was pretty much intact already, I took care of a remaining
>> Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 /
>> mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).
>>
>> I understand the WolfSSL chain management uses a supplied root cert
>> over one delivered by the server:
>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3
>>
>>> Our problem could be either the wolfssl not recognising the X1 cert,
>>> or not supporting the cross-signing arrangement (where there are two
>>> paths to verify - the expired X3 and the provided X1). I suspect the
>>> latter. In an ideal world, having both X1 and X3 trusted shouldn’t
>>> be a problem.
>>
>> Is cross signing supported at all by WolfSSL? The only place I've
>> found it mentioned on wolfssl is this page:
>> https://www.wolfssl.com/certificate-chain-chain-trust/
>> …where they say "(We’ll cover cross-signing below)" … but I don't see
>> them doing that.
>>
>> There is also no commit in their git history regarding cross-signed
>> certificates or multiple signatures, so I don't think updating to
>> 4.8.1 would help.
>>
>> The intermediate "R3" has issuer "C=US, O=Internet Security Research
>> Group, CN=ISRG Root X1", which should match the new ISRG certificate,
>> but not the DST certificate. Maybe that's identified by a fingerprint
>> in the signature though (?), and maybe the DST signature comes first
>> and WolfSSL just doesn't check further signatures if the first fails?
>>
>> I'll look into the code later on, but need to take care of my daily
>> job duties first.
>>
>> If we can't fix this today, I think our best option is to switch back
>> to mbedTLS for the final 3.2 release and try to fix this for the
>> initial 3.3 release.
>>
>> Regards,
>> Michael
>>
>>
>> Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:
>>>
>>> It seems the dexter server returns different certificates depending
>>> on the presence of SNI extension in the TSL negotiation. That is
>>> pretty normal (where a single IP is serving multiple different
>>> websites), but I am not sure if our client library supports that?
>>>
>>> Without SNI, we get:
>>>
>>> Serial Number:
>>> 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
>>> Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
>>> X509v3 Subject Alternative Name:
>>> DNS:ns34.expeedo.de <http://ns34.expeedo.de/>
>>>
>>>
>>> With SNI dexters-web.de <http://dexters-web.de/>, we get:
>>>
>>> Serial Number:
>>> 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
>>> Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
>>> X509v3 Subject Alternative Name:
>>> DNS:dexter.shopdriver.de
>>> <http://dexter.shopdriver.de/>, DNS:dexters-web.de
>>> <http://dexters-web.de/>, DNS:ovms.dexters-web.de
>>> <http://ovms.dexters-web.de/>, DNS:www.dexter.shopdriver.de
>>> <http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de
>>> <http://www.dexters-web.de/>
>>>
>>>
>>> Assuming the SNI is set correctly by our library, looking at the
>>> dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:
>>>
>>> Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
>>> Issuer: C=US, O=Let's Encrypt, CN=R3
>>>
>>> Subject: C=US, O=Let's Encrypt, CN=R3
>>> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>>
>>> Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
>>>
>>> and the components/ovms_tls/trustedca/dst.crt we have is:
>>>
>>> Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
>>> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
>>> (Which expires Sep 30 14:01:15 2021 GMT and was issued just a
>>> year earlier - ridiculous for a root CA)
>>>
>>>
>>> The components/ovms_tls/trustedca/isrg_x1.crt is:
>>>
>>> Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>> Validity
>>> Not Before: Jun 4 11:04:38 2015 GMT
>>> Not After : Jun 4 11:04:38 2035 GMT
>>> sha256WithRSAEncryption 4096 bit
>>>
>>>
>>> Our problem could be either the wolfssl not recognising the X1 cert,
>>> or not supporting the cross-signing arrangement (where there are two
>>> paths to verify - the expired X3 and the provided X1). I suspect the
>>> latter. In an ideal world, having both X1 and X3 trusted shouldn’t
>>> be a problem.
>>>
>>> There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0
>>> released February 2021). Perhaps we can try to update to that?
>>>
>>> Mark
>>>
>>>> On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter at expeedo.de
>>>> <mailto:dexter at expeedo.de>> wrote:
>>>>
>>>> Signed PGP part
>>>> I can now confirm it's a WolfSSL issue :-(
>>>>
>>>> I've switched back to release 3.2.016, i.e. before changing to
>>>> WolfSSL, and the ISRG Root X1 certificate works perfectly, just as
>>>> it should.
>>>>
>>>> Steve, I remember you included a config option to enable using
>>>> WolfSSL, but cannot find it now. Can you give me a pointer, or did
>>>> you remove that option later on?
>>>>
>>>>
>>>>
>>>> Am 28.09.21 um 21:53 schrieb Stephen Casner:
>>>>> I wonder if we are hitting a key size limit in the WolfSSL code. I
>>>>> know there are configuration parameters in user_settings.h for
>>>>> different SHA sizes, but I don't recall one for RSA keys.
>>>>>
>>>>> -- Steve
>>>>>
>>>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>>>
>>>>>> Still no luck, and getting clueless... :-/
>>>>>>
>>>>>> Let's Encrypt have a live test server:
>>>>>> https://valid-isrgrootx1.letsencrypt.org/
>>>>>>
>>>>>> Here's a simple test script to access that server:
>>>>>>
>>>>>> (function(){
>>>>>> HTTP.request({
>>>>>> url:"https://valid-isrgrootx1.letsencrypt.org/",
>>>>>> always: function() { JSON.print(this); }
>>>>>> });
>>>>>> })();
>>>>>>
>>>>>> Just copy this into the editor and evaluate.
>>>>>>
>>>>>> After commenting out the DST certificate from the TLS source, the http request
>>>>>> terminates with an SSL error (-3) with installed...
>>>>>>
>>>>>> * self-signed ISRG certificate
>>>>>> * cross-signed ISRG / DST root certificate
>>>>>> * LE R3 certificate
>>>>>> * any combination of these three.
>>>>>>
>>>>>> But the access works immediately after reinstalling the old DST root
>>>>>> certificate. There is no DST reference in the chain of that server...
>>>>>>
>>>>>>
>>>>>> I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT
>>>>>> lib --; apparently all they needed to do was to add the very same self-signed
>>>>>> ISRG X1 certificate I added:
>>>>>>
>>>>>> *
>>>>>> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5
>>>>>>
>>>>>>
>>>>>> So: why would WolfSSL not use the supplied ISRG certificate? Why would it use
>>>>>> the DST cert without DST being present as an issuer?
>>>>>>
>>>>>> The ISRG certificate is the only one having...
>>>>>>
>>>>>> signed using : RSA with SHA-256
>>>>>> RSA key size : 4096 bits
>>>>>>
>>>>>> But that would surely be supported by WolfSSL, wouldn't it?
>>>>>>
>>>>>> I'm running out of ideas, and this really isn't my primary area. Any help is
>>>>>> appreciated.
>>>>>>
>>>>>> Regards,
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> Am 28.09.21 um 19:15 schrieb Stephen Casner:
>>>>>>> Michael,
>>>>>>>
>>>>>>> Certainly within our code as it stands there is not any mechanism to
>>>>>>> tell the WolfSSL code to adjust its certificate validation procedure.
>>>>>>> The browser certificate substitution that you hypothesize is not clear
>>>>>>> to me. I would expect the validation to simply follow the chain.
>>>>>>>
>>>>>>> -- Steve
>>>>>>>
>>>>>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>>>>>
>>>>>>>> More info:
>>>>>>>>
>>>>>>>> All my browsers already have a builtin ISRG X1 certificate signed by ISRG
>>>>>>>> only, that's the new version:
>>>>>>>>
>>>>>>>> https://crt.sh/?id=9314791
>>>>>>>>
>>>>>>>> My server still sends the ISRG X1 certificate cross signed / issued by DST
>>>>>>>> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on
>>>>>>>> the
>>>>>>>> last renewal (last month!):
>>>>>>>>
>>>>>>>> https://crt.sh/?id=3958242236
>>>>>>>>
>>>>>>>> Without the DST root cert, WolfSSL then fails validating the DST signed X1
>>>>>>>> root certificate (I assume):
>>>>>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/
>>>>>>>>
>>>>>>>> My servers will continue sending that chain including the outdated root
>>>>>>>> cert
>>>>>>>> probably until the next renewal, so it's possible having added the new X1
>>>>>>>> root
>>>>>>>> certificate didn't solve the issue.
>>>>>>>>
>>>>>>>> The browsers seem to know how to substitute the DST signed certificate by
>>>>>>>> the
>>>>>>>> builtin self-signed (?). Is there a similar option in WolfSSL, and do we
>>>>>>>> need
>>>>>>>> to enable that?
>>>>>>>>
>>>>>>>> Steve, can you confirm this, do you know a solution?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Michael
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 28.09.21 um 15:34 schrieb Michael Balzer:
>>>>>>>>> I've tried adding the intermediate cert ("R3") and then also my site
>>>>>>>>> certificate, that didn't help.
>>>>>>>>>
>>>>>>>>> Only adding the DST cert again fixes the connection.
>>>>>>>>>
>>>>>>>>> Any ideas?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> OVMS# tls trust list
>>>>>>>>> ...
>>>>>>>>> ISRG Root X1 length 1939 bytes
>>>>>>>>> 1939 byte certificate: ISRG Root X1
>>>>>>>>> cert. version : 3
>>>>>>>>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
>>>>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>>>> Root
>>>>>>>>> X1
>>>>>>>>> subject name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>>>> Root
>>>>>>>>> X1
>>>>>>>>> issued on : 2015-06-04 11:04:38
>>>>>>>>> expires on : 2035-06-04 11:04:38
>>>>>>>>> signed using : RSA with SHA-256
>>>>>>>>> RSA key size : 4096 bits
>>>>>>>>> basic constraints : CA=true
>>>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>>>> ...
>>>>>>>>> dexter length 1972 bytes
>>>>>>>>> 1972 byte certificate: dexter
>>>>>>>>> cert. version : 3
>>>>>>>>> serial number :
>>>>>>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
>>>>>>>>> issuer name : C=US, O=Let's Encrypt, CN=R3
>>>>>>>>> subject name : CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
>>>>>>>>> issued on : 2021-08-07 05:47:57
>>>>>>>>> expires on : 2021-11-05 05:47:55
>>>>>>>>> signed using : RSA with SHA-256
>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>> basic constraints : CA=false
>>>>>>>>> subject alt name :dexter.shopdriver.de <http://dexter.shopdriver.de/>,dexters-web.de <http://dexters-web.de/>,
>>>>>>>>> ovms.dexters-web.de <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de
>>>>>>>>> key usage : Digital Signature, Key Encipherment
>>>>>>>>> ext key usage : TLS Web Server Authentication, TLS Web Client
>>>>>>>>> Authentication
>>>>>>>>> ...
>>>>>>>>> r3 length 1826 bytes
>>>>>>>>> 1826 byte certificate: r3
>>>>>>>>> cert. version : 3
>>>>>>>>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
>>>>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>>>> Root
>>>>>>>>> X1
>>>>>>>>> subject name : C=US, O=Let's Encrypt, CN=R3
>>>>>>>>> issued on : 2020-09-04 00:00:00
>>>>>>>>> expires on : 2025-09-15 16:00:00
>>>>>>>>> signed using : RSA with SHA-256
>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>> basic constraints : CA=true, max_pathlen=0
>>>>>>>>> key usage : Digital Signature, Key Cert Sign, CRL Sign
>>>>>>>>> ext key usage : TLS Web Client Authentication, TLS Web Server
>>>>>>>>> Authentication
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer:
>>>>>>>>>> We would need to bypass / shortcut the "eap" test phase.
>>>>>>>>>>
>>>>>>>>>> But I agree, "master" is stable, I haven't had any issues or reports,
>>>>>>>>>> so I
>>>>>>>>>> think we could do that. The FreeRTOS timer issue I'm working on only
>>>>>>>>>> affects very specific conditions, so not necessary to wait for that.
>>>>>>>>>>
>>>>>>>>>> Should we remove the expiring DST certificate in that release then?
>>>>>>>>>>
>>>>>>>>>> ...uh oh: just tried removing the DST certificate: the module cannot
>>>>>>>>>> connect to my server anymore...!?
>>>>>>>>>>
>>>>>>>>>> I (490213) ovms-server-v2: Connection isovms.dexters-web.de:6870 <http://ovms.dexters-web.de:6870/>
>>>>>>>>>> TEST1
>>>>>>>>>> I (490213) ovms-server-v2: Status: Connecting...
>>>>>>>>>> V (490723) ovms-server-v2:
>>>>>>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
>>>>>>>>>> W (490723) ovms-server-v2: Connection failed
>>>>>>>>>> E (490723) ovms-server-v2: Status: Error: Connection failed
>>>>>>>>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE)
>>>>>>>>>> I (490723) ovms-server-v2: Status: Disconnected
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:
>>>>>>>>>>> Shall we release a full update? The last 3.2?
>>>>>>>>>>>
>>>>>>>>>>> What we have now in master seems stable.
>>>>>>>>>>>
>>>>>>>>>>> Mark
>>>>>>>>>>>
>>>>>>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer<dexter at expeedo.de>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> the DST root certificate we include (DST Root CA X3) expires on
>>>>>>>>>>>> September 30, i.e. in two days.
>>>>>>>>>>>>
>>>>>>>>>>>> OVMS# tls trust list
>>>>>>>>>>>> DST Root CA X3 length 1200 bytes
>>>>>>>>>>>> 1200 byte certificate: DST Root CA X3
>>>>>>>>>>>> cert. version : 3
>>>>>>>>>>>> serial number :
>>>>>>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
>>>>>>>>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>>>>> CA X3
>>>>>>>>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>>>>> CA X3
>>>>>>>>>>>> issued on : 2000-09-30 21:12:19
>>>>>>>>>>>> * expires on : 2021-09-30 14:01:15*
>>>>>>>>>>>> signed using : RSA with SHA1
>>>>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>>>>> basic constraints : CA=true
>>>>>>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>>>>>>>
>>>>>>>>>>>> AFAICT, this root certificate is currently used by the OVMS to
>>>>>>>>>>>> validate Let's Encrypt certificates.
>>>>>>>>>>>>
>>>>>>>>>>>> *
>>>>>>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
>>>>>>>>>>>> *https://letsencrypt.org/docs/certificate-compatibility/
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately, we missed adding the followup LE root certificate
>>>>>>>>>>>> "ISRG
>>>>>>>>>>>> Root X1" in time.
>>>>>>>>>>>>
>>>>>>>>>>>> I've just added that certificate to our builtin certificate
>>>>>>>>>>>> repository, but it's too late now to roll out a "main" update in
>>>>>>>>>>>> time
>>>>>>>>>>>> (isn't it?).
>>>>>>>>>>>>
>>>>>>>>>>>> So, to prevent losing TLS connectivity with LE servers, users need
>>>>>>>>>>>> to
>>>>>>>>>>>> manually add the ISRG Root X1 certificate to their TLS
>>>>>>>>>>>> repositories.
>>>>>>>>>>>>
>>>>>>>>>>>> I've added a section on this to our user manual:
>>>>>>>>>>>>
>>>>>>>>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html
>>>>>>>>>>>>
>>>>>>>>>>>> If users contact you, point them to that page.
>>>>>>>>>>>>
>>>>>>>>>>>> We probably should also remove the expired DST root certificate
>>>>>>>>>>>> after
>>>>>>>>>>>> September 30.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Michael
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>> --
>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>> --
>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> OvmsDev mailing list
>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>> --
>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OvmsDev mailing list
>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>
>>>> --
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>
>> --
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/491d0eb8/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: httptest3.log
Type: text/x-log
Size: 170185 bytes
Desc: not available
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/491d0eb8/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: serverv2test2.log
Type: text/x-log
Size: 168248 bytes
Desc: not available
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/491d0eb8/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/491d0eb8/attachment-0001.sig>
More information about the OvmsDev
mailing list