[Ovmsdev] Urgent TLS root certificate issue (Let's Encrypt)
Michael Balzer
dexter at expeedo.de
Wed Sep 29 21:01:58 HKT 2021
Patch attached. Additionally I had set the boot debug level (early) to
info (may no longer be necessary as I later found that LOG_LOCAL_LEVEL
#define in esp32-crypt.h), and you need to disable log monitoring in the
USB console to prevent garbled output.
I tried WOLFSSL_ALT_CERT_CHAINS but only got crashes with that.
Regards,
Michael
Am 29.09.21 um 14:44 schrieb Mark Webb-Johnson:
>
> Can you send me the patch to enable debugging, and I will try with 3.3
> branch. I agree that rolling back 3.2 to mbedtls seems sensible for
> the short term.
>
> Regards, Mark
>
> P.S. I managed to fix this on another project running an old OpenSSL
> by removing the X3 trusted ca (it had both x1 and x3 trusted).
> Behavior is highly dependent on the library treatment of multiple
> verification paths and trusted ca vs cert chain conflicts.
>
>> On 29 Sep 2021, at 8:11 PM, Michael Balzer <dexter at expeedo.de> wrote:
>>
>> I managed to get wolfSSL debug logs, but don't know how to
>> interpret them. See files attached.
>>
>> The final error seems to be…
>>
>> I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188
>> I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188
>> I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188
>> E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818
>> file:/home/balzer/esp/Open-Vehic
>> E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507
>> file:/home/balzer/esp/Open-Vehic
>>
>> which is
>> ASN_NO_SIGNER_E = -188, /* ASN no signer to confirm failure */
>>
>> I've had a look at the code, I don't see a chance to find the bug
>> within the remaining time.
>>
>> We're 24 hours from expiration. I'll try reverting the
>> Mongoose-WolfSSL commits now. Steve, if you already prepared
>> something in that direction, tell me.
>>
>> Regards,
>> Michael
>>
>>
>> Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:
>>>
>>> The cross-signing bit seems to be on the X1 cert provided in the
>>> certificate chain by the server. The one provided there is signed by
>>> X3. The problem comes because a different X1 certificate may be
>>> provided as a trusted CA (and that X1 is self-signed with a long
>>> expiration date - 2035). So the verification process has two
>>> possible paths to verify the cert, and the result depends on which
>>> one it chooses (and how it handles a trusted CA cert supplementing /
>>> replacing one provided by the server).
>>>
>>> Overall, a mess. I’m fighting this same issue now with a bunch of
>>> systems in my day job, and likely a large part of the Internet is
>>> going to start to fail tomorrow. Particularly a lot of small
>>> embedded devices.
>>>
>>> Regards, Mark.
>>>
>>>> On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter at expeedo.de
>>>> <mailto:dexter at expeedo.de>> wrote:
>>>>
>>>> Signed PGP part
>>>> SNI was pretty much intact already, I took care of a remaining
>>>> Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73
>>>> / mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).
>>>>
>>>> I understand the WolfSSL chain management uses a supplied root cert
>>>> over one delivered by the server:
>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3
>>>>
>>>>> Our problem could be either the wolfssl not recognising the X1
>>>>> cert, or not supporting the cross-signing arrangement (where there
>>>>> are two paths to verify - the expired X3 and the provided X1). I
>>>>> suspect the latter. In an ideal world, having both X1 and X3
>>>>> trusted shouldn’t be a problem.
>>>>
>>>> Is cross signing supported at all by WolfSSL? The only place I've
>>>> found it mentioned on wolfssl is this page:
>>>> https://www.wolfssl.com/certificate-chain-chain-trust/
>>>> …where they say "(We’ll cover cross-signing below)" … but I don't
>>>> see them doing that.
>>>>
>>>> There is also no commit in their git history regarding cross-signed
>>>> certificates or multiple signatures, so I don't think updating to
>>>> 4.8.1 would help.
>>>>
>>>> The intermediate "R3" has issuer "C=US, O=Internet Security
>>>> Research Group, CN=ISRG Root X1", which should match the new ISRG
>>>> certificate, but not the DST certificate. Maybe that's identified
>>>> by a fingerprint in the signature though (?), and maybe the DST
>>>> signature comes first and WolfSSL just doesn't check further
>>>> signatures if the first fails?
>>>>
>>>> I'll look into the code later on, but need to take care of my daily
>>>> job duties first.
>>>>
>>>> If we can't fix this today, I think our best option is to switch
>>>> back to mbedTLS for the final 3.2 release and try to fix this for
>>>> the initial 3.3 release.
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>>
>>>> Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:
>>>>>
>>>>> It seems the dexter server returns different certificates
>>>>> depending on the presence of SNI extension in the TSL negotiation.
>>>>> That is pretty normal (where a single IP is serving multiple
>>>>> different websites), but I am not sure if our client library
>>>>> supports that?
>>>>>
>>>>> Without SNI, we get:
>>>>>
>>>>> Serial Number:
>>>>> 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
>>>>> Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
>>>>> X509v3 Subject Alternative Name:
>>>>> DNS:ns34.expeedo.de <http://ns34.expeedo.de/>
>>>>>
>>>>>
>>>>> With SNI dexters-web.de <http://dexters-web.de/>, we get:
>>>>>
>>>>> Serial Number:
>>>>> 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
>>>>> Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
>>>>> X509v3 Subject Alternative Name:
>>>>> DNS:dexter.shopdriver.de
>>>>> <http://dexter.shopdriver.de/>, DNS:dexters-web.de
>>>>> <http://dexters-web.de/>, DNS:ovms.dexters-web.de
>>>>> <http://ovms.dexters-web.de/>, DNS:www.dexter.shopdriver.de
>>>>> <http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de
>>>>> <http://www.dexters-web.de/>
>>>>>
>>>>>
>>>>> Assuming the SNI is set correctly by our library, looking at the
>>>>> dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:
>>>>>
>>>>> Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
>>>>> Issuer: C=US, O=Let's Encrypt, CN=R3
>>>>>
>>>>> Subject: C=US, O=Let's Encrypt, CN=R3
>>>>> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>>>>
>>>>> Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>>>> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
>>>>>
>>>>> and the components/ovms_tls/trustedca/dst.crt we have is:
>>>>>
>>>>> Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
>>>>> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
>>>>> (Which expires Sep 30 14:01:15 2021 GMT and was issued just a
>>>>> year earlier - ridiculous for a root CA)
>>>>>
>>>>>
>>>>> The components/ovms_tls/trustedca/isrg_x1.crt is:
>>>>>
>>>>> Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>>>> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>>>>> Validity
>>>>> Not Before: Jun 4 11:04:38 2015 GMT
>>>>> Not After : Jun 4 11:04:38 2035 GMT
>>>>> sha256WithRSAEncryption 4096 bit
>>>>>
>>>>>
>>>>> Our problem could be either the wolfssl not recognising the X1
>>>>> cert, or not supporting the cross-signing arrangement (where there
>>>>> are two paths to verify - the expired X3 and the provided X1). I
>>>>> suspect the latter. In an ideal world, having both X1 and X3
>>>>> trusted shouldn’t be a problem.
>>>>>
>>>>> There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0
>>>>> released February 2021). Perhaps we can try to update to that?
>>>>>
>>>>> Mark
>>>>>
>>>>>> On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter at expeedo.de
>>>>>> <mailto:dexter at expeedo.de>> wrote:
>>>>>>
>>>>>> Signed PGP part
>>>>>> I can now confirm it's a WolfSSL issue :-(
>>>>>>
>>>>>> I've switched back to release 3.2.016, i.e. before changing to
>>>>>> WolfSSL, and the ISRG Root X1 certificate works perfectly, just
>>>>>> as it should.
>>>>>>
>>>>>> Steve, I remember you included a config option to enable using
>>>>>> WolfSSL, but cannot find it now. Can you give me a pointer, or
>>>>>> did you remove that option later on?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Am 28.09.21 um 21:53 schrieb Stephen Casner:
>>>>>>> I wonder if we are hitting a key size limit in the WolfSSL code. I
>>>>>>> know there are configuration parameters in user_settings.h for
>>>>>>> different SHA sizes, but I don't recall one for RSA keys.
>>>>>>>
>>>>>>> -- Steve
>>>>>>>
>>>>>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>>>>>
>>>>>>>> Still no luck, and getting clueless... :-/
>>>>>>>>
>>>>>>>> Let's Encrypt have a live test server:
>>>>>>>> https://valid-isrgrootx1.letsencrypt.org/
>>>>>>>>
>>>>>>>> Here's a simple test script to access that server:
>>>>>>>>
>>>>>>>> (function(){
>>>>>>>> HTTP.request({
>>>>>>>> url:"https://valid-isrgrootx1.letsencrypt.org/",
>>>>>>>> always: function() { JSON.print(this); }
>>>>>>>> });
>>>>>>>> })();
>>>>>>>>
>>>>>>>> Just copy this into the editor and evaluate.
>>>>>>>>
>>>>>>>> After commenting out the DST certificate from the TLS source, the http request
>>>>>>>> terminates with an SSL error (-3) with installed...
>>>>>>>>
>>>>>>>> * self-signed ISRG certificate
>>>>>>>> * cross-signed ISRG / DST root certificate
>>>>>>>> * LE R3 certificate
>>>>>>>> * any combination of these three.
>>>>>>>>
>>>>>>>> But the access works immediately after reinstalling the old DST root
>>>>>>>> certificate. There is no DST reference in the chain of that server...
>>>>>>>>
>>>>>>>>
>>>>>>>> I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT
>>>>>>>> lib --; apparently all they needed to do was to add the very same self-signed
>>>>>>>> ISRG X1 certificate I added:
>>>>>>>>
>>>>>>>> *
>>>>>>>> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5
>>>>>>>>
>>>>>>>>
>>>>>>>> So: why would WolfSSL not use the supplied ISRG certificate? Why would it use
>>>>>>>> the DST cert without DST being present as an issuer?
>>>>>>>>
>>>>>>>> The ISRG certificate is the only one having...
>>>>>>>>
>>>>>>>> signed using : RSA with SHA-256
>>>>>>>> RSA key size : 4096 bits
>>>>>>>>
>>>>>>>> But that would surely be supported by WolfSSL, wouldn't it?
>>>>>>>>
>>>>>>>> I'm running out of ideas, and this really isn't my primary area. Any help is
>>>>>>>> appreciated.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Michael
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 28.09.21 um 19:15 schrieb Stephen Casner:
>>>>>>>>> Michael,
>>>>>>>>>
>>>>>>>>> Certainly within our code as it stands there is not any mechanism to
>>>>>>>>> tell the WolfSSL code to adjust its certificate validation procedure.
>>>>>>>>> The browser certificate substitution that you hypothesize is not clear
>>>>>>>>> to me. I would expect the validation to simply follow the chain.
>>>>>>>>>
>>>>>>>>> -- Steve
>>>>>>>>>
>>>>>>>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>>>>>>>
>>>>>>>>>> More info:
>>>>>>>>>>
>>>>>>>>>> All my browsers already have a builtin ISRG X1 certificate signed by ISRG
>>>>>>>>>> only, that's the new version:
>>>>>>>>>>
>>>>>>>>>> https://crt.sh/?id=9314791
>>>>>>>>>>
>>>>>>>>>> My server still sends the ISRG X1 certificate cross signed / issued by DST
>>>>>>>>>> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on
>>>>>>>>>> the
>>>>>>>>>> last renewal (last month!):
>>>>>>>>>>
>>>>>>>>>> https://crt.sh/?id=3958242236
>>>>>>>>>>
>>>>>>>>>> Without the DST root cert, WolfSSL then fails validating the DST signed X1
>>>>>>>>>> root certificate (I assume):
>>>>>>>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/
>>>>>>>>>>
>>>>>>>>>> My servers will continue sending that chain including the outdated root
>>>>>>>>>> cert
>>>>>>>>>> probably until the next renewal, so it's possible having added the new X1
>>>>>>>>>> root
>>>>>>>>>> certificate didn't solve the issue.
>>>>>>>>>>
>>>>>>>>>> The browsers seem to know how to substitute the DST signed certificate by
>>>>>>>>>> the
>>>>>>>>>> builtin self-signed (?). Is there a similar option in WolfSSL, and do we
>>>>>>>>>> need
>>>>>>>>>> to enable that?
>>>>>>>>>>
>>>>>>>>>> Steve, can you confirm this, do you know a solution?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Michael
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Am 28.09.21 um 15:34 schrieb Michael Balzer:
>>>>>>>>>>> I've tried adding the intermediate cert ("R3") and then also my site
>>>>>>>>>>> certificate, that didn't help.
>>>>>>>>>>>
>>>>>>>>>>> Only adding the DST cert again fixes the connection.
>>>>>>>>>>>
>>>>>>>>>>> Any ideas?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> OVMS# tls trust list
>>>>>>>>>>> ...
>>>>>>>>>>> ISRG Root X1 length 1939 bytes
>>>>>>>>>>> 1939 byte certificate: ISRG Root X1
>>>>>>>>>>> cert. version : 3
>>>>>>>>>>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
>>>>>>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>>>>>> Root
>>>>>>>>>>> X1
>>>>>>>>>>> subject name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>>>>>> Root
>>>>>>>>>>> X1
>>>>>>>>>>> issued on : 2015-06-04 11:04:38
>>>>>>>>>>> expires on : 2035-06-04 11:04:38
>>>>>>>>>>> signed using : RSA with SHA-256
>>>>>>>>>>> RSA key size : 4096 bits
>>>>>>>>>>> basic constraints : CA=true
>>>>>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>>>>>> ...
>>>>>>>>>>> dexter length 1972 bytes
>>>>>>>>>>> 1972 byte certificate: dexter
>>>>>>>>>>> cert. version : 3
>>>>>>>>>>> serial number :
>>>>>>>>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
>>>>>>>>>>> issuer name : C=US, O=Let's Encrypt, CN=R3
>>>>>>>>>>> subject name : CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
>>>>>>>>>>> issued on : 2021-08-07 05:47:57
>>>>>>>>>>> expires on : 2021-11-05 05:47:55
>>>>>>>>>>> signed using : RSA with SHA-256
>>>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>>>> basic constraints : CA=false
>>>>>>>>>>> subject alt name :dexter.shopdriver.de <http://dexter.shopdriver.de/>,dexters-web.de <http://dexters-web.de/>,
>>>>>>>>>>> ovms.dexters-web.de <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de
>>>>>>>>>>> key usage : Digital Signature, Key Encipherment
>>>>>>>>>>> ext key usage : TLS Web Server Authentication, TLS Web Client
>>>>>>>>>>> Authentication
>>>>>>>>>>> ...
>>>>>>>>>>> r3 length 1826 bytes
>>>>>>>>>>> 1826 byte certificate: r3
>>>>>>>>>>> cert. version : 3
>>>>>>>>>>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
>>>>>>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>>>>>> Root
>>>>>>>>>>> X1
>>>>>>>>>>> subject name : C=US, O=Let's Encrypt, CN=R3
>>>>>>>>>>> issued on : 2020-09-04 00:00:00
>>>>>>>>>>> expires on : 2025-09-15 16:00:00
>>>>>>>>>>> signed using : RSA with SHA-256
>>>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>>>> basic constraints : CA=true, max_pathlen=0
>>>>>>>>>>> key usage : Digital Signature, Key Cert Sign, CRL Sign
>>>>>>>>>>> ext key usage : TLS Web Client Authentication, TLS Web Server
>>>>>>>>>>> Authentication
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer:
>>>>>>>>>>>> We would need to bypass / shortcut the "eap" test phase.
>>>>>>>>>>>>
>>>>>>>>>>>> But I agree, "master" is stable, I haven't had any issues or reports,
>>>>>>>>>>>> so I
>>>>>>>>>>>> think we could do that. The FreeRTOS timer issue I'm working on only
>>>>>>>>>>>> affects very specific conditions, so not necessary to wait for that.
>>>>>>>>>>>>
>>>>>>>>>>>> Should we remove the expiring DST certificate in that release then?
>>>>>>>>>>>>
>>>>>>>>>>>> ...uh oh: just tried removing the DST certificate: the module cannot
>>>>>>>>>>>> connect to my server anymore...!?
>>>>>>>>>>>>
>>>>>>>>>>>> I (490213) ovms-server-v2: Connection isovms.dexters-web.de:6870 <http://ovms.dexters-web.de:6870/>
>>>>>>>>>>>> TEST1
>>>>>>>>>>>> I (490213) ovms-server-v2: Status: Connecting...
>>>>>>>>>>>> V (490723) ovms-server-v2:
>>>>>>>>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
>>>>>>>>>>>> W (490723) ovms-server-v2: Connection failed
>>>>>>>>>>>> E (490723) ovms-server-v2: Status: Error: Connection failed
>>>>>>>>>>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE)
>>>>>>>>>>>> I (490723) ovms-server-v2: Status: Disconnected
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:
>>>>>>>>>>>>> Shall we release a full update? The last 3.2?
>>>>>>>>>>>>>
>>>>>>>>>>>>> What we have now in master seems stable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mark
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer<dexter at expeedo.de>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the DST root certificate we include (DST Root CA X3) expires on
>>>>>>>>>>>>>> September 30, i.e. in two days.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OVMS# tls trust list
>>>>>>>>>>>>>> DST Root CA X3 length 1200 bytes
>>>>>>>>>>>>>> 1200 byte certificate: DST Root CA X3
>>>>>>>>>>>>>> cert. version : 3
>>>>>>>>>>>>>> serial number :
>>>>>>>>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
>>>>>>>>>>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>>>>>>> CA X3
>>>>>>>>>>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>>>>>>> CA X3
>>>>>>>>>>>>>> issued on : 2000-09-30 21:12:19
>>>>>>>>>>>>>> * expires on : 2021-09-30 14:01:15*
>>>>>>>>>>>>>> signed using : RSA with SHA1
>>>>>>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>>>>>>> basic constraints : CA=true
>>>>>>>>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AFAICT, this root certificate is currently used by the OVMS to
>>>>>>>>>>>>>> validate Let's Encrypt certificates.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
>>>>>>>>>>>>>> *https://letsencrypt.org/docs/certificate-compatibility/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unfortunately, we missed adding the followup LE root certificate
>>>>>>>>>>>>>> "ISRG
>>>>>>>>>>>>>> Root X1" in time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've just added that certificate to our builtin certificate
>>>>>>>>>>>>>> repository, but it's too late now to roll out a "main" update in
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>> (isn't it?).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, to prevent losing TLS connectivity with LE servers, users need
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> manually add the ISRG Root X1 certificate to their TLS
>>>>>>>>>>>>>> repositories.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've added a section on this to our user manual:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If users contact you, point them to that page.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We probably should also remove the expired DST root certificate
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>> September 30.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Michael
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>>>> --
>>>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>>> --
>>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>>> --
>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>> --
>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OvmsDev mailing list
>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>
>>>> --
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>
>> --
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/3681a5da/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch-wolfdebug.diff
Type: text/x-patch
Size: 3046 bytes
Desc: not available
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/3681a5da/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/3681a5da/attachment-0001.sig>
More information about the OvmsDev
mailing list