[Ovmsdev] Urgent TLS root certificate issue (Let's Encrypt)
Mark Webb-Johnson
mark at webb-johnson.net
Wed Sep 29 09:40:42 HKT 2021
It seems the dexter server returns different certificates depending on the presence of SNI extension in the TSL negotiation. That is pretty normal (where a single IP is serving multiple different websites), but I am not sure if our client library supports that?
Without SNI, we get:
Serial Number:
04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
X509v3 Subject Alternative Name:
DNS:ns34.expeedo.de
With SNI dexters-web.de <http://dexters-web.de/>, we get:
Serial Number:
03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
X509v3 Subject Alternative Name:
DNS:dexter.shopdriver.de, DNS:dexters-web.de, DNS:ovms.dexters-web.de, DNS:www.dexter.shopdriver.de, DNS:www.dexters-web.de
Assuming the SNI is set correctly by our library, looking at the dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:
Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
Issuer: C=US, O=Let's Encrypt, CN=R3
Subject: C=US, O=Let's Encrypt, CN=R3
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
and the components/ovms_tls/trustedca/dst.crt we have is:
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
(Which expires Sep 30 14:01:15 2021 GMT and was issued just a year earlier - ridiculous for a root CA)
The components/ovms_tls/trustedca/isrg_x1.crt is:
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Validity
Not Before: Jun 4 11:04:38 2015 GMT
Not After : Jun 4 11:04:38 2035 GMT
sha256WithRSAEncryption 4096 bit
Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.
There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released February 2021). Perhaps we can try to update to that?
Mark
> On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter at expeedo.de> wrote:
>
> Signed PGP part
> I can now confirm it's a WolfSSL issue :-(
>
> I've switched back to release 3.2.016, i.e. before changing to WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it should.
>
> Steve, I remember you included a config option to enable using WolfSSL, but cannot find it now. Can you give me a pointer, or did you remove that option later on?
>
>
>
> Am 28.09.21 um 21:53 schrieb Stephen Casner:
>> I wonder if we are hitting a key size limit in the WolfSSL code. I
>> know there are configuration parameters in user_settings.h for
>> different SHA sizes, but I don't recall one for RSA keys.
>>
>> -- Steve
>>
>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>
>>> Still no luck, and getting clueless... :-/
>>>
>>> Let's Encrypt have a live test server:
>>> https://valid-isrgrootx1.letsencrypt.org/ <https://valid-isrgrootx1.letsencrypt.org/>
>>>
>>> Here's a simple test script to access that server:
>>>
>>> (function(){
>>> HTTP.request({
>>> url: "https://valid-isrgrootx1.letsencrypt.org/" <https://valid-isrgrootx1.letsencrypt.org/>,
>>> always: function() { JSON.print(this); }
>>> });
>>> })();
>>>
>>> Just copy this into the editor and evaluate.
>>>
>>> After commenting out the DST certificate from the TLS source, the http request
>>> terminates with an SSL error (-3) with installed...
>>>
>>> * self-signed ISRG certificate
>>> * cross-signed ISRG / DST root certificate
>>> * LE R3 certificate
>>> * any combination of these three.
>>>
>>> But the access works immediately after reinstalling the old DST root
>>> certificate. There is no DST reference in the chain of that server...
>>>
>>>
>>> I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT
>>> lib --; apparently all they needed to do was to add the very same self-signed
>>> ISRG X1 certificate I added:
>>>
>>> *
>>> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5 <https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5>
>>>
>>>
>>> So: why would WolfSSL not use the supplied ISRG certificate? Why would it use
>>> the DST cert without DST being present as an issuer?
>>>
>>> The ISRG certificate is the only one having...
>>>
>>> signed using : RSA with SHA-256
>>> RSA key size : 4096 bits
>>>
>>> But that would surely be supported by WolfSSL, wouldn't it?
>>>
>>> I'm running out of ideas, and this really isn't my primary area. Any help is
>>> appreciated.
>>>
>>> Regards,
>>> Michael
>>>
>>>
>>> Am 28.09.21 um 19:15 schrieb Stephen Casner:
>>>> Michael,
>>>>
>>>> Certainly within our code as it stands there is not any mechanism to
>>>> tell the WolfSSL code to adjust its certificate validation procedure.
>>>> The browser certificate substitution that you hypothesize is not clear
>>>> to me. I would expect the validation to simply follow the chain.
>>>>
>>>> -- Steve
>>>>
>>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>>
>>>>> More info:
>>>>>
>>>>> All my browsers already have a builtin ISRG X1 certificate signed by ISRG
>>>>> only, that's the new version:
>>>>>
>>>>> https://crt.sh/?id=9314791 <https://crt.sh/?id=9314791>
>>>>>
>>>>> My server still sends the ISRG X1 certificate cross signed / issued by DST
>>>>> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on
>>>>> the
>>>>> last renewal (last month!):
>>>>>
>>>>> https://crt.sh/?id=3958242236 <https://crt.sh/?id=3958242236>
>>>>>
>>>>> Without the DST root cert, WolfSSL then fails validating the DST signed X1
>>>>> root certificate (I assume):
>>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ <https://www.wolfssl.com/docs/wolfssl-manual/ch7/>
>>>>>
>>>>> My servers will continue sending that chain including the outdated root
>>>>> cert
>>>>> probably until the next renewal, so it's possible having added the new X1
>>>>> root
>>>>> certificate didn't solve the issue.
>>>>>
>>>>> The browsers seem to know how to substitute the DST signed certificate by
>>>>> the
>>>>> builtin self-signed (?). Is there a similar option in WolfSSL, and do we
>>>>> need
>>>>> to enable that?
>>>>>
>>>>> Steve, can you confirm this, do you know a solution?
>>>>>
>>>>> Regards,
>>>>> Michael
>>>>>
>>>>>
>>>>> Am 28.09.21 um 15:34 schrieb Michael Balzer:
>>>>>> I've tried adding the intermediate cert ("R3") and then also my site
>>>>>> certificate, that didn't help.
>>>>>>
>>>>>> Only adding the DST cert again fixes the connection.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>>
>>>>>> OVMS# tls trust list
>>>>>> ...
>>>>>> ISRG Root X1 length 1939 bytes
>>>>>> 1939 byte certificate: ISRG Root X1
>>>>>> cert. version : 3
>>>>>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>> Root
>>>>>> X1
>>>>>> subject name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>> Root
>>>>>> X1
>>>>>> issued on : 2015-06-04 11:04:38
>>>>>> expires on : 2035-06-04 11:04:38
>>>>>> signed using : RSA with SHA-256
>>>>>> RSA key size : 4096 bits
>>>>>> basic constraints : CA=true
>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>> ...
>>>>>> dexter length 1972 bytes
>>>>>> 1972 byte certificate: dexter
>>>>>> cert. version : 3
>>>>>> serial number :
>>>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
>>>>>> issuer name : C=US, O=Let's Encrypt, CN=R3
>>>>>> subject name : CN=dexter.shopdriver.de
>>>>>> issued on : 2021-08-07 05:47:57
>>>>>> expires on : 2021-11-05 05:47:55
>>>>>> signed using : RSA with SHA-256
>>>>>> RSA key size : 2048 bits
>>>>>> basic constraints : CA=false
>>>>>> subject alt name : dexter.shopdriver.de, dexters-web.de,
>>>>>> ovms.dexters-web.de, www.dexter.shopdriver.de <http://www.dexter.shopdriver.de/>, www.dexters-web.de <http://www.dexters-web.de/>
>>>>>> key usage : Digital Signature, Key Encipherment
>>>>>> ext key usage : TLS Web Server Authentication, TLS Web Client
>>>>>> Authentication
>>>>>> ...
>>>>>> r3 length 1826 bytes
>>>>>> 1826 byte certificate: r3
>>>>>> cert. version : 3
>>>>>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>> Root
>>>>>> X1
>>>>>> subject name : C=US, O=Let's Encrypt, CN=R3
>>>>>> issued on : 2020-09-04 00:00:00
>>>>>> expires on : 2025-09-15 16:00:00
>>>>>> signed using : RSA with SHA-256
>>>>>> RSA key size : 2048 bits
>>>>>> basic constraints : CA=true, max_pathlen=0
>>>>>> key usage : Digital Signature, Key Cert Sign, CRL Sign
>>>>>> ext key usage : TLS Web Client Authentication, TLS Web Server
>>>>>> Authentication
>>>>>>
>>>>>>
>>>>>>
>>>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer:
>>>>>>> We would need to bypass / shortcut the "eap" test phase.
>>>>>>>
>>>>>>> But I agree, "master" is stable, I haven't had any issues or reports,
>>>>>>> so I
>>>>>>> think we could do that. The FreeRTOS timer issue I'm working on only
>>>>>>> affects very specific conditions, so not necessary to wait for that.
>>>>>>>
>>>>>>> Should we remove the expiring DST certificate in that release then?
>>>>>>>
>>>>>>> ...uh oh: just tried removing the DST certificate: the module cannot
>>>>>>> connect to my server anymore...!?
>>>>>>>
>>>>>>> I (490213) ovms-server-v2: Connection is ovms.dexters-web.de:6870
>>>>>>> TEST1
>>>>>>> I (490213) ovms-server-v2: Status: Connecting...
>>>>>>> V (490723) ovms-server-v2:
>>>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
>>>>>>> W (490723) ovms-server-v2: Connection failed
>>>>>>> E (490723) ovms-server-v2: Status: Error: Connection failed
>>>>>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE)
>>>>>>> I (490723) ovms-server-v2: Status: Disconnected
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:
>>>>>>>> Shall we release a full update? The last 3.2?
>>>>>>>>
>>>>>>>> What we have now in master seems stable.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer <dexter at expeedo.de> <mailto:dexter at expeedo.de>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Everyone,
>>>>>>>>>
>>>>>>>>> the DST root certificate we include (DST Root CA X3) expires on
>>>>>>>>> September 30, i.e. in two days.
>>>>>>>>>
>>>>>>>>> OVMS# tls trust list
>>>>>>>>> DST Root CA X3 length 1200 bytes
>>>>>>>>> 1200 byte certificate: DST Root CA X3
>>>>>>>>> cert. version : 3
>>>>>>>>> serial number :
>>>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
>>>>>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>> CA X3
>>>>>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>> CA X3
>>>>>>>>> issued on : 2000-09-30 21:12:19
>>>>>>>>> * expires on : 2021-09-30 14:01:15*
>>>>>>>>> signed using : RSA with SHA1
>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>> basic constraints : CA=true
>>>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>>>>
>>>>>>>>> AFAICT, this root certificate is currently used by the OVMS to
>>>>>>>>> validate Let's Encrypt certificates.
>>>>>>>>>
>>>>>>>>> *
>>>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ <https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/>
>>>>>>>>> * https://letsencrypt.org/docs/certificate-compatibility/ <https://letsencrypt.org/docs/certificate-compatibility/>
>>>>>>>>>
>>>>>>>>> Unfortunately, we missed adding the followup LE root certificate
>>>>>>>>> "ISRG
>>>>>>>>> Root X1" in time.
>>>>>>>>>
>>>>>>>>> I've just added that certificate to our builtin certificate
>>>>>>>>> repository, but it's too late now to roll out a "main" update in
>>>>>>>>> time
>>>>>>>>> (isn't it?).
>>>>>>>>>
>>>>>>>>> So, to prevent losing TLS connectivity with LE servers, users need
>>>>>>>>> to
>>>>>>>>> manually add the ISRG Root X1 certificate to their TLS
>>>>>>>>> repositories.
>>>>>>>>>
>>>>>>>>> I've added a section on this to our user manual:
>>>>>>>>>
>>>>>>>>> * https://docs.openvehicles.com/en/latest/userguide/ssltls.html <https://docs.openvehicles.com/en/latest/userguide/ssltls.html>
>>>>>>>>>
>>>>>>>>> If users contact you, point them to that page.
>>>>>>>>>
>>>>>>>>> We probably should also remove the expired DST root certificate
>>>>>>>>> after
>>>>>>>>> September 30.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Michael
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>> _______________________________________________
>>>>>>>>> OvmsDev mailing list
>>>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>>>>> _______________________________________________
>>>>>>>> OvmsDev mailing list
>>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>>>> --
>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OvmsDev mailing list
>>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>> --
>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>> --
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/60f24204/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/60f24204/attachment-0001.sig>
More information about the OvmsDev
mailing list