[Ovmsdev] Urgent TLS root certificate issue (Let's Encrypt)
Michael Balzer
dexter at expeedo.de
Wed Sep 29 14:51:44 HKT 2021
SNI was pretty much intact already, I took care of a remaining Mongoose
SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose
8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).
I understand the WolfSSL chain management uses a supplied root cert over
one delivered by the server:
https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3
> Our problem could be either the wolfssl not recognising the X1 cert,
> or not supporting the cross-signing arrangement (where there are two
> paths to verify - the expired X3 and the provided X1). I suspect the
> latter. In an ideal world, having both X1 and X3 trusted shouldn’t be
> a problem.
Is cross signing supported at all by WolfSSL? The only place I've found
it mentioned on wolfssl is this page:
https://www.wolfssl.com/certificate-chain-chain-trust/
…where they say "(We’ll cover cross-signing below)" … but I don't see
them doing that.
There is also no commit in their git history regarding cross-signed
certificates or multiple signatures, so I don't think updating to 4.8.1
would help.
The intermediate "R3" has issuer "C=US, O=Internet Security Research
Group, CN=ISRG Root X1", which should match the new ISRG certificate,
but not the DST certificate. Maybe that's identified by a fingerprint in
the signature though (?), and maybe the DST signature comes first and
WolfSSL just doesn't check further signatures if the first fails?
I'll look into the code later on, but need to take care of my daily job
duties first.
If we can't fix this today, I think our best option is to switch back to
mbedTLS for the final 3.2 release and try to fix this for the initial
3.3 release.
Regards,
Michael
Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:
>
> It seems the dexter server returns different certificates depending on
> the presence of SNI extension in the TSL negotiation. That is pretty
> normal (where a single IP is serving multiple different websites), but
> I am not sure if our client library supports that?
>
> Without SNI, we get:
>
> Serial Number:
> 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
> Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de>
> X509v3 Subject Alternative Name:
> DNS:ns34.expeedo.de <http://ns34.expeedo.de>
>
>
> With SNI dexters-web.de <http://dexters-web.de>, we get:
>
> Serial Number:
> 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
> Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de>
> X509v3 Subject Alternative Name:
> DNS:dexter.shopdriver.de
> <http://dexter.shopdriver.de>, DNS:dexters-web.de
> <http://dexters-web.de>, DNS:ovms.dexters-web.de
> <http://ovms.dexters-web.de>, DNS:www.dexter.shopdriver.de
> <http://www.dexter.shopdriver.de>, DNS:www.dexters-web.de
> <http://www.dexters-web.de>
>
>
> Assuming the SNI is set correctly by our library, looking at the
> dexter.shopdriver.de <http://dexter.shopdriver.de> chain, we have:
>
> Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de>
> Issuer: C=US, O=Let's Encrypt, CN=R3
>
> Subject: C=US, O=Let's Encrypt, CN=R3
> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
>
> Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
>
> and the components/ovms_tls/trustedca/dst.crt we have is:
>
> Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
> (Which expires Sep 30 14:01:15 2021 GMT and was issued just a year
> earlier - ridiculous for a root CA)
>
>
> The components/ovms_tls/trustedca/isrg_x1.crt is:
>
> Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
> Validity
> Not Before: Jun 4 11:04:38 2015 GMT
> Not After : Jun 4 11:04:38 2035 GMT
> sha256WithRSAEncryption 4096 bit
>
>
> Our problem could be either the wolfssl not recognising the X1 cert,
> or not supporting the cross-signing arrangement (where there are two
> paths to verify - the expired X3 and the provided X1). I suspect the
> latter. In an ideal world, having both X1 and X3 trusted shouldn’t be
> a problem.
>
> There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0
> released February 2021). Perhaps we can try to update to that?
>
> Mark
>
>> On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter at expeedo.de
>> <mailto:dexter at expeedo.de>> wrote:
>>
>> Signed PGP part
>> I can now confirm it's a WolfSSL issue :-(
>>
>> I've switched back to release 3.2.016, i.e. before changing to
>> WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it
>> should.
>>
>> Steve, I remember you included a config option to enable using
>> WolfSSL, but cannot find it now. Can you give me a pointer, or did
>> you remove that option later on?
>>
>>
>>
>> Am 28.09.21 um 21:53 schrieb Stephen Casner:
>>> I wonder if we are hitting a key size limit in the WolfSSL code. I
>>> know there are configuration parameters in user_settings.h for
>>> different SHA sizes, but I don't recall one for RSA keys.
>>>
>>> -- Steve
>>>
>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>
>>>> Still no luck, and getting clueless... :-/
>>>>
>>>> Let's Encrypt have a live test server:
>>>> https://valid-isrgrootx1.letsencrypt.org/
>>>>
>>>> Here's a simple test script to access that server:
>>>>
>>>> (function(){
>>>> HTTP.request({
>>>> url:"https://valid-isrgrootx1.letsencrypt.org/",
>>>> always: function() { JSON.print(this); }
>>>> });
>>>> })();
>>>>
>>>> Just copy this into the editor and evaluate.
>>>>
>>>> After commenting out the DST certificate from the TLS source, the http request
>>>> terminates with an SSL error (-3) with installed...
>>>>
>>>> * self-signed ISRG certificate
>>>> * cross-signed ISRG / DST root certificate
>>>> * LE R3 certificate
>>>> * any combination of these three.
>>>>
>>>> But the access works immediately after reinstalling the old DST root
>>>> certificate. There is no DST reference in the chain of that server...
>>>>
>>>>
>>>> I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT
>>>> lib --; apparently all they needed to do was to add the very same self-signed
>>>> ISRG X1 certificate I added:
>>>>
>>>> *
>>>> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5
>>>>
>>>>
>>>> So: why would WolfSSL not use the supplied ISRG certificate? Why would it use
>>>> the DST cert without DST being present as an issuer?
>>>>
>>>> The ISRG certificate is the only one having...
>>>>
>>>> signed using : RSA with SHA-256
>>>> RSA key size : 4096 bits
>>>>
>>>> But that would surely be supported by WolfSSL, wouldn't it?
>>>>
>>>> I'm running out of ideas, and this really isn't my primary area. Any help is
>>>> appreciated.
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>>
>>>> Am 28.09.21 um 19:15 schrieb Stephen Casner:
>>>>> Michael,
>>>>>
>>>>> Certainly within our code as it stands there is not any mechanism to
>>>>> tell the WolfSSL code to adjust its certificate validation procedure.
>>>>> The browser certificate substitution that you hypothesize is not clear
>>>>> to me. I would expect the validation to simply follow the chain.
>>>>>
>>>>> -- Steve
>>>>>
>>>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>>>
>>>>>> More info:
>>>>>>
>>>>>> All my browsers already have a builtin ISRG X1 certificate signed by ISRG
>>>>>> only, that's the new version:
>>>>>>
>>>>>> https://crt.sh/?id=9314791
>>>>>>
>>>>>> My server still sends the ISRG X1 certificate cross signed / issued by DST
>>>>>> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on
>>>>>> the
>>>>>> last renewal (last month!):
>>>>>>
>>>>>> https://crt.sh/?id=3958242236
>>>>>>
>>>>>> Without the DST root cert, WolfSSL then fails validating the DST signed X1
>>>>>> root certificate (I assume):
>>>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/
>>>>>>
>>>>>> My servers will continue sending that chain including the outdated root
>>>>>> cert
>>>>>> probably until the next renewal, so it's possible having added the new X1
>>>>>> root
>>>>>> certificate didn't solve the issue.
>>>>>>
>>>>>> The browsers seem to know how to substitute the DST signed certificate by
>>>>>> the
>>>>>> builtin self-signed (?). Is there a similar option in WolfSSL, and do we
>>>>>> need
>>>>>> to enable that?
>>>>>>
>>>>>> Steve, can you confirm this, do you know a solution?
>>>>>>
>>>>>> Regards,
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> Am 28.09.21 um 15:34 schrieb Michael Balzer:
>>>>>>> I've tried adding the intermediate cert ("R3") and then also my site
>>>>>>> certificate, that didn't help.
>>>>>>>
>>>>>>> Only adding the DST cert again fixes the connection.
>>>>>>>
>>>>>>> Any ideas?
>>>>>>>
>>>>>>>
>>>>>>> OVMS# tls trust list
>>>>>>> ...
>>>>>>> ISRG Root X1 length 1939 bytes
>>>>>>> 1939 byte certificate: ISRG Root X1
>>>>>>> cert. version : 3
>>>>>>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
>>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>> Root
>>>>>>> X1
>>>>>>> subject name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>> Root
>>>>>>> X1
>>>>>>> issued on : 2015-06-04 11:04:38
>>>>>>> expires on : 2035-06-04 11:04:38
>>>>>>> signed using : RSA with SHA-256
>>>>>>> RSA key size : 4096 bits
>>>>>>> basic constraints : CA=true
>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>> ...
>>>>>>> dexter length 1972 bytes
>>>>>>> 1972 byte certificate: dexter
>>>>>>> cert. version : 3
>>>>>>> serial number :
>>>>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
>>>>>>> issuer name : C=US, O=Let's Encrypt, CN=R3
>>>>>>> subject name : CN=dexter.shopdriver.de <http://dexter.shopdriver.de>
>>>>>>> issued on : 2021-08-07 05:47:57
>>>>>>> expires on : 2021-11-05 05:47:55
>>>>>>> signed using : RSA with SHA-256
>>>>>>> RSA key size : 2048 bits
>>>>>>> basic constraints : CA=false
>>>>>>> subject alt name :dexter.shopdriver.de <http://dexter.shopdriver.de>,dexters-web.de <http://dexters-web.de>,
>>>>>>> ovms.dexters-web.de <http://ovms.dexters-web.de>,www.dexter.shopdriver.de,www.dexters-web.de
>>>>>>> key usage : Digital Signature, Key Encipherment
>>>>>>> ext key usage : TLS Web Server Authentication, TLS Web Client
>>>>>>> Authentication
>>>>>>> ...
>>>>>>> r3 length 1826 bytes
>>>>>>> 1826 byte certificate: r3
>>>>>>> cert. version : 3
>>>>>>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
>>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>>>> Root
>>>>>>> X1
>>>>>>> subject name : C=US, O=Let's Encrypt, CN=R3
>>>>>>> issued on : 2020-09-04 00:00:00
>>>>>>> expires on : 2025-09-15 16:00:00
>>>>>>> signed using : RSA with SHA-256
>>>>>>> RSA key size : 2048 bits
>>>>>>> basic constraints : CA=true, max_pathlen=0
>>>>>>> key usage : Digital Signature, Key Cert Sign, CRL Sign
>>>>>>> ext key usage : TLS Web Client Authentication, TLS Web Server
>>>>>>> Authentication
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer:
>>>>>>>> We would need to bypass / shortcut the "eap" test phase.
>>>>>>>>
>>>>>>>> But I agree, "master" is stable, I haven't had any issues or reports,
>>>>>>>> so I
>>>>>>>> think we could do that. The FreeRTOS timer issue I'm working on only
>>>>>>>> affects very specific conditions, so not necessary to wait for that.
>>>>>>>>
>>>>>>>> Should we remove the expiring DST certificate in that release then?
>>>>>>>>
>>>>>>>> ...uh oh: just tried removing the DST certificate: the module cannot
>>>>>>>> connect to my server anymore...!?
>>>>>>>>
>>>>>>>> I (490213) ovms-server-v2: Connection isovms.dexters-web.de:6870 <http://ovms.dexters-web.de:6870>
>>>>>>>> TEST1
>>>>>>>> I (490213) ovms-server-v2: Status: Connecting...
>>>>>>>> V (490723) ovms-server-v2:
>>>>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
>>>>>>>> W (490723) ovms-server-v2: Connection failed
>>>>>>>> E (490723) ovms-server-v2: Status: Error: Connection failed
>>>>>>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE)
>>>>>>>> I (490723) ovms-server-v2: Status: Disconnected
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:
>>>>>>>>> Shall we release a full update? The last 3.2?
>>>>>>>>>
>>>>>>>>> What we have now in master seems stable.
>>>>>>>>>
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer<dexter at expeedo.de>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Everyone,
>>>>>>>>>>
>>>>>>>>>> the DST root certificate we include (DST Root CA X3) expires on
>>>>>>>>>> September 30, i.e. in two days.
>>>>>>>>>>
>>>>>>>>>> OVMS# tls trust list
>>>>>>>>>> DST Root CA X3 length 1200 bytes
>>>>>>>>>> 1200 byte certificate: DST Root CA X3
>>>>>>>>>> cert. version : 3
>>>>>>>>>> serial number :
>>>>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
>>>>>>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>>> CA X3
>>>>>>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>>>> CA X3
>>>>>>>>>> issued on : 2000-09-30 21:12:19
>>>>>>>>>> * expires on : 2021-09-30 14:01:15*
>>>>>>>>>> signed using : RSA with SHA1
>>>>>>>>>> RSA key size : 2048 bits
>>>>>>>>>> basic constraints : CA=true
>>>>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>>>>>
>>>>>>>>>> AFAICT, this root certificate is currently used by the OVMS to
>>>>>>>>>> validate Let's Encrypt certificates.
>>>>>>>>>>
>>>>>>>>>> *
>>>>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
>>>>>>>>>> *https://letsencrypt.org/docs/certificate-compatibility/
>>>>>>>>>>
>>>>>>>>>> Unfortunately, we missed adding the followup LE root certificate
>>>>>>>>>> "ISRG
>>>>>>>>>> Root X1" in time.
>>>>>>>>>>
>>>>>>>>>> I've just added that certificate to our builtin certificate
>>>>>>>>>> repository, but it's too late now to roll out a "main" update in
>>>>>>>>>> time
>>>>>>>>>> (isn't it?).
>>>>>>>>>>
>>>>>>>>>> So, to prevent losing TLS connectivity with LE servers, users need
>>>>>>>>>> to
>>>>>>>>>> manually add the ISRG Root X1 certificate to their TLS
>>>>>>>>>> repositories.
>>>>>>>>>>
>>>>>>>>>> I've added a section on this to our user manual:
>>>>>>>>>>
>>>>>>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html
>>>>>>>>>>
>>>>>>>>>> If users contact you, point them to that page.
>>>>>>>>>>
>>>>>>>>>> We probably should also remove the expired DST root certificate
>>>>>>>>>> after
>>>>>>>>>> September 30.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Michael
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OvmsDev mailing list
>>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>>> _______________________________________________
>>>>>>>>> OvmsDev mailing list
>>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>>> --
>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OvmsDev mailing list
>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>> --
>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OvmsDev mailing list
>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>> --
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>
>> --
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/848b90ae/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210929/848b90ae/attachment-0001.sig>
More information about the OvmsDev
mailing list