[Ovmsdev] Urgent TLS root certificate issue (Let's Encrypt)
Michael Balzer
dexter at expeedo.de
Wed Sep 29 04:11:18 HKT 2021
I can now confirm it's a WolfSSL issue :-(
I've switched back to release 3.2.016, i.e. before changing to WolfSSL,
and the ISRG Root X1 certificate works perfectly, just as it should.
Steve, I remember you included a config option to enable using WolfSSL,
but cannot find it now. Can you give me a pointer, or did you remove
that option later on?
Am 28.09.21 um 21:53 schrieb Stephen Casner:
> I wonder if we are hitting a key size limit in the WolfSSL code. I
> know there are configuration parameters in user_settings.h for
> different SHA sizes, but I don't recall one for RSA keys.
>
> -- Steve
>
> On Tue, 28 Sep 2021, Michael Balzer wrote:
>
>> Still no luck, and getting clueless... :-/
>>
>> Let's Encrypt have a live test server:
>> https://valid-isrgrootx1.letsencrypt.org/
>>
>> Here's a simple test script to access that server:
>>
>> (function(){
>> HTTP.request({
>> url: "https://valid-isrgrootx1.letsencrypt.org/",
>> always: function() { JSON.print(this); }
>> });
>> })();
>>
>> Just copy this into the editor and evaluate.
>>
>> After commenting out the DST certificate from the TLS source, the http request
>> terminates with an SSL error (-3) with installed...
>>
>> * self-signed ISRG certificate
>> * cross-signed ISRG / DST root certificate
>> * LE R3 certificate
>> * any combination of these three.
>>
>> But the access works immediately after reinstalling the old DST root
>> certificate. There is no DST reference in the chain of that server...
>>
>>
>> I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT
>> lib --; apparently all they needed to do was to add the very same self-signed
>> ISRG X1 certificate I added:
>>
>> *
>> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5
>>
>>
>> So: why would WolfSSL not use the supplied ISRG certificate? Why would it use
>> the DST cert without DST being present as an issuer?
>>
>> The ISRG certificate is the only one having...
>>
>> signed using : RSA with SHA-256
>> RSA key size : 4096 bits
>>
>> But that would surely be supported by WolfSSL, wouldn't it?
>>
>> I'm running out of ideas, and this really isn't my primary area. Any help is
>> appreciated.
>>
>> Regards,
>> Michael
>>
>>
>> Am 28.09.21 um 19:15 schrieb Stephen Casner:
>>> Michael,
>>>
>>> Certainly within our code as it stands there is not any mechanism to
>>> tell the WolfSSL code to adjust its certificate validation procedure.
>>> The browser certificate substitution that you hypothesize is not clear
>>> to me. I would expect the validation to simply follow the chain.
>>>
>>> -- Steve
>>>
>>> On Tue, 28 Sep 2021, Michael Balzer wrote:
>>>
>>>> More info:
>>>>
>>>> All my browsers already have a builtin ISRG X1 certificate signed by ISRG
>>>> only, that's the new version:
>>>>
>>>> https://crt.sh/?id=9314791
>>>>
>>>> My server still sends the ISRG X1 certificate cross signed / issued by DST
>>>> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on
>>>> the
>>>> last renewal (last month!):
>>>>
>>>> https://crt.sh/?id=3958242236
>>>>
>>>> Without the DST root cert, WolfSSL then fails validating the DST signed X1
>>>> root certificate (I assume):
>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/
>>>>
>>>> My servers will continue sending that chain including the outdated root
>>>> cert
>>>> probably until the next renewal, so it's possible having added the new X1
>>>> root
>>>> certificate didn't solve the issue.
>>>>
>>>> The browsers seem to know how to substitute the DST signed certificate by
>>>> the
>>>> builtin self-signed (?). Is there a similar option in WolfSSL, and do we
>>>> need
>>>> to enable that?
>>>>
>>>> Steve, can you confirm this, do you know a solution?
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>>
>>>> Am 28.09.21 um 15:34 schrieb Michael Balzer:
>>>>> I've tried adding the intermediate cert ("R3") and then also my site
>>>>> certificate, that didn't help.
>>>>>
>>>>> Only adding the DST cert again fixes the connection.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>>> OVMS# tls trust list
>>>>> ...
>>>>> ISRG Root X1 length 1939 bytes
>>>>> 1939 byte certificate: ISRG Root X1
>>>>> cert. version : 3
>>>>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>> Root
>>>>> X1
>>>>> subject name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>> Root
>>>>> X1
>>>>> issued on : 2015-06-04 11:04:38
>>>>> expires on : 2035-06-04 11:04:38
>>>>> signed using : RSA with SHA-256
>>>>> RSA key size : 4096 bits
>>>>> basic constraints : CA=true
>>>>> key usage : Key Cert Sign, CRL Sign
>>>>> ...
>>>>> dexter length 1972 bytes
>>>>> 1972 byte certificate: dexter
>>>>> cert. version : 3
>>>>> serial number :
>>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
>>>>> issuer name : C=US, O=Let's Encrypt, CN=R3
>>>>> subject name : CN=dexter.shopdriver.de
>>>>> issued on : 2021-08-07 05:47:57
>>>>> expires on : 2021-11-05 05:47:55
>>>>> signed using : RSA with SHA-256
>>>>> RSA key size : 2048 bits
>>>>> basic constraints : CA=false
>>>>> subject alt name : dexter.shopdriver.de, dexters-web.de,
>>>>> ovms.dexters-web.de, www.dexter.shopdriver.de, www.dexters-web.de
>>>>> key usage : Digital Signature, Key Encipherment
>>>>> ext key usage : TLS Web Server Authentication, TLS Web Client
>>>>> Authentication
>>>>> ...
>>>>> r3 length 1826 bytes
>>>>> 1826 byte certificate: r3
>>>>> cert. version : 3
>>>>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG
>>>>> Root
>>>>> X1
>>>>> subject name : C=US, O=Let's Encrypt, CN=R3
>>>>> issued on : 2020-09-04 00:00:00
>>>>> expires on : 2025-09-15 16:00:00
>>>>> signed using : RSA with SHA-256
>>>>> RSA key size : 2048 bits
>>>>> basic constraints : CA=true, max_pathlen=0
>>>>> key usage : Digital Signature, Key Cert Sign, CRL Sign
>>>>> ext key usage : TLS Web Client Authentication, TLS Web Server
>>>>> Authentication
>>>>>
>>>>>
>>>>>
>>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer:
>>>>>> We would need to bypass / shortcut the "eap" test phase.
>>>>>>
>>>>>> But I agree, "master" is stable, I haven't had any issues or reports,
>>>>>> so I
>>>>>> think we could do that. The FreeRTOS timer issue I'm working on only
>>>>>> affects very specific conditions, so not necessary to wait for that.
>>>>>>
>>>>>> Should we remove the expiring DST certificate in that release then?
>>>>>>
>>>>>> ...uh oh: just tried removing the DST certificate: the module cannot
>>>>>> connect to my server anymore...!?
>>>>>>
>>>>>> I (490213) ovms-server-v2: Connection is ovms.dexters-web.de:6870
>>>>>> TEST1
>>>>>> I (490213) ovms-server-v2: Status: Connecting...
>>>>>> V (490723) ovms-server-v2:
>>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
>>>>>> W (490723) ovms-server-v2: Connection failed
>>>>>> E (490723) ovms-server-v2: Status: Error: Connection failed
>>>>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE)
>>>>>> I (490723) ovms-server-v2: Status: Disconnected
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:
>>>>>>> Shall we release a full update? The last 3.2?
>>>>>>>
>>>>>>> What we have now in master seems stable.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer <dexter at expeedo.de>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Everyone,
>>>>>>>>
>>>>>>>> the DST root certificate we include (DST Root CA X3) expires on
>>>>>>>> September 30, i.e. in two days.
>>>>>>>>
>>>>>>>> OVMS# tls trust list
>>>>>>>> DST Root CA X3 length 1200 bytes
>>>>>>>> 1200 byte certificate: DST Root CA X3
>>>>>>>> cert. version : 3
>>>>>>>> serial number :
>>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
>>>>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>> CA X3
>>>>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root
>>>>>>>> CA X3
>>>>>>>> issued on : 2000-09-30 21:12:19
>>>>>>>> * expires on : 2021-09-30 14:01:15*
>>>>>>>> signed using : RSA with SHA1
>>>>>>>> RSA key size : 2048 bits
>>>>>>>> basic constraints : CA=true
>>>>>>>> key usage : Key Cert Sign, CRL Sign
>>>>>>>>
>>>>>>>> AFAICT, this root certificate is currently used by the OVMS to
>>>>>>>> validate Let's Encrypt certificates.
>>>>>>>>
>>>>>>>> *
>>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
>>>>>>>> * https://letsencrypt.org/docs/certificate-compatibility/
>>>>>>>>
>>>>>>>> Unfortunately, we missed adding the followup LE root certificate
>>>>>>>> "ISRG
>>>>>>>> Root X1" in time.
>>>>>>>>
>>>>>>>> I've just added that certificate to our builtin certificate
>>>>>>>> repository, but it's too late now to roll out a "main" update in
>>>>>>>> time
>>>>>>>> (isn't it?).
>>>>>>>>
>>>>>>>> So, to prevent losing TLS connectivity with LE servers, users need
>>>>>>>> to
>>>>>>>> manually add the ISRG Root X1 certificate to their TLS
>>>>>>>> repositories.
>>>>>>>>
>>>>>>>> I've added a section on this to our user manual:
>>>>>>>>
>>>>>>>> * https://docs.openvehicles.com/en/latest/userguide/ssltls.html
>>>>>>>>
>>>>>>>> If users contact you, point them to that page.
>>>>>>>>
>>>>>>>> We probably should also remove the expired DST root certificate
>>>>>>>> after
>>>>>>>> September 30.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Michael
>>>>>>>>
>>>>>>>> --
>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>> _______________________________________________
>>>>>>>> OvmsDev mailing list
>>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>>> _______________________________________________
>>>>>>> OvmsDev mailing list
>>>>>>> OvmsDev at lists.openvehicles.com
>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>> --
>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>> --
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>> --
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210928/ba1fb063/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20210928/ba1fb063/attachment-0001.sig>
More information about the OvmsDev
mailing list