We would need to bypass / shortcut the "eap" test phase.

But I agree, "master" is stable, I haven't had any issues or reports, so I think we could do that. The FreeRTOS timer issue I'm working on only affects very specific conditions, so not necessary to wait for that.

Should we remove the expiring DST certificate in that release then?

…uh oh: just tried removing the DST certificate: the module cannot connect to my server anymore…!?

I (490213) ovms-server-v2: Connection is ovms.dexters-web.de:6870 TEST1 I (490213) ovms-server-v2: Status: Connecting... V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) W (490723) ovms-server-v2: Connection failed E (490723) ovms-server-v2: Status: Error: Connection failed V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) I (490723) ovms-server-v2: Status: Disconnected

Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:

Shall we release a full update? The last 3.2?

What we have now in master seems stable.

Mark

...

On 28 Sep 2021, at 5:39 PM, Michael Balzer <dexter@expeedo.de> wrote:

 Everyone,

the DST root certificate we include (DST Root CA X3) expires on September 30, i.e. in two days.

OVMS# tls trust list DST Root CA X3 length 1200 bytes 1200 byte certificate: DST Root CA X3 cert. version     : 3 serial number     : 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B issuer name       : O=Digital Signature Trust Co., CN=DST Root CA X3 subject name      : O=Digital Signature Trust Co., CN=DST Root CA X3 issued  on        : 2000-09-30 21:12:19 *  expires on        : 2021-09-30 14:01:15* signed using      : RSA with SHA1 RSA key size      : 2048 bits basic constraints : CA=true key usage         : Key Cert Sign, CRL Sign

AFAICT, this root certificate is currently used by the OVMS to validate Let's Encrypt certificates.

• https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
• https://letsencrypt.org/docs/certificate-compatibility/

Unfortunately, we missed adding the followup LE root certificate "ISRG Root X1" in time.

I've just added that certificate to our builtin certificate repository, but it's too late now to roll out a "main" update in time (isn't it?).

So, to prevent losing TLS connectivity with LE servers, users need to manually add the ISRG Root X1 certificate to their TLS repositories.

I've added a section on this to our user manual:

• https://docs.openvehicles.com/en/latest/userguide/ssltls.html

If users contact you, point them to that page.

We probably should also remove the expired DST root certificate after September 30.

Regards, Michael

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

More info:

All my browsers already have a builtin ISRG X1 certificate signed by ISRG only, that's the new version:

https://crt.sh/?id=9314791

My server still sends the ISRG X1 certificate cross signed / issued by DST Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on the last renewal (last month!):

https://crt.sh/?id=3958242236

Without the DST root cert, WolfSSL then fails validating the DST signed X1 root certificate (I assume): https://www.wolfssl.com/docs/wolfssl-manual/ch7/

My servers will continue sending that chain including the outdated root cert probably until the next renewal, so it's possible having added the new X1 root certificate didn't solve the issue.

The browsers seem to know how to substitute the DST signed certificate by the builtin self-signed (?). Is there a similar option in WolfSSL, and do we need to enable that?

Steve, can you confirm this, do you know a solution?

Regards, Michael

Am 28.09.21 um 15:34 schrieb Michael Balzer:

I've tried adding the intermediate cert ("R3") and then also my site certificate, that didn't help.

Only adding the DST cert again fixes the connection.

Any ideas?

OVMS# tls trust list … ISRG Root X1 length 1939 bytes 1939 byte certificate: ISRG Root X1 cert. version     : 3 serial number     : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 issuer name       : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name      : C=US, O=Internet Security Research Group, CN=ISRG Root X1 issued  on        : 2015-06-04 11:04:38 expires on        : 2035-06-04 11:04:38 signed using      : RSA with SHA-256 RSA key size      : 4096 bits basic constraints : CA=true key usage         : Key Cert Sign, CRL Sign … dexter length 1972 bytes 1972 byte certificate: dexter cert. version     : 3 serial number     : 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 issuer name       : C=US, O=Let's Encrypt, CN=R3 subject name      : CN=dexter.shopdriver.de issued  on        : 2021-08-07 05:47:57 expires on        : 2021-11-05 05:47:55 signed using      : RSA with SHA-256 RSA key size      : 2048 bits basic constraints : CA=false subject alt name  : dexter.shopdriver.de, dexters-web.de, ovms.dexters-web.de, www.dexter.shopdriver.de, www.dexters-web.de key usage         : Digital Signature, Key Encipherment ext key usage     : TLS Web Server Authentication, TLS Web Client Authentication … r3 length 1826 bytes 1826 byte certificate: r3 cert. version     : 3 serial number     : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A issuer name       : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name      : C=US, O=Let's Encrypt, CN=R3 issued  on        : 2020-09-04 00:00:00 expires on        : 2025-09-15 16:00:00 signed using      : RSA with SHA-256 RSA key size      : 2048 bits basic constraints : CA=true, max_pathlen=0 key usage         : Digital Signature, Key Cert Sign, CRL Sign ext key usage     : TLS Web Client Authentication, TLS Web Server Authentication

Am 28.09.21 um 15:07 schrieb Michael Balzer:

...

We would need to bypass / shortcut the "eap" test phase.

But I agree, "master" is stable, I haven't had any issues or reports, so I think we could do that. The FreeRTOS timer issue I'm working on only affects very specific conditions, so not necessary to wait for that.

Should we remove the expiring DST certificate in that release then?

…uh oh: just tried removing the DST certificate: the module cannot connect to my server anymore…!?

I (490213) ovms-server-v2: Connection is ovms.dexters-web.de:6870 TEST1 I (490213) ovms-server-v2: Status: Connecting... V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) W (490723) ovms-server-v2: Connection failed E (490723) ovms-server-v2: Status: Error: Connection failed V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) I (490723) ovms-server-v2: Status: Disconnected

Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:

...

Shall we release a full update? The last 3.2?

What we have now in master seems stable.

Mark

...

On 28 Sep 2021, at 5:39 PM, Michael Balzer <dexter@expeedo.de> wrote:

 Everyone,

the DST root certificate we include (DST Root CA X3) expires on September 30, i.e. in two days.

OVMS# tls trust list DST Root CA X3 length 1200 bytes 1200 byte certificate: DST Root CA X3 cert. version     : 3 serial number     : 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B issuer name       : O=Digital Signature Trust Co., CN=DST Root CA X3 subject name      : O=Digital Signature Trust Co., CN=DST Root CA X3 issued  on        : 2000-09-30 21:12:19 *  expires on        : 2021-09-30 14:01:15* signed using      : RSA with SHA1 RSA key size      : 2048 bits basic constraints : CA=true key usage         : Key Cert Sign, CRL Sign

AFAICT, this root certificate is currently used by the OVMS to validate Let's Encrypt certificates.

• https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
• https://letsencrypt.org/docs/certificate-compatibility/

Unfortunately, we missed adding the followup LE root certificate "ISRG Root X1" in time.

I've just added that certificate to our builtin certificate repository, but it's too late now to roll out a "main" update in time (isn't it?).

So, to prevent losing TLS connectivity with LE servers, users need to manually add the ISRG Root X1 certificate to their TLS repositories.

I've added a section on this to our user manual:

• https://docs.openvehicles.com/en/latest/userguide/ssltls.html

If users contact you, point them to that page.

We probably should also remove the expired DST root certificate after September 30.

Regards, Michael

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Still no luck, and getting clueless… :-/

Let's Encrypt have a live test server: https://valid-isrgrootx1.letsencrypt.org/

Here's a simple test script to access that server:

(function(){ HTTP.request({ url: "https://valid-isrgrootx1.letsencrypt.org/", always: function() { JSON.print(this); } }); })();

Just copy this into the editor and evaluate.

After commenting out the DST certificate from the TLS source, the http request terminates with an SSL error (-3) with installed…

• self-signed ISRG certificate
• cross-signed ISRG / DST root certificate
• LE R3 certificate
• any combination of these three.

But the access works immediately after reinstalling the old DST root certificate. There is no DST reference in the chain of that server…

I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT lib --; apparently all they needed to do was to add the very same self-signed ISRG X1 certificate I added:

• https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118...

So: why would WolfSSL not use the supplied ISRG certificate? Why would it use the DST cert without DST being present as an issuer?

The ISRG certificate is the only one having…

  signed using      : RSA with SHA-256 RSA key size      : 4096 bits

But that would surely be supported by WolfSSL, wouldn't it?

I'm running out of ideas, and this really isn't my primary area. Any help is appreciated.

Regards, Michael

Am 28.09.21 um 19:15 schrieb Stephen Casner:

Michael,

Certainly within our code as it stands there is not any mechanism to tell the WolfSSL code to adjust its certificate validation procedure. The browser certificate substitution that you hypothesize is not clear to me. I would expect the validation to simply follow the chain.

                                                     -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

More info:

All my browsers already have a builtin ISRG X1 certificate signed by ISRG only, that's the new version:

https://crt.sh/?id=9314791

My server still sends the ISRG X1 certificate cross signed / issued by DST Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on the last renewal (last month!):

https://crt.sh/?id=3958242236

Without the DST root cert, WolfSSL then fails validating the DST signed X1 root certificate (I assume): https://www.wolfssl.com/docs/wolfssl-manual/ch7/

My servers will continue sending that chain including the outdated root cert probably until the next renewal, so it's possible having added the new X1 root certificate didn't solve the issue.

The browsers seem to know how to substitute the DST signed certificate by the builtin self-signed (?). Is there a similar option in WolfSSL, and do we need to enable that?

Steve, can you confirm this, do you know a solution?

Regards, Michael

Am 28.09.21 um 15:34 schrieb Michael Balzer:

...

I've tried adding the intermediate cert ("R3") and then also my site certificate, that didn't help.

Only adding the DST cert again fixes the connection.

Any ideas?

OVMS# tls trust list ... ISRG Root X1 length 1939 bytes 1939 byte certificate: ISRG Root X1 cert. version : 3 serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 issuer name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 issued on : 2015-06-04 11:04:38 expires on : 2035-06-04 11:04:38 signed using : RSA with SHA-256 RSA key size : 4096 bits basic constraints : CA=true key usage : Key Cert Sign, CRL Sign ... dexter length 1972 bytes 1972 byte certificate: dexter cert. version : 3 serial number : 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 issuer name : C=US, O=Let's Encrypt, CN=R3 subject name : CN=dexter.shopdriver.de issued on : 2021-08-07 05:47:57 expires on : 2021-11-05 05:47:55 signed using : RSA with SHA-256 RSA key size : 2048 bits basic constraints : CA=false subject alt name : dexter.shopdriver.de, dexters-web.de, ovms.dexters-web.de, www.dexter.shopdriver.de, www.dexters-web.de key usage : Digital Signature, Key Encipherment ext key usage : TLS Web Server Authentication, TLS Web Client Authentication ... r3 length 1826 bytes 1826 byte certificate: r3 cert. version : 3 serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A issuer name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name : C=US, O=Let's Encrypt, CN=R3 issued on : 2020-09-04 00:00:00 expires on : 2025-09-15 16:00:00 signed using : RSA with SHA-256 RSA key size : 2048 bits basic constraints : CA=true, max_pathlen=0 key usage : Digital Signature, Key Cert Sign, CRL Sign ext key usage : TLS Web Client Authentication, TLS Web Server Authentication

Am 28.09.21 um 15:07 schrieb Michael Balzer:

...

We would need to bypass / shortcut the "eap" test phase.

But I agree, "master" is stable, I haven't had any issues or reports, so I think we could do that. The FreeRTOS timer issue I'm working on only affects very specific conditions, so not necessary to wait for that.

Should we remove the expiring DST certificate in that release then?

...uh oh: just tried removing the DST certificate: the module cannot connect to my server anymore...!?

I (490213) ovms-server-v2: Connection is ovms.dexters-web.de:6870 TEST1 I (490213) ovms-server-v2: Status: Connecting... V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) W (490723) ovms-server-v2: Connection failed E (490723) ovms-server-v2: Status: Error: Connection failed V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) I (490723) ovms-server-v2: Status: Disconnected

Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:

...

Shall we release a full update? The last 3.2?

What we have now in master seems stable.

Mark

...

On 28 Sep 2021, at 5:39 PM, Michael Balzer <dexter@expeedo.de> wrote:

Everyone,

the DST root certificate we include (DST Root CA X3) expires on September 30, i.e. in two days.

OVMS# tls trust list DST Root CA X3 length 1200 bytes 1200 byte certificate: DST Root CA X3 cert. version : 3 serial number : 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B issuer name : O=Digital Signature Trust Co., CN=DST Root CA X3 subject name : O=Digital Signature Trust Co., CN=DST Root CA X3 issued on : 2000-09-30 21:12:19

• expires on : 2021-09-30 14:01:15* signed using : RSA with SHA1 RSA key size : 2048 bits basic constraints : CA=true key usage : Key Cert Sign, CRL Sign

AFAICT, this root certificate is currently used by the OVMS to validate Let's Encrypt certificates.

• https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
• https://letsencrypt.org/docs/certificate-compatibility/

Unfortunately, we missed adding the followup LE root certificate "ISRG Root X1" in time.

I've just added that certificate to our builtin certificate repository, but it's too late now to roll out a "main" update in time (isn't it?).

So, to prevent losing TLS connectivity with LE servers, users need to manually add the ISRG Root X1 certificate to their TLS repositories.

I've added a section on this to our user manual:

• https://docs.openvehicles.com/en/latest/userguide/ssltls.html

If users contact you, point them to that page.

We probably should also remove the expired DST root certificate after September 30.

Regards, Michael

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

I can now confirm it's a WolfSSL issue :-(

I've switched back to release 3.2.016, i.e. before changing to WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it should.

Steve, I remember you included a config option to enable using WolfSSL, but cannot find it now. Can you give me a pointer, or did you remove that option later on?

Am 28.09.21 um 21:53 schrieb Stephen Casner:

I wonder if we are hitting a key size limit in the WolfSSL code. I know there are configuration parameters in user_settings.h for different SHA sizes, but I don't recall one for RSA keys.

                                                     -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

Still no luck, and getting clueless... :-/

Let's Encrypt have a live test server: https://valid-isrgrootx1.letsencrypt.org/

Here's a simple test script to access that server:

(function(){ HTTP.request({ url: "https://valid-isrgrootx1.letsencrypt.org/", always: function() { JSON.print(this); } }); })();

Just copy this into the editor and evaluate.

After commenting out the DST certificate from the TLS source, the http request terminates with an SSL error (-3) with installed...

• self-signed ISRG certificate
• cross-signed ISRG / DST root certificate
• LE R3 certificate
• any combination of these three.

But the access works immediately after reinstalling the old DST root certificate. There is no DST reference in the chain of that server...

I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT lib --; apparently all they needed to do was to add the very same self-signed ISRG X1 certificate I added:

• https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118...

So: why would WolfSSL not use the supplied ISRG certificate? Why would it use the DST cert without DST being present as an issuer?

The ISRG certificate is the only one having...

signed using : RSA with SHA-256 RSA key size : 4096 bits

But that would surely be supported by WolfSSL, wouldn't it?

I'm running out of ideas, and this really isn't my primary area. Any help is appreciated.

Regards, Michael

Am 28.09.21 um 19:15 schrieb Stephen Casner:

...

Michael,

Certainly within our code as it stands there is not any mechanism to tell the WolfSSL code to adjust its certificate validation procedure. The browser certificate substitution that you hypothesize is not clear to me. I would expect the validation to simply follow the chain.

                                                      -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

More info:

All my browsers already have a builtin ISRG X1 certificate signed by ISRG only, that's the new version:

https://crt.sh/?id=9314791

My server still sends the ISRG X1 certificate cross signed / issued by DST Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on the last renewal (last month!):

https://crt.sh/?id=3958242236

Without the DST root cert, WolfSSL then fails validating the DST signed X1 root certificate (I assume): https://www.wolfssl.com/docs/wolfssl-manual/ch7/

My servers will continue sending that chain including the outdated root cert probably until the next renewal, so it's possible having added the new X1 root certificate didn't solve the issue.

The browsers seem to know how to substitute the DST signed certificate by the builtin self-signed (?). Is there a similar option in WolfSSL, and do we need to enable that?

Steve, can you confirm this, do you know a solution?

Regards, Michael

Am 28.09.21 um 15:34 schrieb Michael Balzer:

...

I've tried adding the intermediate cert ("R3") and then also my site certificate, that didn't help.

Only adding the DST cert again fixes the connection.

Any ideas?

OVMS# tls trust list ... ISRG Root X1 length 1939 bytes 1939 byte certificate: ISRG Root X1 cert. version : 3 serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 issuer name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 issued on : 2015-06-04 11:04:38 expires on : 2035-06-04 11:04:38 signed using : RSA with SHA-256 RSA key size : 4096 bits basic constraints : CA=true key usage : Key Cert Sign, CRL Sign ... dexter length 1972 bytes 1972 byte certificate: dexter cert. version : 3 serial number : 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 issuer name : C=US, O=Let's Encrypt, CN=R3 subject name : CN=dexter.shopdriver.de issued on : 2021-08-07 05:47:57 expires on : 2021-11-05 05:47:55 signed using : RSA with SHA-256 RSA key size : 2048 bits basic constraints : CA=false subject alt name : dexter.shopdriver.de, dexters-web.de, ovms.dexters-web.de, www.dexter.shopdriver.de, www.dexters-web.de key usage : Digital Signature, Key Encipherment ext key usage : TLS Web Server Authentication, TLS Web Client Authentication ... r3 length 1826 bytes 1826 byte certificate: r3 cert. version : 3 serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A issuer name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name : C=US, O=Let's Encrypt, CN=R3 issued on : 2020-09-04 00:00:00 expires on : 2025-09-15 16:00:00 signed using : RSA with SHA-256 RSA key size : 2048 bits basic constraints : CA=true, max_pathlen=0 key usage : Digital Signature, Key Cert Sign, CRL Sign ext key usage : TLS Web Client Authentication, TLS Web Server Authentication

Am 28.09.21 um 15:07 schrieb Michael Balzer:

...

We would need to bypass / shortcut the "eap" test phase.

But I agree, "master" is stable, I haven't had any issues or reports, so I think we could do that. The FreeRTOS timer issue I'm working on only affects very specific conditions, so not necessary to wait for that.

Should we remove the expiring DST certificate in that release then?

...uh oh: just tried removing the DST certificate: the module cannot connect to my server anymore...!?

I (490213) ovms-server-v2: Connection is ovms.dexters-web.de:6870 TEST1 I (490213) ovms-server-v2: Status: Connecting... V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) W (490723) ovms-server-v2: Connection failed E (490723) ovms-server-v2: Status: Error: Connection failed V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) I (490723) ovms-server-v2: Status: Disconnected

Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: > Shall we release a full update? The last 3.2? > > What we have now in master seems stable. > > Mark > >> On 28 Sep 2021, at 5:39 PM, Michael Balzer <dexter@expeedo.de> >> wrote: >> >> Everyone, >> >> the DST root certificate we include (DST Root CA X3) expires on >> September 30, i.e. in two days. >> >> OVMS# tls trust list >> DST Root CA X3 length 1200 bytes >> 1200 byte certificate: DST Root CA X3 >> cert. version : 3 >> serial number : >> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >> issuer name : O=Digital Signature Trust Co., CN=DST Root >> CA X3 >> subject name : O=Digital Signature Trust Co., CN=DST Root >> CA X3 >> issued on : 2000-09-30 21:12:19 >> * expires on : 2021-09-30 14:01:15* >> signed using : RSA with SHA1 >> RSA key size : 2048 bits >> basic constraints : CA=true >> key usage : Key Cert Sign, CRL Sign >> >> AFAICT, this root certificate is currently used by the OVMS to >> validate Let's Encrypt certificates. >> >> * >> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ >> * https://letsencrypt.org/docs/certificate-compatibility/ >> >> Unfortunately, we missed adding the followup LE root certificate >> "ISRG >> Root X1" in time. >> >> I've just added that certificate to our builtin certificate >> repository, but it's too late now to roll out a "main" update in >> time >> (isn't it?). >> >> So, to prevent losing TLS connectivity with LE servers, users need >> to >> manually add the ISRG Root X1 certificate to their TLS >> repositories. >> >> I've added a section on this to our user manual: >> >> * https://docs.openvehicles.com/en/latest/userguide/ssltls.html >> >> If users contact you, point them to that page. >> >> We probably should also remove the expired DST root certificate >> after >> September 30. >> >> Regards, >> Michael >> >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com > http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

It seems the dexter server returns different certificates depending on the presence of SNI extension in the TSL negotiation. That is pretty normal (where a single IP is serving multiple different websites), but I am not sure if our client library supports that?

Without SNI, we get:

Serial Number: 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54 Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/> X509v3 Subject Alternative Name: DNS:ns34.expeedo.de

With SNI dexters-web.de <http://dexters-web.de/>, we get:

Serial Number: 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0 Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/> X509v3 Subject Alternative Name: DNS:dexter.shopdriver.de, DNS:dexters-web.de, DNS:ovms.dexters-web.de, DNS:www.dexter.shopdriver.de, DNS:www.dexters-web.de

Assuming the SNI is set correctly by our library, looking at the dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:

Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/> Issuer: C=US, O=Let's Encrypt, CN=R3

Subject: C=US, O=Let's Encrypt, CN=R3 Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1 Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3

and the components/ovms_tls/trustedca/dst.crt we have is:

Subject: O=Digital Signature Trust Co., CN=DST Root CA X3 Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 (Which expires Sep 30 14:01:15 2021 GMT and was issued just a year earlier - ridiculous for a root CA)

The components/ovms_tls/trustedca/isrg_x1.crt is:

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1 Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1 Validity Not Before: Jun 4 11:04:38 2015 GMT Not After : Jun 4 11:04:38 2035 GMT sha256WithRSAEncryption 4096 bit

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.

There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released February 2021). Perhaps we can try to update to that?

Mark

On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de> wrote:

Signed PGP part I can now confirm it's a WolfSSL issue :-(

I've switched back to release 3.2.016, i.e. before changing to WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it should.

Steve, I remember you included a config option to enable using WolfSSL, but cannot find it now. Can you give me a pointer, or did you remove that option later on?

Am 28.09.21 um 21:53 schrieb Stephen Casner:

...

I wonder if we are hitting a key size limit in the WolfSSL code. I know there are configuration parameters in user_settings.h for different SHA sizes, but I don't recall one for RSA keys.

                                                    -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

Still no luck, and getting clueless... :-/

Let's Encrypt have a live test server: https://valid-isrgrootx1.letsencrypt.org/ <https://valid-isrgrootx1.letsencrypt.org/>

Here's a simple test script to access that server:

(function(){ HTTP.request({ url: "https://valid-isrgrootx1.letsencrypt.org/" <https://valid-isrgrootx1.letsencrypt.org/>, always: function() { JSON.print(this); } }); })();

Just copy this into the editor and evaluate.

After commenting out the DST certificate from the TLS source, the http request terminates with an SSL error (-3) with installed...

• self-signed ISRG certificate
• cross-signed ISRG / DST root certificate
• LE R3 certificate
• any combination of these three.

But the access works immediately after reinstalling the old DST root certificate. There is no DST reference in the chain of that server...

I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT lib --; apparently all they needed to do was to add the very same self-signed ISRG X1 certificate I added:

• https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118... <https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5>

So: why would WolfSSL not use the supplied ISRG certificate? Why would it use the DST cert without DST being present as an issuer?

The ISRG certificate is the only one having...

signed using : RSA with SHA-256 RSA key size : 4096 bits

But that would surely be supported by WolfSSL, wouldn't it?

I'm running out of ideas, and this really isn't my primary area. Any help is appreciated.

Regards, Michael

Am 28.09.21 um 19:15 schrieb Stephen Casner:

...

Michael,

Certainly within our code as it stands there is not any mechanism to tell the WolfSSL code to adjust its certificate validation procedure. The browser certificate substitution that you hypothesize is not clear to me. I would expect the validation to simply follow the chain.

                                                     -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

More info:

All my browsers already have a builtin ISRG X1 certificate signed by ISRG only, that's the new version:

https://crt.sh/?id=9314791 <https://crt.sh/?id=9314791>

My server still sends the ISRG X1 certificate cross signed / issued by DST Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on the last renewal (last month!):

https://crt.sh/?id=3958242236 <https://crt.sh/?id=3958242236>

Without the DST root cert, WolfSSL then fails validating the DST signed X1 root certificate (I assume): https://www.wolfssl.com/docs/wolfssl-manual/ch7/ <https://www.wolfssl.com/docs/wolfssl-manual/ch7/>

My servers will continue sending that chain including the outdated root cert probably until the next renewal, so it's possible having added the new X1 root certificate didn't solve the issue.

The browsers seem to know how to substitute the DST signed certificate by the builtin self-signed (?). Is there a similar option in WolfSSL, and do we need to enable that?

Steve, can you confirm this, do you know a solution?

Regards, Michael

Am 28.09.21 um 15:34 schrieb Michael Balzer:

...

I've tried adding the intermediate cert ("R3") and then also my site certificate, that didn't help.

Only adding the DST cert again fixes the connection.

Any ideas?

OVMS# tls trust list ... ISRG Root X1 length 1939 bytes 1939 byte certificate: ISRG Root X1 cert. version : 3 serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 issuer name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 issued on : 2015-06-04 11:04:38 expires on : 2035-06-04 11:04:38 signed using : RSA with SHA-256 RSA key size : 4096 bits basic constraints : CA=true key usage : Key Cert Sign, CRL Sign ... dexter length 1972 bytes 1972 byte certificate: dexter cert. version : 3 serial number : 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 issuer name : C=US, O=Let's Encrypt, CN=R3 subject name : CN=dexter.shopdriver.de issued on : 2021-08-07 05:47:57 expires on : 2021-11-05 05:47:55 signed using : RSA with SHA-256 RSA key size : 2048 bits basic constraints : CA=false subject alt name : dexter.shopdriver.de, dexters-web.de, ovms.dexters-web.de, www.dexter.shopdriver.de <http://www.dexter.shopdriver.de/>, www.dexters-web.de <http://www.dexters-web.de/> key usage : Digital Signature, Key Encipherment ext key usage : TLS Web Server Authentication, TLS Web Client Authentication ... r3 length 1826 bytes 1826 byte certificate: r3 cert. version : 3 serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A issuer name : C=US, O=Internet Security Research Group, CN=ISRG Root X1 subject name : C=US, O=Let's Encrypt, CN=R3 issued on : 2020-09-04 00:00:00 expires on : 2025-09-15 16:00:00 signed using : RSA with SHA-256 RSA key size : 2048 bits basic constraints : CA=true, max_pathlen=0 key usage : Digital Signature, Key Cert Sign, CRL Sign ext key usage : TLS Web Client Authentication, TLS Web Server Authentication

Am 28.09.21 um 15:07 schrieb Michael Balzer: > We would need to bypass / shortcut the "eap" test phase. > > But I agree, "master" is stable, I haven't had any issues or reports, > so I > think we could do that. The FreeRTOS timer issue I'm working on only > affects very specific conditions, so not necessary to wait for that. > > Should we remove the expiring DST certificate in that release then? > > ...uh oh: just tried removing the DST certificate: the module cannot > connect to my server anymore...!? > > I (490213) ovms-server-v2: Connection is ovms.dexters-web.de:6870 > TEST1 > I (490213) ovms-server-v2: Status: Connecting... > V (490723) ovms-server-v2: > OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) > W (490723) ovms-server-v2: Connection failed > E (490723) ovms-server-v2: Status: Error: Connection failed > V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) > I (490723) ovms-server-v2: Status: Disconnected > > > > > Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: >> Shall we release a full update? The last 3.2? >> >> What we have now in master seems stable. >> >> Mark >> >>> On 28 Sep 2021, at 5:39 PM, Michael Balzer <dexter@expeedo.de> <mailto:dexter@expeedo.de> >>> wrote: >>> >>> Everyone, >>> >>> the DST root certificate we include (DST Root CA X3) expires on >>> September 30, i.e. in two days. >>> >>> OVMS# tls trust list >>> DST Root CA X3 length 1200 bytes >>> 1200 byte certificate: DST Root CA X3 >>> cert. version : 3 >>> serial number : >>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >>> issuer name : O=Digital Signature Trust Co., CN=DST Root >>> CA X3 >>> subject name : O=Digital Signature Trust Co., CN=DST Root >>> CA X3 >>> issued on : 2000-09-30 21:12:19 >>> * expires on : 2021-09-30 14:01:15* >>> signed using : RSA with SHA1 >>> RSA key size : 2048 bits >>> basic constraints : CA=true >>> key usage : Key Cert Sign, CRL Sign >>> >>> AFAICT, this root certificate is currently used by the OVMS to >>> validate Let's Encrypt certificates. >>> >>> * >>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ <https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/> >>> * https://letsencrypt.org/docs/certificate-compatibility/ <https://letsencrypt.org/docs/certificate-compatibility/> >>> >>> Unfortunately, we missed adding the followup LE root certificate >>> "ISRG >>> Root X1" in time. >>> >>> I've just added that certificate to our builtin certificate >>> repository, but it's too late now to roll out a "main" update in >>> time >>> (isn't it?). >>> >>> So, to prevent losing TLS connectivity with LE servers, users need >>> to >>> manually add the ISRG Root X1 certificate to their TLS >>> repositories. >>> >>> I've added a section on this to our user manual: >>> >>> * https://docs.openvehicles.com/en/latest/userguide/ssltls.html <https://docs.openvehicles.com/en/latest/userguide/ssltls.html> >>> >>> If users contact you, point them to that page. >>> >>> We probably should also remove the expired DST root certificate >>> after >>> September 30. >>> >>> Regards, >>> Michael >>> >>> -- >>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev> > -- > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 > > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

SNI was pretty much intact already, I took care of a remaining Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).

I understand the WolfSSL chain management uses a supplied root cert over one delivered by the server: https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.

Is cross signing supported at all by WolfSSL? The only place I've found it mentioned on wolfssl is this page: https://www.wolfssl.com/certificate-chain-chain-trust/ …where they say "(We’ll cover cross-signing below)" … but I don't see them doing that.

There is also no commit in their git history regarding cross-signed certificates or multiple signatures, so I don't think updating to 4.8.1 would help.

The intermediate "R3" has issuer "C=US, O=Internet Security Research Group, CN=ISRG Root X1", which should match the new ISRG certificate, but not the DST certificate. Maybe that's identified by a fingerprint in the signature though (?), and maybe the DST signature comes first and WolfSSL just doesn't check further signatures if the first fails?

I'll look into the code later on, but need to take care of my daily job duties first.

If we can't fix this today, I think our best option is to switch back to mbedTLS for the final 3.2 release and try to fix this for the initial 3.3 release.

Regards, Michael

Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:

It seems the dexter server returns different certificates depending on the presence of SNI extension in the TSL negotiation. That is pretty normal (where a single IP is serving multiple different websites), but I am not sure if our client library supports that?

Without SNI, we get:

Serial Number:
04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de>
 X509v3 Subject Alternative Name:
                DNS:ns34.expeedo.de <http://ns34.expeedo.de>

With SNI dexters-web.de <http://dexters-web.de>, we get:

Serial Number:
03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de>
X509v3 Subject Alternative Name:
                DNS:dexter.shopdriver.de
<http://dexter.shopdriver.de>, DNS:dexters-web.de
<http://dexters-web.de>, DNS:ovms.dexters-web.de
<http://ovms.dexters-web.de>, DNS:www.dexter.shopdriver.de
<http://www.dexter.shopdriver.de>, DNS:www.dexters-web.de
<http://www.dexters-web.de>

Assuming the SNI is set correctly by our library, looking at the dexter.shopdriver.de <http://dexter.shopdriver.de> chain, we have:

Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de>
Issuer: C=US, O=Let's Encrypt, CN=R3

Subject: C=US, O=Let's Encrypt, CN=R3
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3

and the components/ovms_tls/trustedca/dst.crt we have is:

Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
(Which expires Sep 30 14:01:15 2021 GMT and was issued just a year
earlier - ridiculous for a root CA)

The components/ovms_tls/trustedca/isrg_x1.crt is:

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Validity
            Not Before: Jun  4 11:04:38 2015 GMT
            Not After : Jun  4 11:04:38 2035 GMT
sha256WithRSAEncryption 4096 bit

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.

There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released February 2021). Perhaps we can try to update to that?

Mark

...

On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part I can now confirm it's a WolfSSL issue :-(

I've switched back to release 3.2.016, i.e. before changing to WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it should.

Steve, I remember you included a config option to enable using WolfSSL, but cannot find it now. Can you give me a pointer, or did you remove that option later on?

Am 28.09.21 um 21:53 schrieb Stephen Casner:

...

I wonder if we are hitting a key size limit in the WolfSSL code. I know there are configuration parameters in user_settings.h for different SHA sizes, but I don't recall one for RSA keys.

                                                     -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

Still no luck, and getting clueless... :-/

Let's Encrypt have a live test server: https://valid-isrgrootx1.letsencrypt.org/

Here's a simple test script to access that server:

(function(){ HTTP.request({ url:"https://valid-isrgrootx1.letsencrypt.org/", always: function() { JSON.print(this); } }); })();

Just copy this into the editor and evaluate.

After commenting out the DST certificate from the TLS source, the http request terminates with an SSL error (-3) with installed...

• self-signed ISRG certificate
• cross-signed ISRG / DST root certificate
• LE R3 certificate
• any combination of these three.

But the access works immediately after reinstalling the old DST root certificate. There is no DST reference in the chain of that server...

I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT lib --; apparently all they needed to do was to add the very same self-signed ISRG X1 certificate I added:

• https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118...

So: why would WolfSSL not use the supplied ISRG certificate? Why would it use the DST cert without DST being present as an issuer?

The ISRG certificate is the only one having...

signed using : RSA with SHA-256 RSA key size : 4096 bits

But that would surely be supported by WolfSSL, wouldn't it?

I'm running out of ideas, and this really isn't my primary area. Any help is appreciated.

Regards, Michael

Am 28.09.21 um 19:15 schrieb Stephen Casner:

...

Michael,

Certainly within our code as it stands there is not any mechanism to tell the WolfSSL code to adjust its certificate validation procedure. The browser certificate substitution that you hypothesize is not clear to me. I would expect the validation to simply follow the chain.

                                                      -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

More info:

All my browsers already have a builtin ISRG X1 certificate signed by ISRG only, that's the new version:

https://crt.sh/?id=9314791

My server still sends the ISRG X1 certificate cross signed / issued by DST Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on the last renewal (last month!):

https://crt.sh/?id=3958242236

Without the DST root cert, WolfSSL then fails validating the DST signed X1 root certificate (I assume): https://www.wolfssl.com/docs/wolfssl-manual/ch7/

My servers will continue sending that chain including the outdated root cert probably until the next renewal, so it's possible having added the new X1 root certificate didn't solve the issue.

The browsers seem to know how to substitute the DST signed certificate by the builtin self-signed (?). Is there a similar option in WolfSSL, and do we need to enable that?

Steve, can you confirm this, do you know a solution?

Regards, Michael

Am 28.09.21 um 15:34 schrieb Michael Balzer: > I've tried adding the intermediate cert ("R3") and then also my site > certificate, that didn't help. > > Only adding the DST cert again fixes the connection. > > Any ideas? > > > OVMS# tls trust list > ... > ISRG Root X1 length 1939 bytes > 1939 byte certificate: ISRG Root X1 > cert. version : 3 > serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 > issuer name : C=US, O=Internet Security Research Group, CN=ISRG > Root > X1 > subject name : C=US, O=Internet Security Research Group, CN=ISRG > Root > X1 > issued on : 2015-06-04 11:04:38 > expires on : 2035-06-04 11:04:38 > signed using : RSA with SHA-256 > RSA key size : 4096 bits > basic constraints : CA=true > key usage : Key Cert Sign, CRL Sign > ... > dexter length 1972 bytes > 1972 byte certificate: dexter > cert. version : 3 > serial number : > 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 > issuer name : C=US, O=Let's Encrypt, CN=R3 > subject name : CN=dexter.shopdriver.de <http://dexter.shopdriver.de> > issued on : 2021-08-07 05:47:57 > expires on : 2021-11-05 05:47:55 > signed using : RSA with SHA-256 > RSA key size : 2048 bits > basic constraints : CA=false > subject alt name :dexter.shopdriver.de <http://dexter.shopdriver.de>,dexters-web.de <http://dexters-web.de>, > ovms.dexters-web.de <http://ovms.dexters-web.de>,www.dexter.shopdriver.de,www.dexters-web.de > key usage : Digital Signature, Key Encipherment > ext key usage : TLS Web Server Authentication, TLS Web Client > Authentication > ... > r3 length 1826 bytes > 1826 byte certificate: r3 > cert. version : 3 > serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A > issuer name : C=US, O=Internet Security Research Group, CN=ISRG > Root > X1 > subject name : C=US, O=Let's Encrypt, CN=R3 > issued on : 2020-09-04 00:00:00 > expires on : 2025-09-15 16:00:00 > signed using : RSA with SHA-256 > RSA key size : 2048 bits > basic constraints : CA=true, max_pathlen=0 > key usage : Digital Signature, Key Cert Sign, CRL Sign > ext key usage : TLS Web Client Authentication, TLS Web Server > Authentication > > > > Am 28.09.21 um 15:07 schrieb Michael Balzer: >> We would need to bypass / shortcut the "eap" test phase. >> >> But I agree, "master" is stable, I haven't had any issues or reports, >> so I >> think we could do that. The FreeRTOS timer issue I'm working on only >> affects very specific conditions, so not necessary to wait for that. >> >> Should we remove the expiring DST certificate in that release then? >> >> ...uh oh: just tried removing the DST certificate: the module cannot >> connect to my server anymore...!? >> >> I (490213) ovms-server-v2: Connection isovms.dexters-web.de:6870 <http://ovms.dexters-web.de:6870> >> TEST1 >> I (490213) ovms-server-v2: Status: Connecting... >> V (490723) ovms-server-v2: >> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) >> W (490723) ovms-server-v2: Connection failed >> E (490723) ovms-server-v2: Status: Error: Connection failed >> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) >> I (490723) ovms-server-v2: Status: Disconnected >> >> >> >> >> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: >>> Shall we release a full update? The last 3.2? >>> >>> What we have now in master seems stable. >>> >>> Mark >>> >>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer<dexter@expeedo.de> >>>> wrote: >>>> >>>> Everyone, >>>> >>>> the DST root certificate we include (DST Root CA X3) expires on >>>> September 30, i.e. in two days. >>>> >>>> OVMS# tls trust list >>>> DST Root CA X3 length 1200 bytes >>>> 1200 byte certificate: DST Root CA X3 >>>> cert. version : 3 >>>> serial number : >>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >>>> issuer name : O=Digital Signature Trust Co., CN=DST Root >>>> CA X3 >>>> subject name : O=Digital Signature Trust Co., CN=DST Root >>>> CA X3 >>>> issued on : 2000-09-30 21:12:19 >>>> * expires on : 2021-09-30 14:01:15* >>>> signed using : RSA with SHA1 >>>> RSA key size : 2048 bits >>>> basic constraints : CA=true >>>> key usage : Key Cert Sign, CRL Sign >>>> >>>> AFAICT, this root certificate is currently used by the OVMS to >>>> validate Let's Encrypt certificates. >>>> >>>> * >>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ >>>> *https://letsencrypt.org/docs/certificate-compatibility/ >>>> >>>> Unfortunately, we missed adding the followup LE root certificate >>>> "ISRG >>>> Root X1" in time. >>>> >>>> I've just added that certificate to our builtin certificate >>>> repository, but it's too late now to roll out a "main" update in >>>> time >>>> (isn't it?). >>>> >>>> So, to prevent losing TLS connectivity with LE servers, users need >>>> to >>>> manually add the ISRG Root X1 certificate to their TLS >>>> repositories. >>>> >>>> I've added a section on this to our user manual: >>>> >>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html >>>> >>>> If users contact you, point them to that page. >>>> >>>> We probably should also remove the expired DST root certificate >>>> after >>>> September 30. >>>> >>>> Regards, >>>> Michael >>>> >>>> -- >>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev > -- > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 > > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com > http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

I managed to get wolfSSL debug logs, but don't know how to interpret them. See files attached.

The final error seems to be…

I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188 I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188 I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188 E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818 file:/home/balzer/esp/Open-Vehic E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507 file:/home/balzer/esp/Open-Vehic

which is ASN_NO_SIGNER_E     = -188,  /* ASN no signer to confirm failure */

I've had a look at the code, I don't see a chance to find the bug within the remaining time.

We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL commits now. Steve, if you already prepared something in that direction, tell me.

Regards, Michael

Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:

The cross-signing bit seems to be on the X1 cert provided in the certificate chain by the server. The one provided there is signed by X3. The problem comes because a different X1 certificate may be provided as a trusted CA (and that X1 is self-signed with a long expiration date - 2035). So the verification process has two possible paths to verify the cert, and the result depends on which one it chooses (and how it handles a trusted CA cert supplementing / replacing one provided by the server).

Overall, a mess. I’m fighting this same issue now with a bunch of systems in my day job, and likely a large part of the Internet is going to start to fail tomorrow. Particularly a lot of small embedded devices.

Regards, Mark.

...

On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part SNI was pretty much intact already, I took care of a remaining Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).

I understand the WolfSSL chain management uses a supplied root cert over one delivered by the server: https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3

...

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.

Is cross signing supported at all by WolfSSL? The only place I've found it mentioned on wolfssl is this page: https://www.wolfssl.com/certificate-chain-chain-trust/ …where they say "(We’ll cover cross-signing below)" … but I don't see them doing that.

There is also no commit in their git history regarding cross-signed certificates or multiple signatures, so I don't think updating to 4.8.1 would help.

The intermediate "R3" has issuer "C=US, O=Internet Security Research Group, CN=ISRG Root X1", which should match the new ISRG certificate, but not the DST certificate. Maybe that's identified by a fingerprint in the signature though (?), and maybe the DST signature comes first and WolfSSL just doesn't check further signatures if the first fails?

I'll look into the code later on, but need to take care of my daily job duties first.

If we can't fix this today, I think our best option is to switch back to mbedTLS for the final 3.2 release and try to fix this for the initial 3.3 release.

Regards, Michael

Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:

...

It seems the dexter server returns different certificates depending on the presence of SNI extension in the TSL negotiation. That is pretty normal (where a single IP is serving multiple different websites), but I am not sure if our client library supports that?

Without SNI, we get:

Serial Number:
04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
 X509v3 Subject Alternative Name:
                DNS:ns34.expeedo.de <http://ns34.expeedo.de/>

With SNI dexters-web.de <http://dexters-web.de/>, we get:

Serial Number:
03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
X509v3 Subject Alternative Name:
                DNS:dexter.shopdriver.de
<http://dexter.shopdriver.de/>, DNS:dexters-web.de
<http://dexters-web.de/>, DNS:ovms.dexters-web.de
<http://ovms.dexters-web.de/>, DNS:www.dexter.shopdriver.de
<http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de
<http://www.dexters-web.de/>

Assuming the SNI is set correctly by our library, looking at the dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:

Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
Issuer: C=US, O=Let's Encrypt, CN=R3

Subject: C=US, O=Let's Encrypt, CN=R3
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3

and the components/ovms_tls/trustedca/dst.crt we have is:

Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
(Which expires Sep 30 14:01:15 2021 GMT and was issued just a
year earlier - ridiculous for a root CA)

The components/ovms_tls/trustedca/isrg_x1.crt is:

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Validity
            Not Before: Jun  4 11:04:38 2015 GMT
            Not After : Jun  4 11:04:38 2035 GMT
sha256WithRSAEncryption 4096 bit

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.

There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released February 2021). Perhaps we can try to update to that?

Mark

...

On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part I can now confirm it's a WolfSSL issue :-(

I've switched back to release 3.2.016, i.e. before changing to WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it should.

Steve, I remember you included a config option to enable using WolfSSL, but cannot find it now. Can you give me a pointer, or did you remove that option later on?

Am 28.09.21 um 21:53 schrieb Stephen Casner:

...

I wonder if we are hitting a key size limit in the WolfSSL code. I know there are configuration parameters in user_settings.h for different SHA sizes, but I don't recall one for RSA keys.

                                                     -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

...

Still no luck, and getting clueless... :-/

Let's Encrypt have a live test server: https://valid-isrgrootx1.letsencrypt.org/

Here's a simple test script to access that server:

(function(){ HTTP.request({ url:"https://valid-isrgrootx1.letsencrypt.org/", always: function() { JSON.print(this); } }); })();

Just copy this into the editor and evaluate.

After commenting out the DST certificate from the TLS source, the http request terminates with an SSL error (-3) with installed...

• self-signed ISRG certificate
• cross-signed ISRG / DST root certificate
• LE R3 certificate
• any combination of these three.

But the access works immediately after reinstalling the old DST root certificate. There is no DST reference in the chain of that server...

I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT lib --; apparently all they needed to do was to add the very same self-signed ISRG X1 certificate I added:

• https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118...

So: why would WolfSSL not use the supplied ISRG certificate? Why would it use the DST cert without DST being present as an issuer?

The ISRG certificate is the only one having...

signed using : RSA with SHA-256 RSA key size : 4096 bits

But that would surely be supported by WolfSSL, wouldn't it?

I'm running out of ideas, and this really isn't my primary area. Any help is appreciated.

Regards, Michael

Am 28.09.21 um 19:15 schrieb Stephen Casner: > Michael, > > Certainly within our code as it stands there is not any mechanism to > tell the WolfSSL code to adjust its certificate validation procedure. > The browser certificate substitution that you hypothesize is not clear > to me. I would expect the validation to simply follow the chain. > > -- Steve > > On Tue, 28 Sep 2021, Michael Balzer wrote: > >> More info: >> >> All my browsers already have a builtin ISRG X1 certificate signed by ISRG >> only, that's the new version: >> >> https://crt.sh/?id=9314791 >> >> My server still sends the ISRG X1 certificate cross signed / issued by DST >> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on >> the >> last renewal (last month!): >> >> https://crt.sh/?id=3958242236 >> >> Without the DST root cert, WolfSSL then fails validating the DST signed X1 >> root certificate (I assume): >> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ >> >> My servers will continue sending that chain including the outdated root >> cert >> probably until the next renewal, so it's possible having added the new X1 >> root >> certificate didn't solve the issue. >> >> The browsers seem to know how to substitute the DST signed certificate by >> the >> builtin self-signed (?). Is there a similar option in WolfSSL, and do we >> need >> to enable that? >> >> Steve, can you confirm this, do you know a solution? >> >> Regards, >> Michael >> >> >> Am 28.09.21 um 15:34 schrieb Michael Balzer: >>> I've tried adding the intermediate cert ("R3") and then also my site >>> certificate, that didn't help. >>> >>> Only adding the DST cert again fixes the connection. >>> >>> Any ideas? >>> >>> >>> OVMS# tls trust list >>> ... >>> ISRG Root X1 length 1939 bytes >>> 1939 byte certificate: ISRG Root X1 >>> cert. version : 3 >>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 >>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG >>> Root >>> X1 >>> subject name : C=US, O=Internet Security Research Group, CN=ISRG >>> Root >>> X1 >>> issued on : 2015-06-04 11:04:38 >>> expires on : 2035-06-04 11:04:38 >>> signed using : RSA with SHA-256 >>> RSA key size : 4096 bits >>> basic constraints : CA=true >>> key usage : Key Cert Sign, CRL Sign >>> ... >>> dexter length 1972 bytes >>> 1972 byte certificate: dexter >>> cert. version : 3 >>> serial number : >>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 >>> issuer name : C=US, O=Let's Encrypt, CN=R3 >>> subject name : CN=dexter.shopdriver.de <http://dexter.shopdriver.de/> >>> issued on : 2021-08-07 05:47:57 >>> expires on : 2021-11-05 05:47:55 >>> signed using : RSA with SHA-256 >>> RSA key size : 2048 bits >>> basic constraints : CA=false >>> subject alt name :dexter.shopdriver.de <http://dexter.shopdriver.de/>,dexters-web.de <http://dexters-web.de/>, >>> ovms.dexters-web.de <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de >>> key usage : Digital Signature, Key Encipherment >>> ext key usage : TLS Web Server Authentication, TLS Web Client >>> Authentication >>> ... >>> r3 length 1826 bytes >>> 1826 byte certificate: r3 >>> cert. version : 3 >>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A >>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG >>> Root >>> X1 >>> subject name : C=US, O=Let's Encrypt, CN=R3 >>> issued on : 2020-09-04 00:00:00 >>> expires on : 2025-09-15 16:00:00 >>> signed using : RSA with SHA-256 >>> RSA key size : 2048 bits >>> basic constraints : CA=true, max_pathlen=0 >>> key usage : Digital Signature, Key Cert Sign, CRL Sign >>> ext key usage : TLS Web Client Authentication, TLS Web Server >>> Authentication >>> >>> >>> >>> Am 28.09.21 um 15:07 schrieb Michael Balzer: >>>> We would need to bypass / shortcut the "eap" test phase. >>>> >>>> But I agree, "master" is stable, I haven't had any issues or reports, >>>> so I >>>> think we could do that. The FreeRTOS timer issue I'm working on only >>>> affects very specific conditions, so not necessary to wait for that. >>>> >>>> Should we remove the expiring DST certificate in that release then? >>>> >>>> ...uh oh: just tried removing the DST certificate: the module cannot >>>> connect to my server anymore...!? >>>> >>>> I (490213) ovms-server-v2: Connection isovms.dexters-web.de:6870 <http://ovms.dexters-web.de:6870/> >>>> TEST1 >>>> I (490213) ovms-server-v2: Status: Connecting... >>>> V (490723) ovms-server-v2: >>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) >>>> W (490723) ovms-server-v2: Connection failed >>>> E (490723) ovms-server-v2: Status: Error: Connection failed >>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) >>>> I (490723) ovms-server-v2: Status: Disconnected >>>> >>>> >>>> >>>> >>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: >>>>> Shall we release a full update? The last 3.2? >>>>> >>>>> What we have now in master seems stable. >>>>> >>>>> Mark >>>>> >>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer<dexter@expeedo.de> >>>>>> wrote: >>>>>> >>>>>> Everyone, >>>>>> >>>>>> the DST root certificate we include (DST Root CA X3) expires on >>>>>> September 30, i.e. in two days. >>>>>> >>>>>> OVMS# tls trust list >>>>>> DST Root CA X3 length 1200 bytes >>>>>> 1200 byte certificate: DST Root CA X3 >>>>>> cert. version : 3 >>>>>> serial number : >>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root >>>>>> CA X3 >>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root >>>>>> CA X3 >>>>>> issued on : 2000-09-30 21:12:19 >>>>>> * expires on : 2021-09-30 14:01:15* >>>>>> signed using : RSA with SHA1 >>>>>> RSA key size : 2048 bits >>>>>> basic constraints : CA=true >>>>>> key usage : Key Cert Sign, CRL Sign >>>>>> >>>>>> AFAICT, this root certificate is currently used by the OVMS to >>>>>> validate Let's Encrypt certificates. >>>>>> >>>>>> * >>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ >>>>>> *https://letsencrypt.org/docs/certificate-compatibility/ >>>>>> >>>>>> Unfortunately, we missed adding the followup LE root certificate >>>>>> "ISRG >>>>>> Root X1" in time. >>>>>> >>>>>> I've just added that certificate to our builtin certificate >>>>>> repository, but it's too late now to roll out a "main" update in >>>>>> time >>>>>> (isn't it?). >>>>>> >>>>>> So, to prevent losing TLS connectivity with LE servers, users need >>>>>> to >>>>>> manually add the ISRG Root X1 certificate to their TLS >>>>>> repositories. >>>>>> >>>>>> I've added a section on this to our user manual: >>>>>> >>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html >>>>>> >>>>>> If users contact you, point them to that page. >>>>>> >>>>>> We probably should also remove the expired DST root certificate >>>>>> after >>>>>> September 30. >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> -- >>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>> _______________________________________________ >>>>>> OvmsDev mailing list >>>>>> OvmsDev@lists.openvehicles.com >>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>> -- >>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>> >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>> -- >>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>> >>> _______________________________________________ >>> OvmsDev mailing list >>> OvmsDev@lists.openvehicles.com >>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Patch attached. Additionally I had set the boot debug level (early) to info (may no longer be necessary as I later found that LOG_LOCAL_LEVEL #define in esp32-crypt.h), and you need to disable log monitoring in the USB console to prevent garbled output.

I tried WOLFSSL_ALT_CERT_CHAINS but only got crashes with that.

Regards, Michael

Am 29.09.21 um 14:44 schrieb Mark Webb-Johnson:

Can you send me the patch to enable debugging, and I will try with 3.3 branch. I agree that rolling back 3.2 to mbedtls seems sensible for the short term.

Regards, Mark

P.S. I managed to fix this on another project running an old OpenSSL by removing the X3 trusted ca (it had both x1 and x3 trusted). Behavior is highly dependent on the library treatment of multiple verification paths and trusted ca vs cert chain conflicts.

...

On 29 Sep 2021, at 8:11 PM, Michael Balzer <dexter@expeedo.de> wrote:

 I managed to get wolfSSL debug logs, but don't know how to interpret them. See files attached.

The final error seems to be…

I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188 I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188 I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188 E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818 file:/home/balzer/esp/Open-Vehic E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507 file:/home/balzer/esp/Open-Vehic

which is ASN_NO_SIGNER_E     = -188,  /* ASN no signer to confirm failure */

I've had a look at the code, I don't see a chance to find the bug within the remaining time.

We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL commits now. Steve, if you already prepared something in that direction, tell me.

Regards, Michael

Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:

...

The cross-signing bit seems to be on the X1 cert provided in the certificate chain by the server. The one provided there is signed by X3. The problem comes because a different X1 certificate may be provided as a trusted CA (and that X1 is self-signed with a long expiration date - 2035). So the verification process has two possible paths to verify the cert, and the result depends on which one it chooses (and how it handles a trusted CA cert supplementing / replacing one provided by the server).

Overall, a mess. I’m fighting this same issue now with a bunch of systems in my day job, and likely a large part of the Internet is going to start to fail tomorrow. Particularly a lot of small embedded devices.

Regards, Mark.

...

On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part SNI was pretty much intact already, I took care of a remaining Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).

I understand the WolfSSL chain management uses a supplied root cert over one delivered by the server: https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3

...

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.

Is cross signing supported at all by WolfSSL? The only place I've found it mentioned on wolfssl is this page: https://www.wolfssl.com/certificate-chain-chain-trust/ …where they say "(We’ll cover cross-signing below)" … but I don't see them doing that.

There is also no commit in their git history regarding cross-signed certificates or multiple signatures, so I don't think updating to 4.8.1 would help.

The intermediate "R3" has issuer "C=US, O=Internet Security Research Group, CN=ISRG Root X1", which should match the new ISRG certificate, but not the DST certificate. Maybe that's identified by a fingerprint in the signature though (?), and maybe the DST signature comes first and WolfSSL just doesn't check further signatures if the first fails?

I'll look into the code later on, but need to take care of my daily job duties first.

If we can't fix this today, I think our best option is to switch back to mbedTLS for the final 3.2 release and try to fix this for the initial 3.3 release.

Regards, Michael

Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:

...

It seems the dexter server returns different certificates depending on the presence of SNI extension in the TSL negotiation. That is pretty normal (where a single IP is serving multiple different websites), but I am not sure if our client library supports that?

Without SNI, we get:

Serial Number:
04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
 X509v3 Subject Alternative Name:
                DNS:ns34.expeedo.de <http://ns34.expeedo.de/>

With SNI dexters-web.de <http://dexters-web.de/>, we get:

Serial Number:
03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
X509v3 Subject Alternative Name:
                DNS:dexter.shopdriver.de
<http://dexter.shopdriver.de/>, DNS:dexters-web.de
<http://dexters-web.de/>, DNS:ovms.dexters-web.de
<http://ovms.dexters-web.de/>, DNS:www.dexter.shopdriver.de
<http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de
<http://www.dexters-web.de/>

Assuming the SNI is set correctly by our library, looking at the dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:

Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
Issuer: C=US, O=Let's Encrypt, CN=R3

Subject: C=US, O=Let's Encrypt, CN=R3
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3

and the components/ovms_tls/trustedca/dst.crt we have is:

Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
(Which expires Sep 30 14:01:15 2021 GMT and was issued just a
year earlier - ridiculous for a root CA)

The components/ovms_tls/trustedca/isrg_x1.crt is:

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Validity
            Not Before: Jun  4 11:04:38 2015 GMT
            Not After : Jun  4 11:04:38 2035 GMT
sha256WithRSAEncryption 4096 bit

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn’t be a problem.

There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released February 2021). Perhaps we can try to update to that?

Mark

...

On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part I can now confirm it's a WolfSSL issue :-(

I've switched back to release 3.2.016, i.e. before changing to WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it should.

Steve, I remember you included a config option to enable using WolfSSL, but cannot find it now. Can you give me a pointer, or did you remove that option later on?

Am 28.09.21 um 21:53 schrieb Stephen Casner: > I wonder if we are hitting a key size limit in the WolfSSL code. I > know there are configuration parameters in user_settings.h for > different SHA sizes, but I don't recall one for RSA keys. > > -- Steve > > On Tue, 28 Sep 2021, Michael Balzer wrote: > >> Still no luck, and getting clueless... :-/ >> >> Let's Encrypt have a live test server: >> https://valid-isrgrootx1.letsencrypt.org/ >> >> Here's a simple test script to access that server: >> >> (function(){ >> HTTP.request({ >> url:"https://valid-isrgrootx1.letsencrypt.org/", >> always: function() { JSON.print(this); } >> }); >> })(); >> >> Just copy this into the editor and evaluate. >> >> After commenting out the DST certificate from the TLS source, the http request >> terminates with an SSL error (-3) with installed... >> >> * self-signed ISRG certificate >> * cross-signed ISRG / DST root certificate >> * LE R3 certificate >> * any combination of these three. >> >> But the access works immediately after reinstalling the old DST root >> certificate. There is no DST reference in the chain of that server... >> >> >> I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT >> lib --; apparently all they needed to do was to add the very same self-signed >> ISRG X1 certificate I added: >> >> * >> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118... >> >> >> So: why would WolfSSL not use the supplied ISRG certificate? Why would it use >> the DST cert without DST being present as an issuer? >> >> The ISRG certificate is the only one having... >> >> signed using : RSA with SHA-256 >> RSA key size : 4096 bits >> >> But that would surely be supported by WolfSSL, wouldn't it? >> >> I'm running out of ideas, and this really isn't my primary area. Any help is >> appreciated. >> >> Regards, >> Michael >> >> >> Am 28.09.21 um 19:15 schrieb Stephen Casner: >>> Michael, >>> >>> Certainly within our code as it stands there is not any mechanism to >>> tell the WolfSSL code to adjust its certificate validation procedure. >>> The browser certificate substitution that you hypothesize is not clear >>> to me. I would expect the validation to simply follow the chain. >>> >>> -- Steve >>> >>> On Tue, 28 Sep 2021, Michael Balzer wrote: >>> >>>> More info: >>>> >>>> All my browsers already have a builtin ISRG X1 certificate signed by ISRG >>>> only, that's the new version: >>>> >>>> https://crt.sh/?id=9314791 >>>> >>>> My server still sends the ISRG X1 certificate cross signed / issued by DST >>>> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on >>>> the >>>> last renewal (last month!): >>>> >>>> https://crt.sh/?id=3958242236 >>>> >>>> Without the DST root cert, WolfSSL then fails validating the DST signed X1 >>>> root certificate (I assume): >>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ >>>> >>>> My servers will continue sending that chain including the outdated root >>>> cert >>>> probably until the next renewal, so it's possible having added the new X1 >>>> root >>>> certificate didn't solve the issue. >>>> >>>> The browsers seem to know how to substitute the DST signed certificate by >>>> the >>>> builtin self-signed (?). Is there a similar option in WolfSSL, and do we >>>> need >>>> to enable that? >>>> >>>> Steve, can you confirm this, do you know a solution? >>>> >>>> Regards, >>>> Michael >>>> >>>> >>>> Am 28.09.21 um 15:34 schrieb Michael Balzer: >>>>> I've tried adding the intermediate cert ("R3") and then also my site >>>>> certificate, that didn't help. >>>>> >>>>> Only adding the DST cert again fixes the connection. >>>>> >>>>> Any ideas? >>>>> >>>>> >>>>> OVMS# tls trust list >>>>> ... >>>>> ISRG Root X1 length 1939 bytes >>>>> 1939 byte certificate: ISRG Root X1 >>>>> cert. version : 3 >>>>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 >>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG >>>>> Root >>>>> X1 >>>>> subject name : C=US, O=Internet Security Research Group, CN=ISRG >>>>> Root >>>>> X1 >>>>> issued on : 2015-06-04 11:04:38 >>>>> expires on : 2035-06-04 11:04:38 >>>>> signed using : RSA with SHA-256 >>>>> RSA key size : 4096 bits >>>>> basic constraints : CA=true >>>>> key usage : Key Cert Sign, CRL Sign >>>>> ... >>>>> dexter length 1972 bytes >>>>> 1972 byte certificate: dexter >>>>> cert. version : 3 >>>>> serial number : >>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 >>>>> issuer name : C=US, O=Let's Encrypt, CN=R3 >>>>> subject name : CN=dexter.shopdriver.de <http://dexter.shopdriver.de/> >>>>> issued on : 2021-08-07 05:47:57 >>>>> expires on : 2021-11-05 05:47:55 >>>>> signed using : RSA with SHA-256 >>>>> RSA key size : 2048 bits >>>>> basic constraints : CA=false >>>>> subject alt name :dexter.shopdriver.de <http://dexter.shopdriver.de/>,dexters-web.de <http://dexters-web.de/>, >>>>> ovms.dexters-web.de <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de >>>>> key usage : Digital Signature, Key Encipherment >>>>> ext key usage : TLS Web Server Authentication, TLS Web Client >>>>> Authentication >>>>> ... >>>>> r3 length 1826 bytes >>>>> 1826 byte certificate: r3 >>>>> cert. version : 3 >>>>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A >>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG >>>>> Root >>>>> X1 >>>>> subject name : C=US, O=Let's Encrypt, CN=R3 >>>>> issued on : 2020-09-04 00:00:00 >>>>> expires on : 2025-09-15 16:00:00 >>>>> signed using : RSA with SHA-256 >>>>> RSA key size : 2048 bits >>>>> basic constraints : CA=true, max_pathlen=0 >>>>> key usage : Digital Signature, Key Cert Sign, CRL Sign >>>>> ext key usage : TLS Web Client Authentication, TLS Web Server >>>>> Authentication >>>>> >>>>> >>>>> >>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer: >>>>>> We would need to bypass / shortcut the "eap" test phase. >>>>>> >>>>>> But I agree, "master" is stable, I haven't had any issues or reports, >>>>>> so I >>>>>> think we could do that. The FreeRTOS timer issue I'm working on only >>>>>> affects very specific conditions, so not necessary to wait for that. >>>>>> >>>>>> Should we remove the expiring DST certificate in that release then? >>>>>> >>>>>> ...uh oh: just tried removing the DST certificate: the module cannot >>>>>> connect to my server anymore...!? >>>>>> >>>>>> I (490213) ovms-server-v2: Connection isovms.dexters-web.de:6870 <http://ovms.dexters-web.de:6870/> >>>>>> TEST1 >>>>>> I (490213) ovms-server-v2: Status: Connecting... >>>>>> V (490723) ovms-server-v2: >>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) >>>>>> W (490723) ovms-server-v2: Connection failed >>>>>> E (490723) ovms-server-v2: Status: Error: Connection failed >>>>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) >>>>>> I (490723) ovms-server-v2: Status: Disconnected >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: >>>>>>> Shall we release a full update? The last 3.2? >>>>>>> >>>>>>> What we have now in master seems stable. >>>>>>> >>>>>>> Mark >>>>>>> >>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer<dexter@expeedo.de> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Everyone, >>>>>>>> >>>>>>>> the DST root certificate we include (DST Root CA X3) expires on >>>>>>>> September 30, i.e. in two days. >>>>>>>> >>>>>>>> OVMS# tls trust list >>>>>>>> DST Root CA X3 length 1200 bytes >>>>>>>> 1200 byte certificate: DST Root CA X3 >>>>>>>> cert. version : 3 >>>>>>>> serial number : >>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >>>>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root >>>>>>>> CA X3 >>>>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root >>>>>>>> CA X3 >>>>>>>> issued on : 2000-09-30 21:12:19 >>>>>>>> * expires on : 2021-09-30 14:01:15* >>>>>>>> signed using : RSA with SHA1 >>>>>>>> RSA key size : 2048 bits >>>>>>>> basic constraints : CA=true >>>>>>>> key usage : Key Cert Sign, CRL Sign >>>>>>>> >>>>>>>> AFAICT, this root certificate is currently used by the OVMS to >>>>>>>> validate Let's Encrypt certificates. >>>>>>>> >>>>>>>> * >>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ >>>>>>>> *https://letsencrypt.org/docs/certificate-compatibility/ >>>>>>>> >>>>>>>> Unfortunately, we missed adding the followup LE root certificate >>>>>>>> "ISRG >>>>>>>> Root X1" in time. >>>>>>>> >>>>>>>> I've just added that certificate to our builtin certificate >>>>>>>> repository, but it's too late now to roll out a "main" update in >>>>>>>> time >>>>>>>> (isn't it?). >>>>>>>> >>>>>>>> So, to prevent losing TLS connectivity with LE servers, users need >>>>>>>> to >>>>>>>> manually add the ISRG Root X1 certificate to their TLS >>>>>>>> repositories. >>>>>>>> >>>>>>>> I've added a section on this to our user manual: >>>>>>>> >>>>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html >>>>>>>> >>>>>>>> If users contact you, point them to that page. >>>>>>>> >>>>>>>> We probably should also remove the expired DST root certificate >>>>>>>> after >>>>>>>> September 30. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Michael >>>>>>>> >>>>>>>> -- >>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>>> _______________________________________________ >>>>>>>> OvmsDev mailing list >>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>> _______________________________________________ >>>>>>> OvmsDev mailing list >>>>>>> OvmsDev@lists.openvehicles.com >>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>> -- >>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>> >>>>>> _______________________________________________ >>>>>> OvmsDev mailing list >>>>>> OvmsDev@lists.openvehicles.com >>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>> -- >>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>> >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>> -- >>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>> >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

...

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

I'll tag this as release 3.2.017 and roll it out directly to "eap" and "main" now.

In case something's seriously wrong, we can still do a 3.2.018 release afterwards.

Regards, Michael

Am 29.09.21 um 15:36 schrieb Michael Balzer:

Everyone,

I've added a build configuration so we can easily switch between mbedTLS and WolfSSL. Default is to use mbedTLS for now.

I also included removing the almost expired DST root certificate (no point in including that anymore) and the changes to avoid using xTimerIsTimerActive() (have been testing that on both my modules during the last days without issues and with improved stability).

I've pushed everything, please have a look at the changes and test the new build.

I've already rolled this out into "edge" on my server (3.2.016-299-g3d9cf7f5), if you don't want to build yourself.

This would now be release candidate 3.2.017. We need to get this rolled out in "main" ASAP, as module OTA updates will be distributed over 24 hours.

Regards, Michael

Am 29.09.21 um 15:01 schrieb Michael Balzer:

...

Patch attached. Additionally I had set the boot debug level (early) to info (may no longer be necessary as I later found that LOG_LOCAL_LEVEL #define in esp32-crypt.h), and you need to disable log monitoring in the USB console to prevent garbled output.

I tried WOLFSSL_ALT_CERT_CHAINS but only got crashes with that.

Regards, Michael

Am 29.09.21 um 14:44 schrieb Mark Webb-Johnson:

...

Can you send me the patch to enable debugging, and I will try with 3.3 branch. I agree that rolling back 3.2 to mbedtls seems sensible for the short term.

Regards, Mark

P.S. I managed to fix this on another project running an old OpenSSL by removing the X3 trusted ca (it had both x1 and x3 trusted). Behavior is highly dependent on the library treatment of multiple verification paths and trusted ca vs cert chain conflicts.

...

On 29 Sep 2021, at 8:11 PM, Michael Balzer <dexter@expeedo.de> wrote:

 I managed to get wolfSSL debug logs, but don't know how to interpret them. See files attached.

The final error seems to be…

I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188 I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188 I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188 E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818 file:/home/balzer/esp/Open-Vehic E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507 file:/home/balzer/esp/Open-Vehic

which is ASN_NO_SIGNER_E     = -188,  /* ASN no signer to confirm failure */

I've had a look at the code, I don't see a chance to find the bug within the remaining time.

We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL commits now. Steve, if you already prepared something in that direction, tell me.

Regards, Michael

Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:

...

The cross-signing bit seems to be on the X1 cert provided in the certificate chain by the server. The one provided there is signed by X3. The problem comes because a different X1 certificate may be provided as a trusted CA (and that X1 is self-signed with a long expiration date - 2035). So the verification process has two possible paths to verify the cert, and the result depends on which one it chooses (and how it handles a trusted CA cert supplementing / replacing one provided by the server).

Overall, a mess. I’m fighting this same issue now with a bunch of systems in my day job, and likely a large part of the Internet is going to start to fail tomorrow. Particularly a lot of small embedded devices.

Regards, Mark.

...

On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part SNI was pretty much intact already, I took care of a remaining Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).

I understand the WolfSSL chain management uses a supplied root cert over one delivered by the server: https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3

> Our problem could be either the wolfssl not recognising the X1 > cert, or not supporting the cross-signing arrangement (where > there are two paths to verify - the expired X3 and the provided > X1). I suspect the latter. In an ideal world, having both X1 and > X3 trusted shouldn’t be a problem.

Is cross signing supported at all by WolfSSL? The only place I've found it mentioned on wolfssl is this page: https://www.wolfssl.com/certificate-chain-chain-trust/ …where they say "(We’ll cover cross-signing below)" … but I don't see them doing that.

There is also no commit in their git history regarding cross-signed certificates or multiple signatures, so I don't think updating to 4.8.1 would help.

The intermediate "R3" has issuer "C=US, O=Internet Security Research Group, CN=ISRG Root X1", which should match the new ISRG certificate, but not the DST certificate. Maybe that's identified by a fingerprint in the signature though (?), and maybe the DST signature comes first and WolfSSL just doesn't check further signatures if the first fails?

I'll look into the code later on, but need to take care of my daily job duties first.

If we can't fix this today, I think our best option is to switch back to mbedTLS for the final 3.2 release and try to fix this for the initial 3.3 release.

Regards, Michael

Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson: > > It seems the dexter server returns different certificates > depending on the presence of SNI extension in the TSL > negotiation. That is pretty normal (where a single IP is serving > multiple different websites), but I am not sure if our client > library supports that? > > Without SNI, we get: > > Serial Number: > 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54 > Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/> >  X509v3 Subject Alternative Name: >                 DNS:ns34.expeedo.de <http://ns34.expeedo.de/> > > > With SNI dexters-web.de <http://dexters-web.de/>, we get: > > Serial Number: > 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0 > Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/> > X509v3 Subject Alternative Name: > DNS:dexter.shopdriver.de <http://dexter.shopdriver.de/>, > DNS:dexters-web.de <http://dexters-web.de/>, > DNS:ovms.dexters-web.de <http://ovms.dexters-web.de/>, > DNS:www.dexter.shopdriver.de > <http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de > <http://www.dexters-web.de/> > > > Assuming the SNI is set correctly by our library, looking at the > dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have: > > Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/> > Issuer: C=US, O=Let's Encrypt, CN=R3 > > Subject: C=US, O=Let's Encrypt, CN=R3 > Issuer: C=US, O=Internet Security Research Group, CN=ISRG > Root X1 > > Subject: C=US, O=Internet Security Research Group, CN=ISRG > Root X1 > Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 > > and the components/ovms_tls/trustedca/dst.crt we have is: > > Subject: O=Digital Signature Trust Co., CN=DST Root CA X3 > Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 > (Which expires Sep 30 14:01:15 2021 GMT and was issued just > a year earlier - ridiculous for a root CA) > > > The components/ovms_tls/trustedca/isrg_x1.crt is: > > Subject: C=US, O=Internet Security Research Group, CN=ISRG > Root X1 > Issuer: C=US, O=Internet Security Research Group, CN=ISRG > Root X1 > Validity >             Not Before: Jun  4 11:04:38 2015 GMT >             Not After : Jun  4 11:04:38 2035 GMT > sha256WithRSAEncryption 4096 bit > > > Our problem could be either the wolfssl not recognising the X1 > cert, or not supporting the cross-signing arrangement (where > there are two paths to verify - the expired X3 and the provided > X1). I suspect the latter. In an ideal world, having both X1 and > X3 trusted shouldn’t be a problem. > > There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 > released February 2021). Perhaps we can try to update to that? > > Mark > >> On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de >> <mailto:dexter@expeedo.de>> wrote: >> >> Signed PGP part >> I can now confirm it's a WolfSSL issue :-( >> >> I've switched back to release 3.2.016, i.e. before changing to >> WolfSSL, and the ISRG Root X1 certificate works perfectly, just >> as it should. >> >> Steve, I remember you included a config option to enable using >> WolfSSL, but cannot find it now. Can you give me a pointer, or >> did you remove that option later on? >> >> >> >> Am 28.09.21 um 21:53 schrieb Stephen Casner: >>> I wonder if we are hitting a key size limit in the WolfSSL code. I >>> know there are configuration parameters in user_settings.h for >>> different SHA sizes, but I don't recall one for RSA keys. >>> >>> -- Steve >>> >>> On Tue, 28 Sep 2021, Michael Balzer wrote: >>> >>>> Still no luck, and getting clueless... :-/ >>>> >>>> Let's Encrypt have a live test server: >>>> https://valid-isrgrootx1.letsencrypt.org/ >>>> >>>> Here's a simple test script to access that server: >>>> >>>> (function(){ >>>> HTTP.request({ >>>> url:"https://valid-isrgrootx1.letsencrypt.org/", >>>> always: function() { JSON.print(this); } >>>> }); >>>> })(); >>>> >>>> Just copy this into the editor and evaluate. >>>> >>>> After commenting out the DST certificate from the TLS source, the http request >>>> terminates with an SSL error (-3) with installed... >>>> >>>> * self-signed ISRG certificate >>>> * cross-signed ISRG / DST root certificate >>>> * LE R3 certificate >>>> * any combination of these three. >>>> >>>> But the access works immediately after reinstalling the old DST root >>>> certificate. There is no DST reference in the chain of that server... >>>> >>>> >>>> I found another ESP32 project -- not using WolfSSL but the esp-idf SSL & MQTT >>>> lib --; apparently all they needed to do was to add the very same self-signed >>>> ISRG X1 certificate I added: >>>> >>>> * >>>> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118... >>>> >>>> >>>> So: why would WolfSSL not use the supplied ISRG certificate? Why would it use >>>> the DST cert without DST being present as an issuer? >>>> >>>> The ISRG certificate is the only one having... >>>> >>>> signed using : RSA with SHA-256 >>>> RSA key size : 4096 bits >>>> >>>> But that would surely be supported by WolfSSL, wouldn't it? >>>> >>>> I'm running out of ideas, and this really isn't my primary area. Any help is >>>> appreciated. >>>> >>>> Regards, >>>> Michael >>>> >>>> >>>> Am 28.09.21 um 19:15 schrieb Stephen Casner: >>>>> Michael, >>>>> >>>>> Certainly within our code as it stands there is not any mechanism to >>>>> tell the WolfSSL code to adjust its certificate validation procedure. >>>>> The browser certificate substitution that you hypothesize is not clear >>>>> to me. I would expect the validation to simply follow the chain. >>>>> >>>>> -- Steve >>>>> >>>>> On Tue, 28 Sep 2021, Michael Balzer wrote: >>>>> >>>>>> More info: >>>>>> >>>>>> All my browsers already have a builtin ISRG X1 certificate signed by ISRG >>>>>> only, that's the new version: >>>>>> >>>>>> https://crt.sh/?id=9314791 >>>>>> >>>>>> My server still sends the ISRG X1 certificate cross signed / issued by DST >>>>>> Root CA X3. That's the chain it got from Let's Encrypt (via certbot) on >>>>>> the >>>>>> last renewal (last month!): >>>>>> >>>>>> https://crt.sh/?id=3958242236 >>>>>> >>>>>> Without the DST root cert, WolfSSL then fails validating the DST signed X1 >>>>>> root certificate (I assume): >>>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ >>>>>> >>>>>> My servers will continue sending that chain including the outdated root >>>>>> cert >>>>>> probably until the next renewal, so it's possible having added the new X1 >>>>>> root >>>>>> certificate didn't solve the issue. >>>>>> >>>>>> The browsers seem to know how to substitute the DST signed certificate by >>>>>> the >>>>>> builtin self-signed (?). Is there a similar option in WolfSSL, and do we >>>>>> need >>>>>> to enable that? >>>>>> >>>>>> Steve, can you confirm this, do you know a solution? >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> >>>>>> Am 28.09.21 um 15:34 schrieb Michael Balzer: >>>>>>> I've tried adding the intermediate cert ("R3") and then also my site >>>>>>> certificate, that didn't help. >>>>>>> >>>>>>> Only adding the DST cert again fixes the connection. >>>>>>> >>>>>>> Any ideas? >>>>>>> >>>>>>> >>>>>>> OVMS# tls trust list >>>>>>> ... >>>>>>> ISRG Root X1 length 1939 bytes >>>>>>> 1939 byte certificate: ISRG Root X1 >>>>>>> cert. version : 3 >>>>>>> serial number : 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 >>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG >>>>>>> Root >>>>>>> X1 >>>>>>> subject name : C=US, O=Internet Security Research Group, CN=ISRG >>>>>>> Root >>>>>>> X1 >>>>>>> issued on : 2015-06-04 11:04:38 >>>>>>> expires on : 2035-06-04 11:04:38 >>>>>>> signed using : RSA with SHA-256 >>>>>>> RSA key size : 4096 bits >>>>>>> basic constraints : CA=true >>>>>>> key usage : Key Cert Sign, CRL Sign >>>>>>> ... >>>>>>> dexter length 1972 bytes >>>>>>> 1972 byte certificate: dexter >>>>>>> cert. version : 3 >>>>>>> serial number : >>>>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 >>>>>>> issuer name : C=US, O=Let's Encrypt, CN=R3 >>>>>>> subject name : CN=dexter.shopdriver.de <http://dexter.shopdriver.de/> >>>>>>> issued on : 2021-08-07 05:47:57 >>>>>>> expires on : 2021-11-05 05:47:55 >>>>>>> signed using : RSA with SHA-256 >>>>>>> RSA key size : 2048 bits >>>>>>> basic constraints : CA=false >>>>>>> subject alt name :dexter.shopdriver.de <http://dexter.shopdriver.de/>,dexters-web.de <http://dexters-web.de/>, >>>>>>> ovms.dexters-web.de <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de >>>>>>> key usage : Digital Signature, Key Encipherment >>>>>>> ext key usage : TLS Web Server Authentication, TLS Web Client >>>>>>> Authentication >>>>>>> ... >>>>>>> r3 length 1826 bytes >>>>>>> 1826 byte certificate: r3 >>>>>>> cert. version : 3 >>>>>>> serial number : 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A >>>>>>> issuer name : C=US, O=Internet Security Research Group, CN=ISRG >>>>>>> Root >>>>>>> X1 >>>>>>> subject name : C=US, O=Let's Encrypt, CN=R3 >>>>>>> issued on : 2020-09-04 00:00:00 >>>>>>> expires on : 2025-09-15 16:00:00 >>>>>>> signed using : RSA with SHA-256 >>>>>>> RSA key size : 2048 bits >>>>>>> basic constraints : CA=true, max_pathlen=0 >>>>>>> key usage : Digital Signature, Key Cert Sign, CRL Sign >>>>>>> ext key usage : TLS Web Client Authentication, TLS Web Server >>>>>>> Authentication >>>>>>> >>>>>>> >>>>>>> >>>>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer: >>>>>>>> We would need to bypass / shortcut the "eap" test phase. >>>>>>>> >>>>>>>> But I agree, "master" is stable, I haven't had any issues or reports, >>>>>>>> so I >>>>>>>> think we could do that. The FreeRTOS timer issue I'm working on only >>>>>>>> affects very specific conditions, so not necessary to wait for that. >>>>>>>> >>>>>>>> Should we remove the expiring DST certificate in that release then? >>>>>>>> >>>>>>>> ...uh oh: just tried removing the DST certificate: the module cannot >>>>>>>> connect to my server anymore...!? >>>>>>>> >>>>>>>> I (490213) ovms-server-v2: Connection isovms.dexters-web.de:6870 <http://ovms.dexters-web.de:6870/> >>>>>>>> TEST1 >>>>>>>> I (490213) ovms-server-v2: Status: Connecting... >>>>>>>> V (490723) ovms-server-v2: >>>>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) >>>>>>>> W (490723) ovms-server-v2: Connection failed >>>>>>>> E (490723) ovms-server-v2: Status: Error: Connection failed >>>>>>>> V (490723) ovms-server-v2: OvmsServerV2MongooseCallback(MG_EV_CLOSE) >>>>>>>> I (490723) ovms-server-v2: Status: Disconnected >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: >>>>>>>>> Shall we release a full update? The last 3.2? >>>>>>>>> >>>>>>>>> What we have now in master seems stable. >>>>>>>>> >>>>>>>>> Mark >>>>>>>>> >>>>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael Balzer<dexter@expeedo.de> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Everyone, >>>>>>>>>> >>>>>>>>>> the DST root certificate we include (DST Root CA X3) expires on >>>>>>>>>> September 30, i.e. in two days. >>>>>>>>>> >>>>>>>>>> OVMS# tls trust list >>>>>>>>>> DST Root CA X3 length 1200 bytes >>>>>>>>>> 1200 byte certificate: DST Root CA X3 >>>>>>>>>> cert. version : 3 >>>>>>>>>> serial number : >>>>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >>>>>>>>>> issuer name : O=Digital Signature Trust Co., CN=DST Root >>>>>>>>>> CA X3 >>>>>>>>>> subject name : O=Digital Signature Trust Co., CN=DST Root >>>>>>>>>> CA X3 >>>>>>>>>> issued on : 2000-09-30 21:12:19 >>>>>>>>>> * expires on : 2021-09-30 14:01:15* >>>>>>>>>> signed using : RSA with SHA1 >>>>>>>>>> RSA key size : 2048 bits >>>>>>>>>> basic constraints : CA=true >>>>>>>>>> key usage : Key Cert Sign, CRL Sign >>>>>>>>>> >>>>>>>>>> AFAICT, this root certificate is currently used by the OVMS to >>>>>>>>>> validate Let's Encrypt certificates. >>>>>>>>>> >>>>>>>>>> * >>>>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ >>>>>>>>>> *https://letsencrypt.org/docs/certificate-compatibility/ >>>>>>>>>> >>>>>>>>>> Unfortunately, we missed adding the followup LE root certificate >>>>>>>>>> "ISRG >>>>>>>>>> Root X1" in time. >>>>>>>>>> >>>>>>>>>> I've just added that certificate to our builtin certificate >>>>>>>>>> repository, but it's too late now to roll out a "main" update in >>>>>>>>>> time >>>>>>>>>> (isn't it?). >>>>>>>>>> >>>>>>>>>> So, to prevent losing TLS connectivity with LE servers, users need >>>>>>>>>> to >>>>>>>>>> manually add the ISRG Root X1 certificate to their TLS >>>>>>>>>> repositories. >>>>>>>>>> >>>>>>>>>> I've added a section on this to our user manual: >>>>>>>>>> >>>>>>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html >>>>>>>>>> >>>>>>>>>> If users contact you, point them to that page. >>>>>>>>>> >>>>>>>>>> We probably should also remove the expired DST root certificate >>>>>>>>>> after >>>>>>>>>> September 30. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Michael >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>>>>> _______________________________________________ >>>>>>>>>> OvmsDev mailing list >>>>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>>>> _______________________________________________ >>>>>>>>> OvmsDev mailing list >>>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>>> -- >>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OvmsDev mailing list >>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>> -- >>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OvmsDev mailing list >>>>>>> OvmsDev@lists.openvehicles.com >>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>> -- >>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>> >>>>>> _______________________________________________ >>>>>> OvmsDev mailing list >>>>>> OvmsDev@lists.openvehicles.com >>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>> -- >>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>> >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >> >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> > > > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com > http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

...

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Steve,

you're right, typo. I focused on getting it to build with mbedTLS, will fix that.

Thanks, Michael

Am 29.09.21 um 21:12 schrieb Stephen Casner:

Michael,

In 92ea05f4122e52b9c74e05f904c0c23eadbcbbb3 I think there may be inconsistencies between CONFIG_MG_SSL_IF_WOLFTLS and CONFIG_MG_SSL_IF_WOLFSSL.

                                                     -- Steve

On Wed, 29 Sep 2021, Michael Balzer wrote:

...

Everyone,

I've added a build configuration so we can easily switch between mbedTLS and WolfSSL. Default is to use mbedTLS for now.

I also included removing the almost expired DST root certificate (no point in including that anymore) and the changes to avoid using xTimerIsTimerActive() (have been testing that on both my modules during the last days without issues and with improved stability).

I've pushed everything, please have a look at the changes and test the new build.

I've already rolled this out into "edge" on my server (3.2.016-299-g3d9cf7f5), if you don't want to build yourself.

This would now be release candidate 3.2.017. We need to get this rolled out in "main" ASAP, as module OTA updates will be distributed over 24 hours.

Regards, Michael

Am 29.09.21 um 15:01 schrieb Michael Balzer:

...

Patch attached. Additionally I had set the boot debug level (early) to info (may no longer be necessary as I later found that LOG_LOCAL_LEVEL #define in esp32-crypt.h), and you need to disable log monitoring in the USB console to prevent garbled output.

I tried WOLFSSL_ALT_CERT_CHAINS but only got crashes with that.

Regards, Michael

Am 29.09.21 um 14:44 schrieb Mark Webb-Johnson:

...

Can you send me the patch to enable debugging, and I will try with 3.3 branch. I agree that rolling back 3.2 to mbedtls seems sensible for the short term.

Regards, Mark

P.S. I managed to fix this on another project running an old OpenSSL by removing the X3 trusted ca (it had both x1 and x3 trusted). Behavior is highly dependent on the library treatment of multiple verification paths and trusted ca vs cert chain conflicts.

...

On 29 Sep 2021, at 8:11 PM, Michael Balzer <dexter@expeedo.de> wrote:

I managed to get wolfSSL debug logs, but don't know how to interpret them. See files attached.

The final error seems to be...

I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188 I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188 I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188 E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818 file:/home/balzer/esp/Open-Vehic E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507 file:/home/balzer/esp/Open-Vehic

which is ASN_NO_SIGNER_E = -188, /* ASN no signer to confirm failure */

I've had a look at the code, I don't see a chance to find the bug within the remaining time.

We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL commits now. Steve, if you already prepared something in that direction, tell me.

Regards, Michael

Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:

...

The cross-signing bit seems to be on the X1 cert provided in the certificate chain by the server. The one provided there is signed by X3. The problem comes because a different X1 certificate may be provided as a trusted CA (and that X1 is self-signed with a long expiration date - 2035). So the verification process has two possible paths to verify the cert, and the result depends on which one it chooses (and how it handles a trusted CA cert supplementing / replacing one provided by the server).

Overall, a mess. I'm fighting this same issue now with a bunch of systems in my day job, and likely a large part of the Internet is going to start to fail tomorrow. Particularly a lot of small embedded devices.

Regards, Mark.

> On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter@expeedo.de > <mailto:dexter@expeedo.de>> wrote: > > Signed PGP part > SNI was pretty much intact already, I took care of a remaining > Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 > / mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020). > > I understand the WolfSSL chain management uses a supplied root cert > over one delivered by the server: > https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3 > >> Our problem could be either the wolfssl not recognising the X1 >> cert, or not supporting the cross-signing arrangement (where there >> are two paths to verify - the expired X3 and the provided X1). I >> suspect the latter. In an ideal world, having both X1 and X3 >> trusted shouldn't be a problem. > Is cross signing supported at all by WolfSSL? The only place I've > found it mentioned on wolfssl is this page: > https://www.wolfssl.com/certificate-chain-chain-trust/ > ...where they say "(We'll cover cross-signing below)" ... but I > don't see them doing that. > > There is also no commit in their git history regarding cross-signed > certificates or multiple signatures, so I don't think updating to > 4.8.1 would help. > > The intermediate "R3" has issuer "C=US, O=Internet Security Research > Group, CN=ISRG Root X1", which should match the new ISRG > certificate, but not the DST certificate. Maybe that's identified by > a fingerprint in the signature though (?), and maybe the DST > signature comes first and WolfSSL just doesn't check further > signatures if the first fails? > > I'll look into the code later on, but need to take care of my daily > job duties first. > > If we can't fix this today, I think our best option is to switch > back to mbedTLS for the final 3.2 release and try to fix this for > the initial 3.3 release. > > Regards, > Michael > > > Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson: >> It seems the dexter server returns different certificates >> depending on the presence of SNI extension in the TSL negotiation. >> That is pretty normal (where a single IP is serving multiple >> different websites), but I am not sure if our client library >> supports that? >> >> Without SNI, we get: >> >> Serial Number: >> 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54 >> Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/> >> X509v3 Subject Alternative Name: >> DNS:ns34.expeedo.de <http://ns34.expeedo.de/> >> >> >> With SNI dexters-web.de <http://dexters-web.de/>, we get: >> >> Serial Number: >> 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0 >> Subject: CN=dexter.shopdriver.de >> <http://dexter.shopdriver.de/> >> X509v3 Subject Alternative Name: >> DNS:dexter.shopdriver.de <http://dexter.shopdriver.de/>, >> DNS:dexters-web.de <http://dexters-web.de/>, >> DNS:ovms.dexters-web.de <http://ovms.dexters-web.de/>, >> DNS:www.dexter.shopdriver.de >> <http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de >> <http://www.dexters-web.de/> >> >> >> Assuming the SNI is set correctly by our library, looking at the >> dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we >> have: >> >> Subject: CN=dexter.shopdriver.de >> <http://dexter.shopdriver.de/> >> Issuer: C=US, O=Let's Encrypt, CN=R3 >> >> Subject: C=US, O=Let's Encrypt, CN=R3 >> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root >> X1 >> >> Subject: C=US, O=Internet Security Research Group, CN=ISRG >> Root X1 >> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 >> >> and the components/ovms_tls/trustedca/dst.crt we have is: >> >> Subject: O=Digital Signature Trust Co., CN=DST Root CA X3 >> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 >> (Which expires Sep 30 14:01:15 2021 GMT and was issued just a >> year earlier - ridiculous for a root CA) >> >> >> The components/ovms_tls/trustedca/isrg_x1.crt is: >> >> Subject: C=US, O=Internet Security Research Group, CN=ISRG >> Root X1 >> Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root >> X1 >> Validity >> Not Before: Jun 4 11:04:38 2015 GMT >> Not After : Jun 4 11:04:38 2035 GMT >> sha256WithRSAEncryption 4096 bit >> >> >> Our problem could be either the wolfssl not recognising the X1 >> cert, or not supporting the cross-signing arrangement (where there >> are two paths to verify - the expired X3 and the provided X1). I >> suspect the latter. In an ideal world, having both X1 and X3 >> trusted shouldn't be a problem. >> >> There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 >> released February 2021). Perhaps we can try to update to that? >> >> Mark >> >>> On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de >>> <mailto:dexter@expeedo.de>> wrote: >>> >>> Signed PGP part >>> I can now confirm it's a WolfSSL issue :-( >>> >>> I've switched back to release 3.2.016, i.e. before changing to >>> WolfSSL, and the ISRG Root X1 certificate works perfectly, just >>> as it should. >>> >>> Steve, I remember you included a config option to enable using >>> WolfSSL, but cannot find it now. Can you give me a pointer, or >>> did you remove that option later on? >>> >>> >>> >>> Am 28.09.21 um 21:53 schrieb Stephen Casner: >>>> I wonder if we are hitting a key size limit in the WolfSSL >>>> code. I >>>> know there are configuration parameters in user_settings.h for >>>> different SHA sizes, but I don't recall one for RSA keys. >>>> >>>> -- >>>> Steve >>>> >>>> On Tue, 28 Sep 2021, Michael Balzer wrote: >>>> >>>>> Still no luck, and getting clueless... :-/ >>>>> >>>>> Let's Encrypt have a live test server: >>>>> https://valid-isrgrootx1.letsencrypt.org/ >>>>> >>>>> Here's a simple test script to access that server: >>>>> >>>>> (function(){ >>>>> HTTP.request({ >>>>> url:"https://valid-isrgrootx1.letsencrypt.org/", >>>>> always: function() { JSON.print(this); } >>>>> }); >>>>> })(); >>>>> >>>>> Just copy this into the editor and evaluate. >>>>> >>>>> After commenting out the DST certificate from the TLS >>>>> source, the http request >>>>> terminates with an SSL error (-3) with installed... >>>>> >>>>> * self-signed ISRG certificate >>>>> * cross-signed ISRG / DST root certificate >>>>> * LE R3 certificate >>>>> * any combination of these three. >>>>> >>>>> But the access works immediately after reinstalling the old >>>>> DST root >>>>> certificate. There is no DST reference in the chain of that >>>>> server... >>>>> >>>>> >>>>> I found another ESP32 project -- not using WolfSSL but the >>>>> esp-idf SSL & MQTT >>>>> lib --; apparently all they needed to do was to add the very >>>>> same self-signed >>>>> ISRG X1 certificate I added: >>>>> >>>>> * >>>>> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118... >>>>> >>>>> >>>>> So: why would WolfSSL not use the supplied ISRG certificate? >>>>> Why would it use >>>>> the DST cert without DST being present as an issuer? >>>>> >>>>> The ISRG certificate is the only one having... >>>>> >>>>> signed using : RSA with SHA-256 >>>>> RSA key size : 4096 bits >>>>> >>>>> But that would surely be supported by WolfSSL, wouldn't it? >>>>> >>>>> I'm running out of ideas, and this really isn't my primary >>>>> area. Any help is >>>>> appreciated. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> >>>>> Am 28.09.21 um 19:15 schrieb Stephen Casner: >>>>>> Michael, >>>>>> >>>>>> Certainly within our code as it stands there is not any >>>>>> mechanism to >>>>>> tell the WolfSSL code to adjust its certificate validation >>>>>> procedure. >>>>>> The browser certificate substitution that you hypothesize >>>>>> is not clear >>>>>> to me. I would expect the validation to simply follow the >>>>>> chain. >>>>>> >>>>>> -- >>>>>> Steve >>>>>> >>>>>> On Tue, 28 Sep 2021, Michael Balzer wrote: >>>>>> >>>>>>> More info: >>>>>>> >>>>>>> All my browsers already have a builtin ISRG X1 >>>>>>> certificate signed by ISRG >>>>>>> only, that's the new version: >>>>>>> >>>>>>> https://crt.sh/?id=9314791 >>>>>>> >>>>>>> My server still sends the ISRG X1 certificate cross >>>>>>> signed / issued by DST >>>>>>> Root CA X3. That's the chain it got from Let's Encrypt >>>>>>> (via certbot) on >>>>>>> the >>>>>>> last renewal (last month!): >>>>>>> >>>>>>> https://crt.sh/?id=3958242236 >>>>>>> >>>>>>> Without the DST root cert, WolfSSL then fails validating >>>>>>> the DST signed X1 >>>>>>> root certificate (I assume): >>>>>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ >>>>>>> >>>>>>> My servers will continue sending that chain including >>>>>>> the outdated root >>>>>>> cert >>>>>>> probably until the next renewal, so it's possible having >>>>>>> added the new X1 >>>>>>> root >>>>>>> certificate didn't solve the issue. >>>>>>> >>>>>>> The browsers seem to know how to substitute the DST >>>>>>> signed certificate by >>>>>>> the >>>>>>> builtin self-signed (?). Is there a similar option in >>>>>>> WolfSSL, and do we >>>>>>> need >>>>>>> to enable that? >>>>>>> >>>>>>> Steve, can you confirm this, do you know a solution? >>>>>>> >>>>>>> Regards, >>>>>>> Michael >>>>>>> >>>>>>> >>>>>>> Am 28.09.21 um 15:34 schrieb Michael Balzer: >>>>>>>> I've tried adding the intermediate cert ("R3") and >>>>>>>> then also my site >>>>>>>> certificate, that didn't help. >>>>>>>> >>>>>>>> Only adding the DST cert again fixes the connection. >>>>>>>> >>>>>>>> Any ideas? >>>>>>>> >>>>>>>> >>>>>>>> OVMS# tls trust list >>>>>>>> ... >>>>>>>> ISRG Root X1 length 1939 bytes >>>>>>>> 1939 byte certificate: ISRG Root X1 >>>>>>>> cert. version : 3 >>>>>>>> serial number : >>>>>>>> 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 >>>>>>>> issuer name : C=US, O=Internet Security >>>>>>>> Research Group, CN=ISRG >>>>>>>> Root >>>>>>>> X1 >>>>>>>> subject name : C=US, O=Internet Security >>>>>>>> Research Group, CN=ISRG >>>>>>>> Root >>>>>>>> X1 >>>>>>>> issued on : 2015-06-04 11:04:38 >>>>>>>> expires on : 2035-06-04 11:04:38 >>>>>>>> signed using : RSA with SHA-256 >>>>>>>> RSA key size : 4096 bits >>>>>>>> basic constraints : CA=true >>>>>>>> key usage : Key Cert Sign, CRL Sign >>>>>>>> ... >>>>>>>> dexter length 1972 bytes >>>>>>>> 1972 byte certificate: dexter >>>>>>>> cert. version : 3 >>>>>>>> serial number : >>>>>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 >>>>>>>> issuer name : C=US, O=Let's Encrypt, CN=R3 >>>>>>>> subject name : CN=dexter.shopdriver.de >>>>>>>> <http://dexter.shopdriver.de/> >>>>>>>> issued on : 2021-08-07 05:47:57 >>>>>>>> expires on : 2021-11-05 05:47:55 >>>>>>>> signed using : RSA with SHA-256 >>>>>>>> RSA key size : 2048 bits >>>>>>>> basic constraints : CA=false >>>>>>>> subject alt name :dexter.shopdriver.de >>>>>>>> <http://dexter.shopdriver.de/>,dexters-web.de >>>>>>>> <http://dexters-web.de/>, >>>>>>>> ovms.dexters-web.de >>>>>>>> <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de >>>>>>>> key usage : Digital Signature, Key >>>>>>>> Encipherment >>>>>>>> ext key usage : TLS Web Server Authentication, >>>>>>>> TLS Web Client >>>>>>>> Authentication >>>>>>>> ... >>>>>>>> r3 length 1826 bytes >>>>>>>> 1826 byte certificate: r3 >>>>>>>> cert. version : 3 >>>>>>>> serial number : >>>>>>>> 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A >>>>>>>> issuer name : C=US, O=Internet Security >>>>>>>> Research Group, CN=ISRG >>>>>>>> Root >>>>>>>> X1 >>>>>>>> subject name : C=US, O=Let's Encrypt, CN=R3 >>>>>>>> issued on : 2020-09-04 00:00:00 >>>>>>>> expires on : 2025-09-15 16:00:00 >>>>>>>> signed using : RSA with SHA-256 >>>>>>>> RSA key size : 2048 bits >>>>>>>> basic constraints : CA=true, max_pathlen=0 >>>>>>>> key usage : Digital Signature, Key Cert >>>>>>>> Sign, CRL Sign >>>>>>>> ext key usage : TLS Web Client Authentication, >>>>>>>> TLS Web Server >>>>>>>> Authentication >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer: >>>>>>>>> We would need to bypass / shortcut the "eap" test >>>>>>>>> phase. >>>>>>>>> >>>>>>>>> But I agree, "master" is stable, I haven't had any >>>>>>>>> issues or reports, >>>>>>>>> so I >>>>>>>>> think we could do that. The FreeRTOS timer issue I'm >>>>>>>>> working on only >>>>>>>>> affects very specific conditions, so not necessary >>>>>>>>> to wait for that. >>>>>>>>> >>>>>>>>> Should we remove the expiring DST certificate in >>>>>>>>> that release then? >>>>>>>>> >>>>>>>>> ...uh oh: just tried removing the DST certificate: >>>>>>>>> the module cannot >>>>>>>>> connect to my server anymore...!? >>>>>>>>> >>>>>>>>> I (490213) ovms-server-v2: Connection >>>>>>>>> isovms.dexters-web.de:6870 >>>>>>>>> <http://ovms.dexters-web.de:6870/> >>>>>>>>> TEST1 >>>>>>>>> I (490213) ovms-server-v2: Status: Connecting... >>>>>>>>> V (490723) ovms-server-v2: >>>>>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) >>>>>>>>> W (490723) ovms-server-v2: Connection failed >>>>>>>>> E (490723) ovms-server-v2: Status: Error: Connection >>>>>>>>> failed >>>>>>>>> V (490723) ovms-server-v2: >>>>>>>>> OvmsServerV2MongooseCallback(MG_EV_CLOSE) >>>>>>>>> I (490723) ovms-server-v2: Status: Disconnected >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: >>>>>>>>>> Shall we release a full update? The last 3.2? >>>>>>>>>> >>>>>>>>>> What we have now in master seems stable. >>>>>>>>>> >>>>>>>>>> Mark >>>>>>>>>> >>>>>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael >>>>>>>>>>> Balzer<dexter@expeedo.de> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Everyone, >>>>>>>>>>> >>>>>>>>>>> the DST root certificate we include (DST Root CA >>>>>>>>>>> X3) expires on >>>>>>>>>>> September 30, i.e. in two days. >>>>>>>>>>> >>>>>>>>>>> OVMS# tls trust list >>>>>>>>>>> DST Root CA X3 length 1200 bytes >>>>>>>>>>> 1200 byte certificate: DST Root CA X3 >>>>>>>>>>> cert. version : 3 >>>>>>>>>>> serial number : >>>>>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >>>>>>>>>>> issuer name : O=Digital Signature >>>>>>>>>>> Trust Co., CN=DST Root >>>>>>>>>>> CA X3 >>>>>>>>>>> subject name : O=Digital Signature >>>>>>>>>>> Trust Co., CN=DST Root >>>>>>>>>>> CA X3 >>>>>>>>>>> issued on : 2000-09-30 21:12:19 >>>>>>>>>>> * expires on : 2021-09-30 14:01:15* >>>>>>>>>>> signed using : RSA with SHA1 >>>>>>>>>>> RSA key size : 2048 bits >>>>>>>>>>> basic constraints : CA=true >>>>>>>>>>> key usage : Key Cert Sign, CRL Sign >>>>>>>>>>> >>>>>>>>>>> AFAICT, this root certificate is currently used >>>>>>>>>>> by the OVMS to >>>>>>>>>>> validate Let's Encrypt certificates. >>>>>>>>>>> >>>>>>>>>>> * >>>>>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ >>>>>>>>>>> *https://letsencrypt.org/docs/certificate-compatibility/ >>>>>>>>>>> >>>>>>>>>>> Unfortunately, we missed adding the followup LE >>>>>>>>>>> root certificate >>>>>>>>>>> "ISRG >>>>>>>>>>> Root X1" in time. >>>>>>>>>>> >>>>>>>>>>> I've just added that certificate to our builtin >>>>>>>>>>> certificate >>>>>>>>>>> repository, but it's too late now to roll out a >>>>>>>>>>> "main" update in >>>>>>>>>>> time >>>>>>>>>>> (isn't it?). >>>>>>>>>>> >>>>>>>>>>> So, to prevent losing TLS connectivity with LE >>>>>>>>>>> servers, users need >>>>>>>>>>> to >>>>>>>>>>> manually add the ISRG Root X1 certificate to >>>>>>>>>>> their TLS >>>>>>>>>>> repositories. >>>>>>>>>>> >>>>>>>>>>> I've added a section on this to our user manual: >>>>>>>>>>> >>>>>>>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html >>>>>>>>>>> >>>>>>>>>>> If users contact you, point them to that page. >>>>>>>>>>> >>>>>>>>>>> We probably should also remove the expired DST >>>>>>>>>>> root certificate >>>>>>>>>>> after >>>>>>>>>>> September 30. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Michael >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 >>>>>>>>>>> Ennepetal >>>>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> OvmsDev mailing list >>>>>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>>>>> _______________________________________________ >>>>>>>>>> OvmsDev mailing list >>>>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>>>> -- >>>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 >>>>>>>>> Ennepetal >>>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> OvmsDev mailing list >>>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>>> -- >>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 >>>>>>>> Ennepetal >>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OvmsDev mailing list >>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>> -- >>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OvmsDev mailing list >>>>>>> OvmsDev@lists.openvehicles.com >>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>> -- >>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>> >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>> -- >>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>> >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev > -- > Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal > Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

...

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

On Wed, 29 Sep 2021, Craig Leres wrote:

And I assume switching to mbedTLS is why I can no longer use https with my modules? Is there any reason I can't just continue using wolfssl?

If the certificates you are using work with WolfSSL (and will continue to do so), then go ahead and build with the configuration changed to use WolfSSL.

                                                    -- Steve

Brian,

Let’s Encrypt say that because there are a lot of embedded devices that do not update firmware / trusted CA lists, they did it this way. They are also worried about older Android devices that can’t be updated.

IMHO, they made a bigger mess of it by using cross-signing and introducing SSL/TLS library compatibility issues. The X3 trusted CA cert that was expiring could have simply been renewed without requiring a new one, and maintaining compatibility and a smooth transition. These CA certs are self-signed anyway, so you can do whatever you want with them (and I’ve done the 10 year re-sign twice already for my day-job primary CA).

Regards, Mark.

On 30 Sep 2021, at 12:42 PM, HONDA S2000 <s2000@audiobanshee.com> wrote:

Sorry to jump in the middle (end?) of a long thread...

Working on another system at work a couple of years ago with embedded security, I recall a couple of topics from the design meetings.

1. Certificate handling should be designed to allow for a transition period where two equivalent certificates are valid at the same time. That's because it's almost impossible to roll out a certificate change in the deployed fleet if only one can be valid at a time. The duration of the overlap needs to be long enough for all systems to transition, and only then can the old certs be removed.

2. Intermediate certificates are what enables these changes. The top or root cert is usually too protected to allow easy changes, or managed by another entity, so the team / project gets an intermediate that can be used to sign multiple leaf certs with varying expirations.

Leaf certs change relatively quickly, while intermediates change relatively infrequently. In both cases, some overlap time needs to be supported. I suppose this means that the root cert also needs a plan for transition that doesn't abruptly cut off all working systems.

Unfortunately, you probably already know all of this, and I'm just not familiar with how OVMS handles certs. But basically, I'm wondering whether a smoother transition can be designed around the *next* expiration. ... or is this something that Let's Encrypt didn't plan for?

Brian Willoughby

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Day #1 reports coming in:

https://www.zdnet.com/article/fortinet-shopify-others-report-issues-after-ro...

“Helme told ZDNet that he confirmed issues with Palo Alto, Bluecoat, Cisco Umbrella, Catchpoint, Guardian Firewall, Monday.com, PFsense, Google Cloud Monitoring, Azure Application Gateway, OVH, Auth0, Shopify, Xero, QuickBooks, Fortinet, Heroku, Rocket League, InstaPage, Ledger, Netlifyand Cloudflare Pages, but noted that there may be more.”

Mark

On 30 Sep 2021, at 2:02 PM, Michael Balzer <dexter@expeedo.de> wrote:

 Scott Helme has written some excellent articles on this: https://scotthelme.co.uk/lets-encrypt-to-transition-to-isrg-root/ https://scotthelme.co.uk/lets-encrypt-old-root-expiration/ Regards, Michael

Am 30.09.21 um 07:02 schrieb Mark Webb-Johnson:

...

Brian,

Let’s Encrypt say that because there are a lot of embedded devices that do not update firmware / trusted CA lists, they did it this way. They are also worried about older Android devices that can’t be updated.

IMHO, they made a bigger mess of it by using cross-signing and introducing SSL/TLS library compatibility issues. The X3 trusted CA cert that was expiring could have simply been renewed without requiring a new one, and maintaining compatibility and a smooth transition. These CA certs are self-signed anyway, so you can do whatever you want with them (and I’ve done the 10 year re-sign twice already for my day-job primary CA).

Regards, Mark.

...

On 30 Sep 2021, at 12:42 PM, HONDA S2000 <s2000@audiobanshee.com> wrote:

Sorry to jump in the middle (end?) of a long thread...

Working on another system at work a couple of years ago with embedded security, I recall a couple of topics from the design meetings.

1. Certificate handling should be designed to allow for a transition period where two equivalent certificates are valid at the same time. That's because it's almost impossible to roll out a certificate change in the deployed fleet if only one can be valid at a time. The duration of the overlap needs to be long enough for all systems to transition, and only then can the old certs be removed.

2. Intermediate certificates are what enables these changes. The top or root cert is usually too protected to allow easy changes, or managed by another entity, so the team / project gets an intermediate that can be used to sign multiple leaf certs with varying expirations.

Leaf certs change relatively quickly, while intermediates change relatively infrequently. In both cases, some overlap time needs to be supported. I suppose this means that the root cert also needs a plan for transition that doesn't abruptly cut off all working systems.

Unfortunately, you probably already know all of this, and I'm just not familiar with how OVMS handles certs. But basically, I'm wondering whether a smoother transition can be designed around the *next* expiration. ... or is this something that Let's Encrypt didn't plan for?

Brian Willoughby

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Thanks and kudos to the OVMS Development Leadership for staying in front of this. It is very much appreciated.

On Oct 1, 2021, at 1:56 PM, Michael Balzer <dexter@expeedo.de> wrote:

Looks like we managed to mitigate this just in time. I've had only one user who didn't get the update before the expiration. Another one reported Firefox wouldn't open my website anymore, it turned out he was using some Firefox version well below 50 and wasn't willing to update, because he didn't like the UI changes of newer releases…

The Let's Encrypt forum is running hot with users having issues. I think this could have been communicated in a better way.

Regards, Michael

Am 01.10.21 um 09:03 schrieb Mark Webb-Johnson:

...

Day #1 reports coming in:

https://www.zdnet.com/article/fortinet-shopify-others-report-issues-after-ro... <https://www.zdnet.com/article/fortinet-shopify-others-report-issues-after-root-ca-certificate-from-lets-encrypt-expires/>

“Helme told ZDNet that he confirmed issues with Palo Alto, Bluecoat, Cisco Umbrella, Catchpoint <https://status.catchpoint.com/incidents/f5yl89kygf12>, Guardian Firewall, Monday.com, PFsense, Google Cloud Monitoring, Azure Application Gateway, OVH, Auth0, Shopify <https://www.shopifystatus.com/incidents>, Xero, QuickBooks, Fortinet, Heroku, Rocket League, InstaPage, Ledger, Netlify <https://answers.netlify.com/t/ways-to-work-around-ssl-caused-build-failures-server-certificate-verification-failed/44945>and Cloudflare Pages, but noted that there may be more.”

Mark

...

On 30 Sep 2021, at 2:02 PM, Michael Balzer <dexter@expeedo.de> <mailto:dexter@expeedo.de> wrote:

 Scott Helme has written some excellent articles on this: https://scotthelme.co.uk/lets-encrypt-to-transition-to-isrg-root/ <https://scotthelme.co.uk/lets-encrypt-to-transition-to-isrg-root/> https://scotthelme.co.uk/lets-encrypt-old-root-expiration/ <https://scotthelme.co.uk/lets-encrypt-old-root-expiration/> Regards, Michael

Am 30.09.21 um 07:02 schrieb Mark Webb-Johnson:

...

Brian,

Let’s Encrypt say that because there are a lot of embedded devices that do not update firmware / trusted CA lists, they did it this way. They are also worried about older Android devices that can’t be updated.

IMHO, they made a bigger mess of it by using cross-signing and introducing SSL/TLS library compatibility issues. The X3 trusted CA cert that was expiring could have simply been renewed without requiring a new one, and maintaining compatibility and a smooth transition. These CA certs are self-signed anyway, so you can do whatever you want with them (and I’ve done the 10 year re-sign twice already for my day-job primary CA).

Regards, Mark.

...

On 30 Sep 2021, at 12:42 PM, HONDA S2000 <s2000@audiobanshee.com> <mailto:s2000@audiobanshee.com> wrote:

Sorry to jump in the middle (end?) of a long thread...

Working on another system at work a couple of years ago with embedded security, I recall a couple of topics from the design meetings.

1. Certificate handling should be designed to allow for a transition period where two equivalent certificates are valid at the same time. That's because it's almost impossible to roll out a certificate change in the deployed fleet if only one can be valid at a time. The duration of the overlap needs to be long enough for all systems to transition, and only then can the old certs be removed.

2. Intermediate certificates are what enables these changes. The top or root cert is usually too protected to allow easy changes, or managed by another entity, so the team / project gets an intermediate that can be used to sign multiple leaf certs with varying expirations.

Leaf certs change relatively quickly, while intermediates change relatively infrequently. In both cases, some overlap time needs to be supported. I suppose this means that the root cert also needs a plan for transition that doesn't abruptly cut off all working systems.

Unfortunately, you probably already know all of this, and I'm just not familiar with how OVMS handles certs. But basically, I'm wondering whether a smoother transition can be designed around the *next* expiration. ... or is this something that Let's Encrypt didn't plan for?

Brian Willoughby

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Steve,

Yes, it does seem that WOLFSSL_ALT_CERT_CHAINS is the correct solution for this.

Mark, how soon will v3.3 modules be available so I could have a usable bench unit?

The modules are in the certification lab now. Still targeting end of November, for certification, and sometime in December for first production batch. But that depends on the lab result. EM testing is always painful (and not really relevant to in-vehicle devices anyway, but a hurdle we must cross).

I can send you a 3.2 wifi unit if it helps? I have a few repaired from RMA returns.

Regards, Mark.

On 10 Oct 2021, at 2:36 AM, Stephen Casner <casner@acm.org> wrote:

On Wed, 29 Sep 2021, Michael Balzer wrote:

...

I tried WOLFSSL_ALT_CERT_CHAINS but only got crashes with that.

This morning I found an issue thread for wolfssl that discusses the Let's Encrypt certificate problem and indicates that WolfSSL is not intending to make any changes:

https://github.com/wolfSSL/wolfssl/issues/4443

The answer in that thread is to build with SNI and WOLFSSL_ALT_CERT_CHAINS. So that means the answer for us is that I need to figure out why that option causes a crash. I remember that I tried that option without success when doing the porting work. I need to refresh my memory as to what was the problem.

Mark, how soon will v3.3 modules be available so I could have a usable bench unit?

                                                   -- Steve

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

No problem. This is also a general open offer. If any active developers here, contributing to the project, need hardware to help with their contributions; then message me privately and I will see what can be done to help.

Regards, Mark.

On 12 Oct 2021, at 11:11 AM, Stephen Casner <casner@acm.org> wrote:

Mark,

Yes, thanks, a repaired 3.2 wifi unit would be sufficient for this testing. I think you have my address.

                                                   -- Steve

On Tue, 12 Oct 2021, Mark Webb-Johnson wrote:

...

Steve,

Yes, it does seem that WOLFSSL_ALT_CERT_CHAINS is the correct solution for this.

...

Mark, how soon will v3.3 modules be available so I could have a usable bench unit?

The modules are in the certification lab now. Still targeting end of November, for certification, and sometime in December for first production batch. But that depends on the lab result. EM testing is always painful (and not really relevant to in-vehicle devices anyway, but a hurdle we must cross).

I can send you a 3.2 wifi unit if it helps? I have a few repaired from RMA returns.

Regards, Mark.

...

On 10 Oct 2021, at 2:36 AM, Stephen Casner <casner@acm.org> wrote:

On Wed, 29 Sep 2021, Michael Balzer wrote:

...

I tried WOLFSSL_ALT_CERT_CHAINS but only got crashes with that.

This morning I found an issue thread for wolfssl that discusses the Let's Encrypt certificate problem and indicates that WolfSSL is not intending to make any changes:

https://github.com/wolfSSL/wolfssl/issues/4443

The answer in that thread is to build with SNI and WOLFSSL_ALT_CERT_CHAINS. So that means the answer for us is that I need to figure out why that option causes a crash. I remember that I tried that option without success when doing the porting work. I need to refresh my memory as to what was the problem.

Mark, how soon will v3.3 modules be available so I could have a usable bench unit?

                                                  -- Steve

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Steve,

I find it strange this hasn't been detected by other WolfSSL users before. May be a specific issue of the ESP32 integration or our build configuration, as you said.

I thought about building WolfSSL on my PC to check that theory, but dropped that idea due to our time constraint.

Regards, Michael

Am 29.09.21 um 16:56 schrieb Stephen Casner:

Michael,

No, I was sleeping at that hour.

I can send a question about cross-signing support to WolfSSL support if you like.

                                                     -- Steve

On Wed, 29 Sep 2021, Michael Balzer wrote:

...

I managed to get wolfSSL debug logs, but don't know how to interpret them. See files attached.

The final error seems to be...

I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188 I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188 I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188 E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818 file:/home/balzer/esp/Open-Vehic E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507 file:/home/balzer/esp/Open-Vehic

which is ASN_NO_SIGNER_E = -188, /* ASN no signer to confirm failure */

I've had a look at the code, I don't see a chance to find the bug within the remaining time.

We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL commits now. Steve, if you already prepared something in that direction, tell me.

Regards, Michael

Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:

...

The cross-signing bit seems to be on the X1 cert provided in the certificate chain by the server. The one provided there is signed by X3. The problem comes because a different X1 certificate may be provided as a trusted CA (and that X1 is self-signed with a long expiration date - 2035). So the verification process has two possible paths to verify the cert, and the result depends on which one it chooses (and how it handles a trusted CA cert supplementing / replacing one provided by the server).

Overall, a mess. I'm fighting this same issue now with a bunch of systems in my day job, and likely a large part of the Internet is going to start to fail tomorrow. Particularly a lot of small embedded devices.

Regards, Mark.

...

On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part SNI was pretty much intact already, I took care of a remaining Mongoose SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose 8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).

I understand the WolfSSL chain management uses a supplied root cert over one delivered by the server: https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3

...

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn't be a problem. Is cross signing supported at all by WolfSSL? The only place I've found it mentioned on wolfssl is this page: https://www.wolfssl.com/certificate-chain-chain-trust/ ...where they say "(We'll cover cross-signing below)" ... but I don't see them doing that.

There is also no commit in their git history regarding cross-signed certificates or multiple signatures, so I don't think updating to 4.8.1 would help.

The intermediate "R3" has issuer "C=US, O=Internet Security Research Group, CN=ISRG Root X1", which should match the new ISRG certificate, but not the DST certificate. Maybe that's identified by a fingerprint in the signature though (?), and maybe the DST signature comes first and WolfSSL just doesn't check further signatures if the first fails?

I'll look into the code later on, but need to take care of my daily job duties first.

If we can't fix this today, I think our best option is to switch back to mbedTLS for the final 3.2 release and try to fix this for the initial 3.3 release.

Regards, Michael

Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:

...

It seems the dexter server returns different certificates depending on the presence of SNI extension in the TSL negotiation. That is pretty normal (where a single IP is serving multiple different websites), but I am not sure if our client library supports that?

Without SNI, we get:

 Serial Number:
 04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
 Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
  X509v3 Subject Alternative Name:
                 DNS:ns34.expeedo.de <http://ns34.expeedo.de/>

With SNI dexters-web.de <http://dexters-web.de/>, we get:

 Serial Number:
 03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
 Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
 X509v3 Subject Alternative Name:
                 DNS:dexter.shopdriver.de
 <http://dexter.shopdriver.de/>, DNS:dexters-web.de
 <http://dexters-web.de/>, DNS:ovms.dexters-web.de
 <http://ovms.dexters-web.de/>, DNS:www.dexter.shopdriver.de
 <http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de
 <http://www.dexters-web.de/>

Assuming the SNI is set correctly by our library, looking at the dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:

 Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
 Issuer: C=US, O=Let's Encrypt, CN=R3

 Subject: C=US, O=Let's Encrypt, CN=R3
 Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1

 Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
 Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3

 and the components/ovms_tls/trustedca/dst.crt we have is:

 Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
 Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
 (Which expires Sep 30 14:01:15 2021 GMT and was issued just a
 year earlier - ridiculous for a root CA)

The components/ovms_tls/trustedca/isrg_x1.crt is:

 Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
 Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
 Validity
             Not Before: Jun  4 11:04:38 2015 GMT
             Not After : Jun  4 11:04:38 2035 GMT
 sha256WithRSAEncryption 4096 bit

Our problem could be either the wolfssl not recognising the X1 cert, or not supporting the cross-signing arrangement (where there are two paths to verify - the expired X3 and the provided X1). I suspect the latter. In an ideal world, having both X1 and X3 trusted shouldn't be a problem.

There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released February 2021). Perhaps we can try to update to that?

Mark

...

On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de <mailto:dexter@expeedo.de>> wrote:

Signed PGP part I can now confirm it's a WolfSSL issue :-(

I've switched back to release 3.2.016, i.e. before changing to WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it should.

Steve, I remember you included a config option to enable using WolfSSL, but cannot find it now. Can you give me a pointer, or did you remove that option later on?

Am 28.09.21 um 21:53 schrieb Stephen Casner: > I wonder if we are hitting a key size limit in the WolfSSL code. I > know there are configuration parameters in user_settings.h for > different SHA sizes, but I don't recall one for RSA keys. > > -- Steve > > On Tue, 28 Sep 2021, Michael Balzer wrote: > >> Still no luck, and getting clueless... :-/ >> >> Let's Encrypt have a live test server: >> https://valid-isrgrootx1.letsencrypt.org/ >> >> Here's a simple test script to access that server: >> >> (function(){ >> HTTP.request({ >> url:"https://valid-isrgrootx1.letsencrypt.org/", >> always: function() { JSON.print(this); } >> }); >> })(); >> >> Just copy this into the editor and evaluate. >> >> After commenting out the DST certificate from the TLS source, the >> http request >> terminates with an SSL error (-3) with installed... >> >> * self-signed ISRG certificate >> * cross-signed ISRG / DST root certificate >> * LE R3 certificate >> * any combination of these three. >> >> But the access works immediately after reinstalling the old DST >> root >> certificate. There is no DST reference in the chain of that >> server... >> >> >> I found another ESP32 project -- not using WolfSSL but the esp-idf >> SSL & MQTT >> lib --; apparently all they needed to do was to add the very same >> self-signed >> ISRG X1 certificate I added: >> >> * >> https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118... >> >> >> So: why would WolfSSL not use the supplied ISRG certificate? Why >> would it use >> the DST cert without DST being present as an issuer? >> >> The ISRG certificate is the only one having... >> >> signed using : RSA with SHA-256 >> RSA key size : 4096 bits >> >> But that would surely be supported by WolfSSL, wouldn't it? >> >> I'm running out of ideas, and this really isn't my primary area. >> Any help is >> appreciated. >> >> Regards, >> Michael >> >> >> Am 28.09.21 um 19:15 schrieb Stephen Casner: >>> Michael, >>> >>> Certainly within our code as it stands there is not any >>> mechanism to >>> tell the WolfSSL code to adjust its certificate validation >>> procedure. >>> The browser certificate substitution that you hypothesize is not >>> clear >>> to me. I would expect the validation to simply follow the >>> chain. >>> >>> -- >>> Steve >>> >>> On Tue, 28 Sep 2021, Michael Balzer wrote: >>> >>>> More info: >>>> >>>> All my browsers already have a builtin ISRG X1 certificate >>>> signed by ISRG >>>> only, that's the new version: >>>> >>>> https://crt.sh/?id=9314791 >>>> >>>> My server still sends the ISRG X1 certificate cross signed / >>>> issued by DST >>>> Root CA X3. That's the chain it got from Let's Encrypt (via >>>> certbot) on >>>> the >>>> last renewal (last month!): >>>> >>>> https://crt.sh/?id=3958242236 >>>> >>>> Without the DST root cert, WolfSSL then fails validating the >>>> DST signed X1 >>>> root certificate (I assume): >>>> https://www.wolfssl.com/docs/wolfssl-manual/ch7/ >>>> >>>> My servers will continue sending that chain including the >>>> outdated root >>>> cert >>>> probably until the next renewal, so it's possible having added >>>> the new X1 >>>> root >>>> certificate didn't solve the issue. >>>> >>>> The browsers seem to know how to substitute the DST signed >>>> certificate by >>>> the >>>> builtin self-signed (?). Is there a similar option in WolfSSL, >>>> and do we >>>> need >>>> to enable that? >>>> >>>> Steve, can you confirm this, do you know a solution? >>>> >>>> Regards, >>>> Michael >>>> >>>> >>>> Am 28.09.21 um 15:34 schrieb Michael Balzer: >>>>> I've tried adding the intermediate cert ("R3") and then also >>>>> my site >>>>> certificate, that didn't help. >>>>> >>>>> Only adding the DST cert again fixes the connection. >>>>> >>>>> Any ideas? >>>>> >>>>> >>>>> OVMS# tls trust list >>>>> ... >>>>> ISRG Root X1 length 1939 bytes >>>>> 1939 byte certificate: ISRG Root X1 >>>>> cert. version : 3 >>>>> serial number : >>>>> 82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00 >>>>> issuer name : C=US, O=Internet Security Research >>>>> Group, CN=ISRG >>>>> Root >>>>> X1 >>>>> subject name : C=US, O=Internet Security Research >>>>> Group, CN=ISRG >>>>> Root >>>>> X1 >>>>> issued on : 2015-06-04 11:04:38 >>>>> expires on : 2035-06-04 11:04:38 >>>>> signed using : RSA with SHA-256 >>>>> RSA key size : 4096 bits >>>>> basic constraints : CA=true >>>>> key usage : Key Cert Sign, CRL Sign >>>>> ... >>>>> dexter length 1972 bytes >>>>> 1972 byte certificate: dexter >>>>> cert. version : 3 >>>>> serial number : >>>>> 04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5 >>>>> issuer name : C=US, O=Let's Encrypt, CN=R3 >>>>> subject name : CN=dexter.shopdriver.de >>>>> <http://dexter.shopdriver.de/> >>>>> issued on : 2021-08-07 05:47:57 >>>>> expires on : 2021-11-05 05:47:55 >>>>> signed using : RSA with SHA-256 >>>>> RSA key size : 2048 bits >>>>> basic constraints : CA=false >>>>> subject alt name :dexter.shopdriver.de >>>>> <http://dexter.shopdriver.de/>,dexters-web.de >>>>> <http://dexters-web.de/>, >>>>> ovms.dexters-web.de >>>>> <http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de >>>>> key usage : Digital Signature, Key Encipherment >>>>> ext key usage : TLS Web Server Authentication, TLS >>>>> Web Client >>>>> Authentication >>>>> ... >>>>> r3 length 1826 bytes >>>>> 1826 byte certificate: r3 >>>>> cert. version : 3 >>>>> serial number : >>>>> 91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A >>>>> issuer name : C=US, O=Internet Security Research >>>>> Group, CN=ISRG >>>>> Root >>>>> X1 >>>>> subject name : C=US, O=Let's Encrypt, CN=R3 >>>>> issued on : 2020-09-04 00:00:00 >>>>> expires on : 2025-09-15 16:00:00 >>>>> signed using : RSA with SHA-256 >>>>> RSA key size : 2048 bits >>>>> basic constraints : CA=true, max_pathlen=0 >>>>> key usage : Digital Signature, Key Cert Sign, >>>>> CRL Sign >>>>> ext key usage : TLS Web Client Authentication, TLS >>>>> Web Server >>>>> Authentication >>>>> >>>>> >>>>> >>>>> Am 28.09.21 um 15:07 schrieb Michael Balzer: >>>>>> We would need to bypass / shortcut the "eap" test phase. >>>>>> >>>>>> But I agree, "master" is stable, I haven't had any issues >>>>>> or reports, >>>>>> so I >>>>>> think we could do that. The FreeRTOS timer issue I'm >>>>>> working on only >>>>>> affects very specific conditions, so not necessary to wait >>>>>> for that. >>>>>> >>>>>> Should we remove the expiring DST certificate in that >>>>>> release then? >>>>>> >>>>>> ...uh oh: just tried removing the DST certificate: the >>>>>> module cannot >>>>>> connect to my server anymore...!? >>>>>> >>>>>> I (490213) ovms-server-v2: Connection >>>>>> isovms.dexters-web.de:6870 >>>>>> <http://ovms.dexters-web.de:6870/> >>>>>> TEST1 >>>>>> I (490213) ovms-server-v2: Status: Connecting... >>>>>> V (490723) ovms-server-v2: >>>>>> OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3) >>>>>> W (490723) ovms-server-v2: Connection failed >>>>>> E (490723) ovms-server-v2: Status: Error: Connection >>>>>> failed >>>>>> V (490723) ovms-server-v2: >>>>>> OvmsServerV2MongooseCallback(MG_EV_CLOSE) >>>>>> I (490723) ovms-server-v2: Status: Disconnected >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson: >>>>>>> Shall we release a full update? The last 3.2? >>>>>>> >>>>>>> What we have now in master seems stable. >>>>>>> >>>>>>> Mark >>>>>>> >>>>>>>> On 28 Sep 2021, at 5:39 PM, Michael >>>>>>>> Balzer<dexter@expeedo.de> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Everyone, >>>>>>>> >>>>>>>> the DST root certificate we include (DST Root CA X3) >>>>>>>> expires on >>>>>>>> September 30, i.e. in two days. >>>>>>>> >>>>>>>> OVMS# tls trust list >>>>>>>> DST Root CA X3 length 1200 bytes >>>>>>>> 1200 byte certificate: DST Root CA X3 >>>>>>>> cert. version : 3 >>>>>>>> serial number : >>>>>>>> 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B >>>>>>>> issuer name : O=Digital Signature Trust Co., >>>>>>>> CN=DST Root >>>>>>>> CA X3 >>>>>>>> subject name : O=Digital Signature Trust Co., >>>>>>>> CN=DST Root >>>>>>>> CA X3 >>>>>>>> issued on : 2000-09-30 21:12:19 >>>>>>>> * expires on : 2021-09-30 14:01:15* >>>>>>>> signed using : RSA with SHA1 >>>>>>>> RSA key size : 2048 bits >>>>>>>> basic constraints : CA=true >>>>>>>> key usage : Key Cert Sign, CRL Sign >>>>>>>> >>>>>>>> AFAICT, this root certificate is currently used by the >>>>>>>> OVMS to >>>>>>>> validate Let's Encrypt certificates. >>>>>>>> >>>>>>>> * >>>>>>>> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ >>>>>>>> *https://letsencrypt.org/docs/certificate-compatibility/ >>>>>>>> >>>>>>>> Unfortunately, we missed adding the followup LE root >>>>>>>> certificate >>>>>>>> "ISRG >>>>>>>> Root X1" in time. >>>>>>>> >>>>>>>> I've just added that certificate to our builtin >>>>>>>> certificate >>>>>>>> repository, but it's too late now to roll out a "main" >>>>>>>> update in >>>>>>>> time >>>>>>>> (isn't it?). >>>>>>>> >>>>>>>> So, to prevent losing TLS connectivity with LE >>>>>>>> servers, users need >>>>>>>> to >>>>>>>> manually add the ISRG Root X1 certificate to their TLS >>>>>>>> repositories. >>>>>>>> >>>>>>>> I've added a section on this to our user manual: >>>>>>>> >>>>>>>> *https://docs.openvehicles.com/en/latest/userguide/ssltls.html >>>>>>>> >>>>>>>> If users contact you, point them to that page. >>>>>>>> >>>>>>>> We probably should also remove the expired DST root >>>>>>>> certificate >>>>>>>> after >>>>>>>> September 30. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Michael >>>>>>>> >>>>>>>> -- >>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 >>>>>>>> Ennepetal >>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>>>> _______________________________________________ >>>>>>>> OvmsDev mailing list >>>>>>>> OvmsDev@lists.openvehicles.com >>>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>>> _______________________________________________ >>>>>>> OvmsDev mailing list >>>>>>> OvmsDev@lists.openvehicles.com >>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>>> -- >>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>>> >>>>>> _______________________________________________ >>>>>> OvmsDev mailing list >>>>>> OvmsDev@lists.openvehicles.com >>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>>> -- >>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>>> >>>>> _______________________________________________ >>>>> OvmsDev mailing list >>>>> OvmsDev@lists.openvehicles.com >>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >>>> -- >>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >>>> >>>> _______________________________________________ >>>> OvmsDev mailing list >>>> OvmsDev@lists.openvehicles.com >>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev >> -- >> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal >> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26 >> >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

Ok

On 28 Sep 2021, at 10:52 PM, Stephen Casner <casner@acm.org> wrote:

Mark,

May I commit the change to vehicle_teslaroadster that removes the hack of faking the timerwait state to be stopped? This goes with the change you are releasing for the iOS app to allow manually starting a charge when in timerwait mode.

Michael, I bought a used Android phone so I could test that version of the app; it already works correctly with timerwait state.

I could make this change about an hour from now if you guys agree.

                                                   -- Steve

...

On Tue, 28 Sep 2021, Mark Webb-Johnson wrote:

Shall we release a full update? The last 3.2?

What we have now in master seems stable.

Mark

---

OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

Craig,

Am 28.09.21 um 19:38 schrieb Craig Leres:

As I understand it the problem is with the let's encrypt cert you are currently using on your server; would switching to a commercial cert that validates with one of the "trustedca" certs already in deployed ovms make this problem go away? Then months down the road switch your server back to a let's encrypt cert?

uh… I wouldn't like that. I've been using LE certificates for all my servers and customers since 2017.

That would also be an option only for dexters-web.de. It wouldn't solve the issue for all the other sites and web services using LE certs. There may be quite some private and non-public OVMS servers running on LE certs also.

Regards, Michael

-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26