Michael,

No, I was sleeping at that hour.

I can send a question about cross-signing support to WolfSSL support
if you like.

                                                        -- Steve

On Wed, 29 Sep 2021, Michael Balzer wrote:

I managed to get wolfSSL debug logs, but don't know how to interpret them. See
files attached.

The final error seems to be...

I (597167) wolfssl: wolfSSL Leaving DoCertificate, return -188
I (597177) wolfssl: wolfSSL Leaving DoHandShakeMsgType(), return -188
I (597187) wolfssl: wolfSSL Leaving DoHandShakeMsg(), return -188
E (597187) wolfssl: wolfSSL error occurred, error = 188 line:15818
file:/home/balzer/esp/Open-Vehic
E (597197) wolfssl: wolfSSL error occurred, error = 188 line:12507
file:/home/balzer/esp/Open-Vehic

which is
    ASN_NO_SIGNER_E     = -188,  /* ASN no signer to confirm failure */

I've had a look at the code, I don't see a chance to find the bug within the
remaining time.

We're 24 hours from expiration. I'll try reverting the Mongoose-WolfSSL
commits now. Steve, if you already prepared something in that direction, tell
me.

Regards,
Michael


Am 29.09.21 um 09:56 schrieb Mark Webb-Johnson:

The cross-signing bit seems to be on the X1 cert provided in the certificate
chain by the server. The one provided there is signed by X3. The problem
comes because a different X1 certificate may be provided as a trusted CA
(and that X1 is self-signed with a long expiration date - 2035). So the
verification process has two possible paths to verify the cert, and the
result depends on which one it chooses (and how it handles a trusted CA cert
supplementing / replacing one provided by the server).

Overall, a mess. I'm fighting this same issue now with a bunch of systems in
my day job, and likely a large part of the Internet is going to start to
fail tomorrow. Particularly a lot of small embedded devices.

Regards, Mark.

On 29 Sep 2021, at 2:51 PM, Michael Balzer <dexter@expeedo.de
<mailto:dexter@expeedo.de>> wrote:

Signed PGP part
SNI was pretty much intact already, I took care of a remaining Mongoose
SNI bug in commit 9e4f31249856a67317f99920e47efa34f6734c73 / mongoose
8dc9012b57b85e062974fbdec17db30a501bf68f (April 2020).

I understand the WolfSSL chain management uses a supplied root cert over
one delivered by the server:
https://www.wolfssl.com/docs/wolfssl-manual/ch7/ → section 7.3

Our problem could be either the wolfssl not recognising the X1 cert, or
not supporting the cross-signing arrangement (where there are two paths
to verify - the expired X3 and the provided X1). I suspect the latter.
In an ideal world, having both X1 and X3 trusted shouldn't be a problem.

Is cross signing supported at all by WolfSSL? The only place I've found it
mentioned on wolfssl is this page:
https://www.wolfssl.com/certificate-chain-chain-trust/
...where they say "(We'll cover cross-signing below)" ... but I don't see
them doing that.

There is also no commit in their git history regarding cross-signed
certificates or multiple signatures, so I don't think updating to 4.8.1
would help.

The intermediate "R3" has issuer "C=US, O=Internet Security Research
Group, CN=ISRG Root X1", which should match the new ISRG certificate, but
not the DST certificate. Maybe that's identified by a fingerprint in the
signature though (?), and maybe the DST signature comes first and WolfSSL
just doesn't check further signatures if the first fails?

I'll look into the code later on, but need to take care of my daily job
duties first.

If we can't fix this today, I think our best option is to switch back to
mbedTLS for the final 3.2 release and try to fix this for the initial 3.3
release.

Regards,
Michael


Am 29.09.21 um 03:40 schrieb Mark Webb-Johnson:

It seems the dexter server returns different certificates depending on
the presence of SNI extension in the TSL negotiation. That is pretty
normal (where a single IP is serving multiple different websites), but I
am not sure if our client library supports that?

Without SNI, we get:

    Serial Number:
    04:ab:25:e5:50:75:49:7b:9f:1b:39:8f:ed:3e:53:44:b4:54
    Subject: CN=ns34.expeedo.de <http://ns34.expeedo.de/>
     X509v3 Subject Alternative Name:
                    DNS:ns34.expeedo.de <http://ns34.expeedo.de/>


With SNI dexters-web.de <http://dexters-web.de/>, we get:

    Serial Number:
    03:be:45:05:45:aa:e3:da:cb:4c:38:ae:f6:90:2f:65:65:e0
    Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
    X509v3 Subject Alternative Name:
                    DNS:dexter.shopdriver.de
    <http://dexter.shopdriver.de/>, DNS:dexters-web.de
    <http://dexters-web.de/>, DNS:ovms.dexters-web.de
    <http://ovms.dexters-web.de/>, DNS:www.dexter.shopdriver.de
    <http://www.dexter.shopdriver.de/>, DNS:www.dexters-web.de
    <http://www.dexters-web.de/>


Assuming the SNI is set correctly by our library, looking at the
dexter.shopdriver.de <http://dexter.shopdriver.de/> chain, we have:

    Subject: CN=dexter.shopdriver.de <http://dexter.shopdriver.de/>
    Issuer: C=US, O=Let's Encrypt, CN=R3

    Subject: C=US, O=Let's Encrypt, CN=R3
    Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1

    Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
    Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3

    and the components/ovms_tls/trustedca/dst.crt we have is:

    Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
    Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
    (Which expires Sep 30 14:01:15 2021 GMT and was issued just a
    year earlier - ridiculous for a root CA)


The components/ovms_tls/trustedca/isrg_x1.crt is:

    Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
    Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
    Validity
                Not Before: Jun  4 11:04:38 2015 GMT
                Not After : Jun  4 11:04:38 2035 GMT
    sha256WithRSAEncryption 4096 bit


Our problem could be either the wolfssl not recognising the X1 cert, or
not supporting the cross-signing arrangement (where there are two paths
to verify - the expired X3 and the provided X1). I suspect the latter.
In an ideal world, having both X1 and X3 trusted shouldn't be a problem.

There is a later 4.8.1 version of wolfssl (we seem to use 4.7.0 released
February 2021). Perhaps we can try to update to that?

Mark

On 29 Sep 2021, at 4:11 AM, Michael Balzer <dexter@expeedo.de
<mailto:dexter@expeedo.de>> wrote:

Signed PGP part
I can now confirm it's a WolfSSL issue :-(

I've switched back to release 3.2.016, i.e. before changing to
WolfSSL, and the ISRG Root X1 certificate works perfectly, just as it
should.

Steve, I remember you included a config option to enable using
WolfSSL, but cannot find it now. Can you give me a pointer, or did you
remove that option later on?



Am 28.09.21 um 21:53 schrieb Stephen Casner:

I wonder if we are hitting a key size limit in the WolfSSL code.  I
know there are configuration parameters in user_settings.h for
different SHA sizes, but I don't recall one for RSA keys.

                                                         -- Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

Still no luck, and getting clueless... :-/

Let's Encrypt have a live test server:
https://valid-isrgrootx1.letsencrypt.org/

Here's a simple test script to access that server:

(function(){
     HTTP.request({
       url:"https://valid-isrgrootx1.letsencrypt.org/",
       always: function() { JSON.print(this); }
     });
})();

Just copy this into the editor and evaluate.

After commenting out the DST certificate from the TLS source, the
http request
terminates with an SSL error (-3) with installed...

  * self-signed ISRG certificate
  * cross-signed ISRG / DST root certificate
  * LE R3 certificate
  * any combination of these three.

But the access works immediately after reinstalling the old DST
root
certificate. There is no DST reference in the chain of that
server...


I found another ESP32 project -- not using WolfSSL but the esp-idf
SSL & MQTT
lib --; apparently all they needed to do was to add the very same
self-signed
ISRG X1 certificate I added:

  *
https://github.com/LibreSolar/esp32-edge-firmware/commit/d6b6307cbdb60feb118355fda973eba11d52f8f5


So: why would WolfSSL not use the supplied ISRG certificate? Why
would it use
the DST cert without DST being present as an issuer?

The ISRG certificate is the only one having...

   signed using      : RSA with SHA-256
   RSA key size      : 4096 bits

But that would surely be supported by WolfSSL, wouldn't it?

I'm running out of ideas, and this really isn't my primary area.
Any help is
appreciated.

Regards,
Michael


Am 28.09.21 um 19:15 schrieb Stephen Casner:

Michael,

Certainly within our code as it stands there is not any
mechanism to
tell the WolfSSL code to adjust its certificate validation
procedure.
The browser certificate substitution that you hypothesize is not
clear
to me.  I would expect the validation to simply follow the
chain.

                                                          --
Steve

On Tue, 28 Sep 2021, Michael Balzer wrote:

More info:

All my browsers already have a builtin ISRG X1 certificate
signed by ISRG
only, that's the new version:

https://crt.sh/?id=9314791

My server still sends the ISRG X1 certificate cross signed /
issued by DST
Root CA X3. That's the chain it got from Let's Encrypt (via
certbot) on
the
last renewal (last month!):

https://crt.sh/?id=3958242236

Without the DST root cert, WolfSSL then fails validating the
DST signed X1
root certificate (I assume):
https://www.wolfssl.com/docs/wolfssl-manual/ch7/

My servers will continue sending that chain including the
outdated root
cert
probably until the next renewal, so it's possible having added
the new X1
root
certificate didn't solve the issue.

The browsers seem to know how to substitute the DST signed
certificate by
the
builtin self-signed (?). Is there a similar option in WolfSSL,
and do we
need
to enable that?

Steve, can you confirm this, do you know a solution?

Regards,
Michael


Am 28.09.21 um 15:34 schrieb Michael Balzer:

I've tried adding the intermediate cert ("R3") and then also
my site
certificate, that didn't help.

Only adding the DST cert again fixes the connection.

Any ideas?


OVMS# tls trust list
...
ISRG Root X1 length 1939 bytes
1939 byte certificate: ISRG Root X1
    cert. version     : 3
    serial number     :
82:10:CF:B0:D2:40:E3:59:44:63:E0:BB:63:82:8B:00
    issuer name       : C=US, O=Internet Security Research
Group, CN=ISRG
Root
X1
    subject name      : C=US, O=Internet Security Research
Group, CN=ISRG
Root
X1
    issued  on        : 2015-06-04 11:04:38
    expires on        : 2035-06-04 11:04:38
    signed using      : RSA with SHA-256
    RSA key size      : 4096 bits
    basic constraints : CA=true
    key usage         : Key Cert Sign, CRL Sign
...
dexter length 1972 bytes
1972 byte certificate: dexter
    cert. version     : 3
    serial number     :
04:55:1D:F4:27:A3:7D:E9:E4:A8:5C:37:F6:A1:61:87:3C:E5
    issuer name       : C=US, O=Let's Encrypt, CN=R3
    subject name      : CN=dexter.shopdriver.de
<http://dexter.shopdriver.de/>
    issued  on        : 2021-08-07 05:47:57
    expires on        : 2021-11-05 05:47:55
    signed using      : RSA with SHA-256
    RSA key size      : 2048 bits
    basic constraints : CA=false
    subject alt name  :dexter.shopdriver.de
<http://dexter.shopdriver.de/>,dexters-web.de
<http://dexters-web.de/>,
ovms.dexters-web.de
<http://ovms.dexters-web.de/>,www.dexter.shopdriver.de,www.dexters-web.de
    key usage         : Digital Signature, Key Encipherment
    ext key usage     : TLS Web Server Authentication, TLS
Web Client
Authentication
...
r3 length 1826 bytes
1826 byte certificate: r3
    cert. version     : 3
    serial number     :
91:2B:08:4A:CF:0C:18:A7:53:F6:D6:2E:25:A7:5F:5A
    issuer name       : C=US, O=Internet Security Research
Group, CN=ISRG
Root
X1
    subject name      : C=US, O=Let's Encrypt, CN=R3
    issued  on        : 2020-09-04 00:00:00
    expires on        : 2025-09-15 16:00:00
    signed using      : RSA with SHA-256
    RSA key size      : 2048 bits
    basic constraints : CA=true, max_pathlen=0
    key usage         : Digital Signature, Key Cert Sign,
CRL Sign
    ext key usage     : TLS Web Client Authentication, TLS
Web Server
Authentication



Am 28.09.21 um 15:07 schrieb Michael Balzer:

We would need to bypass / shortcut the "eap" test phase.

But I agree, "master" is stable, I haven't had any issues
or reports,
so I
think we could do that. The FreeRTOS timer issue I'm
working on only
affects very specific conditions, so not necessary to wait
for that.

Should we remove the expiring DST certificate in that
release then?

...uh oh: just tried removing the DST certificate: the
module cannot
connect to my server anymore...!?

I (490213) ovms-server-v2: Connection
isovms.dexters-web.de:6870
<http://ovms.dexters-web.de:6870/>
TEST1
I (490213) ovms-server-v2: Status: Connecting...
V (490723) ovms-server-v2:
OvmsServerV2MongooseCallback(MG_EV_CONNECT=-3)
W (490723) ovms-server-v2: Connection failed
E (490723) ovms-server-v2: Status: Error: Connection
failed
V (490723) ovms-server-v2:
OvmsServerV2MongooseCallback(MG_EV_CLOSE)
I (490723) ovms-server-v2: Status: Disconnected




Am 28.09.21 um 14:32 schrieb Mark Webb-Johnson:

Shall we release a full update? The last 3.2?

What we have now in master seems stable.

Mark

On 28 Sep 2021, at 5:39 PM, Michael
Balzer<dexter@expeedo.de>
wrote:

   Everyone,

the DST root certificate we include (DST Root CA X3)
expires on
September 30, i.e. in two days.

OVMS# tls trust list
DST Root CA X3 length 1200 bytes
1200 byte certificate: DST Root CA X3
    cert. version     : 3
    serial number     :
44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
    issuer name       : O=Digital Signature Trust Co.,
CN=DST Root
CA X3
    subject name      : O=Digital Signature Trust Co.,
CN=DST Root
CA X3
    issued  on        : 2000-09-30 21:12:19
*  expires on        : 2021-09-30 14:01:15*
    signed using      : RSA with SHA1
    RSA key size      : 2048 bits
    basic constraints : CA=true
    key usage         : Key Cert Sign, CRL Sign

AFAICT, this root certificate is currently used by the
OVMS to
validate Let's Encrypt certificates.

    *
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
    *https://letsencrypt.org/docs/certificate-compatibility/

Unfortunately, we missed adding the followup LE root
certificate
"ISRG
Root X1" in time.

I've just added that certificate to our builtin
certificate
repository, but it's too late now to roll out a "main"
update in
time
(isn't it?).

So, to prevent losing TLS connectivity with LE
servers, users need
to
manually add the ISRG Root X1 certificate to their TLS
repositories.

I've added a section on this to our user manual:

    *https://docs.openvehicles.com/en/latest/userguide/ssltls.html

If users contact you, point them to that page.

We probably should also remove the expired DST root
certificate
after
September 30.

Regards,
Michael

--
Michael Balzer * Helkenberger Weg 9 * D-58256
Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

>

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26