[Ovmsdev] Urgent TLS root certificate issue (Let's Encrypt)

Sat Oct 2 07:58:27 HKT 2021

Thanks and kudos to the OVMS Development Leadership for staying in front of this.  It is very much appreciated.

> On Oct 1, 2021, at 1:56 PM, Michael Balzer <dexter at expeedo.de> wrote:
> 
> Looks like we managed to mitigate this just in time. I've had only one user who didn't get the update before the expiration. Another one reported Firefox wouldn't open my website anymore, it turned out he was using some Firefox version well below 50 and wasn't willing to update, because he didn't like the UI changes of newer releases…
> 
> The Let's Encrypt forum is running hot with users having issues. I think this could have been communicated in a better way.
> 
> Regards,
> Michael
> 
> 
> Am 01.10.21 um 09:03 schrieb Mark Webb-Johnson:
>> 
>> Day #1 reports coming in:
>> 
>> https://www.zdnet.com/article/fortinet-shopify-others-report-issues-after-root-ca-certificate-from-lets-encrypt-expires/ <https://www.zdnet.com/article/fortinet-shopify-others-report-issues-after-root-ca-certificate-from-lets-encrypt-expires/>
>> 
>> “Helme told ZDNet that he confirmed issues with Palo Alto, Bluecoat, Cisco Umbrella, Catchpoint <https://status.catchpoint.com/incidents/f5yl89kygf12>, Guardian Firewall, Monday.com, PFsense, Google Cloud Monitoring, Azure Application Gateway, OVH, Auth0, Shopify <https://www.shopifystatus.com/incidents>, Xero, QuickBooks, Fortinet, Heroku, Rocket League, InstaPage, Ledger, Netlify <https://answers.netlify.com/t/ways-to-work-around-ssl-caused-build-failures-server-certificate-verification-failed/44945>and Cloudflare Pages, but noted that there may be more.”
>> 
>> Mark
>> 
>>> On 30 Sep 2021, at 2:02 PM, Michael Balzer <dexter at expeedo.de> <mailto:dexter at expeedo.de> wrote:
>>> 
>>>  Scott Helme has written some excellent articles on this:
>>> https://scotthelme.co.uk/lets-encrypt-to-transition-to-isrg-root/ <https://scotthelme.co.uk/lets-encrypt-to-transition-to-isrg-root/>
>>> https://scotthelme.co.uk/lets-encrypt-old-root-expiration/ <https://scotthelme.co.uk/lets-encrypt-old-root-expiration/>
>>> Regards,
>>> Michael
>>> 
>>> 
>>> Am 30.09.21 um 07:02 schrieb Mark Webb-Johnson:
>>>> Brian,
>>>> 
>>>> Let’s Encrypt say that because there are a lot of embedded devices that do not update firmware / trusted CA lists, they did it this way. They are also worried about older Android devices that can’t be updated.
>>>> 
>>>> IMHO, they made a bigger mess of it by using cross-signing and introducing SSL/TLS library compatibility issues. The X3 trusted CA cert that was expiring could have simply been renewed without requiring a new one, and maintaining compatibility and a smooth transition. These CA certs are self-signed anyway, so you can do whatever you want with them (and I’ve done the 10 year re-sign twice already for my day-job primary CA).
>>>> 
>>>> Regards, Mark.
>>>> 
>>>>> On 30 Sep 2021, at 12:42 PM, HONDA S2000 <s2000 at audiobanshee.com> <mailto:s2000 at audiobanshee.com> wrote:
>>>>> 
>>>>> Sorry to jump in the middle (end?) of a long thread...
>>>>> 
>>>>> Working on another system at work a couple of years ago with embedded security, I recall a couple of topics from the design meetings.
>>>>> 
>>>>> 1) Certificate handling should be designed to allow for a transition period where two equivalent certificates are valid at the same time. That's because it's almost impossible to roll out a certificate change in the deployed fleet if only one can be valid at a time. The duration of the overlap needs to be long enough for all systems to transition, and only then can the old certs be removed.
>>>>> 
>>>>> 2) Intermediate certificates are what enables these changes. The top or root cert is usually too protected to allow easy changes, or managed by another entity, so the team / project gets an intermediate that can be used to sign multiple leaf certs with varying expirations.
>>>>> 
>>>>> Leaf certs change relatively quickly, while intermediates change relatively infrequently. In both cases, some overlap time needs to be supported. I suppose this means that the root cert also needs a plan for transition that doesn't abruptly cut off all working systems.
>>>>> 
>>>>> Unfortunately, you probably already know all of this, and I'm just not familiar with how OVMS handles certs. But basically, I'm wondering whether a smoother transition can be designed around the *next* expiration. ... or is this something that Let's Encrypt didn't plan for?
>>>>> 
>>>>> Brian Willoughby
>>>>> 
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>> 
>>> -- 
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>> 
>> 
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
> 
> -- 
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20211001/b380902e/attachment.htm>