[Ovmsdev] ESP-IDF v4 / v5 - baby steps

Thu Jan 26 08:56:56 HKT 2023

Hi,

A quick note to share with you the progress on the ESP-IDFv4 + cmake 
endeavour:

  * The branch
    https://github.com/llange/Open-Vehicle-Monitoring-System-3/tree/experimental-esp-idf-v4-rework-network
    is now ready for greater review. Most of the warnings are gone, and
    I'll need some help for the last ones.

  * I created a draft PR
    https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/806
    . Not that I want to merge it as-is nor soon, but it may help the
    review process to have some central discussion point.

  * I'll now switch to ESP-IDFv5 porting - on a new branch (based on the
    v4) :
    https://github.com/llange/Open-Vehicle-Monitoring-System-3/tree/experimental-esp-idf-v5
    (empty at the moment).

Regards,

Le 23/01/2023 à 22:19, Ludovic LANGE a écrit :
> Michael,
>
> Thanks for your help. FYI I've just pushed the last touches to the 
> vehicle components which are now all converted to cmake and "should 
> work". Still compiles under 3.3.x.
>
> A new things that need to be checked : in 
> https://github.com/llange/Open-Vehicle-Monitoring-System-3/commit/f31117a48d4e293b2be01e8b48c1d580d0af834e 
> i had to replace `INT` with another type. This time I choose `int32_t` 
> but I'm not really sure because I could not find a way to see where 
> and how INT was defined. I could only verify (by compilation tricks) 
> that sizeof(INT) == sizeof(int32_t).
>
> Regarding cmake, I understand your concern with dependencies, and I 
> did have the same reaction. There certainly is a way - I did not spend 
> too much time on it. We will however have some "bugs" in the current 
> state of things when disabling components in menuconfig where a 
> dependency was implied and is now exposed. I'll let the cmake experts 
> guide us :-)
>
> I'll now work on the compilation warnings.
>
> Btw, another thing that I need to mention is that from 3.3 to 5.x, 
> many constants (used in sdkconfig) have been renamed etc... So our 
> sdkconfig is upgraded by the build toolchain when we compile with 4.x 
> or 5.x. I recommend doing a backup if it has been customized - in case of.
>
> Regards,
>
> Ludovic
>
>
> Le 23/01/2023 à 21:11, Michael Balzer a écrit :
>> Ludovic,
>>
>> really great & appreciated you're going ahead on this. I second 
>> skipping idf 4 and going directly for version 5. I'll try to get my 
>> idf 5 build environment up ASAP.
>>
>> I need to get into cmake myself as well. Needing to hard code 
>> dependencies doesn't sound good though, I thought cmake is supposed 
>> to be smart about this.
>>
>> I'll have a look at porting the custom crash handler to idf 5. There 
>> were some other additions that haven't been accepted by Espressif, 
>> IIRC. I'll see if I can check them as well.
>>
>> Regards,
>> Michael
>>
>>
>> Am 23.01.23 um 11:04 schrieb Ludovic LANGE:
>>>
>>> Hi list,
>>>
>>> I am interested in upgrading our codebase to a latest ESP-IDF 
>>> version (preferentially v5 which has just happened end of 2022 - 
>>> Happy New Year ! :-)) - mostly to be able to use some components for 
>>> which a backport seems difficult (for example : 
>>> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/752 
>>> ), and as a matter of balance between stability vs latest bugfixes, 
>>> security fixes, new features etc...
>>>
>>> In the past it seems that some endeavours already set the basis to 
>>> make this happen 
>>> (https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/263, 
>>> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/360 
>>> ).
>>>
>>> I'm now trying to revive this effort, and after some work managed to 
>>> have:
>>>
>>>   * OVMS codebase building under v4.x using Makefile
>>>   * OVMS codebase building under v4.x using cmake / idf.py (not 100%
>>>     finished, multiple vehicle_* components are missing)
>>>
>>> (/*WARNING*: some important features of OVMS have been deactivated 
>>> for the moment, see end of post)/
>>>
>>> The produced binaries seem to "work on my machine board", as they 
>>> end with a "OVMS#" prompt and a few (very) basic tests look OK (to 
>>> be honest, unfortunately no real tests have been done except to 
>>> check that the binary does not crash during boot and a few seconds 
>>> of run, checking that WiFi / web Interface is OK, sdcard is OK - but 
>>> very basic)
>>>
>>>
>>> You'll find the changes here : 
>>> https://github.com/llange/Open-Vehicle-Monitoring-System-3/tree/experimental-esp-idf-v4-rework-network
>>> (In the README.md, there is a short notice at the top with a few 
>>> useful informations)
>>>
>>>
>>> I'm now reaching out here mostly for :
>>>
>>>   * gathering feedback both around the effort (is it worth pursuing
>>>     this goal ?) and the implementation (do my CMakeLists.txt look
>>>     weird ? yes they do.)
>>>   * having volunteers using those builds on real-world workload
>>>     (which I cannot do at the moment) and finding bugs
>>>   * guidance and fixes regarding the CMakeLists.txt files
>>>
>>>
>>> My next steps could be (of course, subject to feedback) :
>>>
>>>  1. Have 100% of the cmake implementation finished (all the missing
>>>     vehicle_* components) -> easy (only missing time)
>>>  2. Work on the warnings (currently disabled in the builds) and try
>>>     to fix as much as possible
>>>  3. Iron out all the bugs, improve the dev documentation -> need
>>>     help on finding them with real-world use
>>>  4. 1st goal is to propose this result as a merge request (either as
>>>     a whole, or split depending on feedback - ideally I would like
>>>     to have a first (big) MR accepted quickly so that people can
>>>     play with it, and deliver incremental updates as expected)
>>>  5. then work on the v5 and do the same.
>>>
>>>
>>> (Please note that my cmake skills are inexistent - so be careful 
>>> when opening those files to not hurt yourself.)
>>>
>>> # The guidelines I tried to follow:
>>>
>>>   * Stay compatible with our 3.3.x branch which should still compile
>>>     without any issue nor loss of functionality
>>>   * Keep Makefiles running for 3.3.x (and 4.x if possible)
>>>   * When possible, isolate differences (between 3.x, 4.x, 5.x) by
>>>     testing the versions of ESP-IDF.
>>>   * Focus on 5.x (if there are impossibilities, let's make it easier
>>>     to build v5.x than v4.x)
>>>   * https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32/api-reference/network/tcpip_adapter_migration.html
>>>   * https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32/api-guides/build-system.html#migrating-from-esp-idf-gnu-make-system
>>>   * https://docs.espressif.com/projects/esp-idf/en/latest/esp32/migration-guides/release-5.x/5.0/index.html
>>>   * https://docs.espressif.com/projects/esp-idf/en/latest/esp32/migration-guides/release-5.x/5.1/index.html
>>>
>>>
>>> # Missing features:
>>>
>>> In the course of this work I had to make some decisions to speed up 
>>> the work  - that will need to be acted upon:
>>>
>>>   * I disabled the crash handler (`xt_set_error_handler_callback`
>>>     and `esp_task_wdt_get_trigger_tasknames`) for the moment, we
>>>     need to decide whether we "fork" ESP-IDF again to port it ; or
>>>     if the new APIs are enough to (partially ?) reimplement it
>>>   * There is a crash in `OvmsConsole::Poll` which I did not analyse
>>>     (yet) and which I worked around by declaring a variable static.
>>>   * I decided to transform our local copies of `wolfssh` and
>>>     `wolfssl` in submodules (and move them one level below in terms
>>>     of directories) - mainly to be able to have a CMakeLists.txt
>>>     different from the upstream one.
>>>     In the process, I "lost" one of our patches :
>>>     https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/51444539047daef7bd2accb23ef40d1bc14fdb20
>>>     and we need to decide how to handle this.
>>>   * A lot of dependencies are now explicitly (hard-)coded in the
>>>     CMakeLists.txt - which may, or may not be a good thing. Let's
>>>     discuss it.
>>>   * I had to change a set of defines into a header generation (in
>>>     ovms_webserver) because I was unable to implement those in a
>>>     satisfying manner in cmake. If you manage to do it, it's a win !
>>>   * Build warnings were disabled to keep the output clean and focus
>>>     on the build issues - but now they need to be re-activated.
>>>   * etc...
>>>   * Even if there is experimental support for cmake / idf.py in
>>>     ESP-IDF 3.3.x, I did not make any effort to be compatible with it:
>>>       o Our own fork does not seem to include this build system
>>>       o It was experimental, the component API changed and I did not
>>>         want to have edge case for something not used at the moment.
>>>
>>>
>>> Let me know your comments and questions.
>>>
>>> Best regards,
>>>
>>
>> -- 
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20230126/ec7d0a08/attachment-0001.htm>