[Ovmsdev] ESP-IDF v4 / v5 - baby steps

Fri Jan 27 10:17:17 HKT 2023

I may try this on my spare board as a start! (The Aux 12v switch didn't
work... But otherwise it's mostly OK)

Michael.

On Thu, 26 Jan 2023, 4:41 pm Mark Webb-Johnson, <mark at webb-johnson.net>
wrote:

> Progress looks good on this. Thanks for stepping up to try to tackle it.
>
> Pain points for me with our current approach are:
>
>
>    1. RAM usage (in particular IRAM)
>    2. IDF bugs that may be improved in newer versions
>    3. Support for newer hardware (including ESP32-S3)
>
>
> #3 would give us a welcome 12 more GPIOs with minimal hardware changes,
> but possibly worse RAM situation (520KB->512KB, see #1).
>
> My preference would be v5 IDF (for the newer hardware support).
>
> I think it would be fine to have this as a new branch. Not particularly
> important to make it work in the master one.
>
> Regards, Mark.
>
> On 26 Jan 2023, at 8:56 AM, Ludovic LANGE <ll-ovmsdev at lange.nom.fr> wrote:
>
> Hi,
>
> A quick note to share with you the progress on the ESP-IDFv4 + cmake
> endeavour:
>
>    - The branch
>    https://github.com/llange/Open-Vehicle-Monitoring-System-3/tree/experimental-esp-idf-v4-rework-network
>    is now ready for greater review. Most of the warnings are gone, and I'll
>    need some help for the last ones.
>
>    - I created a draft PR
>    https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/806
>    . Not that I want to merge it as-is nor soon, but it may help the review
>    process to have some central discussion point.
>
>    - I'll now switch to ESP-IDFv5 porting - on a new branch (based on the
>    v4) :
>    https://github.com/llange/Open-Vehicle-Monitoring-System-3/tree/experimental-esp-idf-v5
>    (empty at the moment).
>
>
> Regards,
>
> Le 23/01/2023 à 22:19, Ludovic LANGE a écrit :
>
> Michael,
>
> Thanks for your help. FYI I've just pushed the last touches to the vehicle
> components which are now all converted to cmake and "should work". Still
> compiles under 3.3.x.
>
> A new things that need to be checked : in
> https://github.com/llange/Open-Vehicle-Monitoring-System-3/commit/f31117a48d4e293b2be01e8b48c1d580d0af834e
> i had to replace `INT` with another type. This time I choose `int32_t` but
> I'm not really sure because I could not find a way to see where and how INT
> was defined. I could only verify (by compilation tricks) that sizeof(INT)
> == sizeof(int32_t).
>
> Regarding cmake, I understand your concern with dependencies, and I did
> have the same reaction. There certainly is a way - I did not spend too much
> time on it. We will however have some "bugs" in the current state of things
> when disabling components in menuconfig where a dependency was implied and
> is now exposed. I'll let the cmake experts guide us :-)
>
> I'll now work on the compilation warnings.
>
> Btw, another thing that I need to mention is that from 3.3 to 5.x, many
> constants (used in sdkconfig) have been renamed etc... So our sdkconfig is
> upgraded by the build toolchain when we compile with 4.x or 5.x. I
> recommend doing a backup if it has been customized - in case of.
>
> Regards,
>
> Ludovic
>
>
> Le 23/01/2023 à 21:11, Michael Balzer a écrit :
>
> Ludovic,
>
> really great & appreciated you're going ahead on this. I second skipping
> idf 4 and going directly for version 5. I'll try to get my idf 5 build
> environment up ASAP.
>
> I need to get into cmake myself as well. Needing to hard code dependencies
> doesn't sound good though, I thought cmake is supposed to be smart about
> this.
>
> I'll have a look at porting the custom crash handler to idf 5. There were
> some other additions that haven't been accepted by Espressif, IIRC. I'll
> see if I can check them as well.
>
> Regards,
> Michael
>
>
> Am 23.01.23 um 11:04 schrieb Ludovic LANGE:
>
> Hi list,
>
> I am interested in upgrading our codebase to a latest ESP-IDF version
> (preferentially v5 which has just happened end of 2022 - Happy New Year !
> :-)) - mostly to be able to use some components for which a backport seems
> difficult (for example :
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/752
> ), and as a matter of balance between stability vs latest bugfixes,
> security fixes, new features etc...
>
> In the past it seems that some endeavours already set the basis to make
> this happen (
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/263,
> https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/360
> ).
>
> I'm now trying to revive this effort, and after some work managed to have:
>
>    - OVMS codebase building under v4.x using Makefile
>    - OVMS codebase building under v4.x using cmake / idf.py (not 100%
>    finished, multiple vehicle_* components are missing)
>
> (*WARNING: some important features of OVMS have been deactivated for the
> moment, see end of post)*
>
> The produced binaries seem to "work on my machine board", as they end
> with a "OVMS#" prompt and a few (very) basic tests look OK (to be honest,
> unfortunately no real tests have been done except to check that the binary
> does not crash during boot and a few seconds of run, checking that WiFi /
> web Interface is OK, sdcard is OK - but very basic)
>
>
> You'll find the changes here :
> https://github.com/llange/Open-Vehicle-Monitoring-System-3/tree/experimental-esp-idf-v4-rework-network
> (In the README.md, there is a short notice at the top with a few useful
> informations)
>
>
> I'm now reaching out here mostly for :
>
>    - gathering feedback both around the effort (is it worth pursuing this
>    goal ?) and the implementation (do my CMakeLists.txt look weird ? yes they
>    do.)
>    - having volunteers using those builds on real-world workload (which I
>    cannot do at the moment) and finding bugs
>    - guidance and fixes regarding the CMakeLists.txt files
>
>
> My next steps could be (of course, subject to feedback) :
>
>    1. Have 100% of the cmake implementation finished (all the missing
>    vehicle_* components) -> easy (only missing time)
>    2. Work on the warnings (currently disabled in the builds) and try to
>    fix as much as possible
>    3. Iron out all the bugs, improve the dev documentation -> need help
>    on finding them with real-world use
>    4. 1st goal is to propose this result as a merge request (either as a
>    whole, or split depending on feedback - ideally I would like to have a
>    first (big) MR accepted quickly so that people can play with it, and
>    deliver incremental updates as expected)
>    5. then work on the v5 and do the same.
>
>
> (Please note that my cmake skills are inexistent - so be careful when
> opening those files to not hurt yourself.)
>
> # The guidelines I tried to follow:
>
>    - Stay compatible with our 3.3.x branch which should still compile
>    without any issue nor loss of functionality
>    - Keep Makefiles running for 3.3.x (and 4.x if possible)
>    - When possible, isolate differences (between 3.x, 4.x, 5.x) by
>    testing the versions of ESP-IDF.
>    - Focus on 5.x (if there are impossibilities, let's make it easier to
>    build v5.x than v4.x)
>    -
>    https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32/api-reference/network/tcpip_adapter_migration.html
>    -
>    https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32/api-guides/build-system.html#migrating-from-esp-idf-gnu-make-system
>    -
>    https://docs.espressif.com/projects/esp-idf/en/latest/esp32/migration-guides/release-5.x/5.0/index.html
>    -
>    https://docs.espressif.com/projects/esp-idf/en/latest/esp32/migration-guides/release-5.x/5.1/index.html
>
>
> # Missing features:
>
> In the course of this work I had to make some decisions to speed up the
> work  - that will need to be acted upon:
>
>    - I disabled the crash handler (`xt_set_error_handler_callback` and
>    `esp_task_wdt_get_trigger_tasknames`) for the moment, we need to decide
>    whether we "fork" ESP-IDF again to port it ; or if the new APIs are enough
>    to (partially ?) reimplement it
>    - There is a crash in `OvmsConsole::Poll` which I did not analyse
>    (yet) and which I worked around by declaring a variable static.
>    - I decided to transform our local copies of `wolfssh` and `wolfssl`
>    in submodules (and move them one level below in terms of directories) -
>    mainly to be able to have a CMakeLists.txt different from the upstream one.
>    In the process, I "lost" one of our patches :
>    https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/commit/51444539047daef7bd2accb23ef40d1bc14fdb20
>    and we need to decide how to handle this.
>    - A lot of dependencies are now explicitly (hard-)coded in the
>    CMakeLists.txt - which may, or may not be a good thing. Let's discuss it.
>    - I had to change a set of defines into a header generation (in
>    ovms_webserver) because I was unable to implement those in a satisfying
>    manner in cmake. If you manage to do it, it's a win !
>    - Build warnings were disabled to keep the output clean and focus on
>    the build issues - but now they need to be re-activated.
>    - etc...
>    - Even if there is experimental support for cmake / idf.py in ESP-IDF
>    3.3.x, I did not make any effort to be compatible with it:
>       - Our own fork does not seem to include this build system
>       - It was experimental, the component API changed and I did not want
>       to have edge case for something not used at the moment.
>
>
> Let me know your comments and questions.
>
> Best regards,
>
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
>
> _______________________________________________
> OvmsDev mailing listOvmsDev at lists.openvehicles.comhttp://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
>
>
> _______________________________________________
> OvmsDev mailing listOvmsDev at lists.openvehicles.comhttp://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20230127/05a0c0ff/attachment-0001.htm>