February 2020 - OvmsDev - Open Vehicles

ESP32 CAN controller hardware bug / interrupt issue
by Michael Balzer 03 Apr '21

03 Apr '21

Second finding & commit: TL;DR: * ESP32 CAN controller is broken, delivers trashed & duplicated RX frames * we need to optimize interrupt latency Users have been reporting occasional false BMS readings on the Twizy for more than a year. The error always trashed the reading of voltage #10 and/or temperature #8, so I initially thought that must be a variant of the SPIRAM/cache bug. But the SPIRAM fix didn't help, and the bug still occured with all BMS data in internal RAM. The issue correlated with high load situations and the SIMCOM UART issue. On testing the latest changes by Marko, my car module began showing that issue regularly, so I could take a closer look. It turned out the wrong data came from the CAN bus. Checking with a CAN logger didn't show any wrong data on the bus. So I scanned our CAN driver for potential issues, found & solved some (even a serious race condition) but that didn't fix it. I then added debug dumps of all frames read, dug into the SJA1000 spec sheet and found this: The SJA1000 RX FIFO has a capacity of 64 bytes. It's supposed to work as a ring buffer, with frames wrapping around as necessary. If no space is left in the FIFO for a new frame coming in, that frame should be discarded completely, not counted, just the overflow indicator should be set & the according interrupt be generated. Reality differs. The frame causing the overflow is added to the FIFO partially. It's also counted both in the RMC register and indicated by RBS as a valid frame when retrieving the FIFO contents. The read buffer delivers the partial frame + some trashed bytes up to the nominal frame length. In the BMS frame sequence it would normally trash bytes 7 & 8 on the sixth frame of length 8 in a sequence. A standard frame needs a 3 byte header + the data bytes in the FIFO, so 6 frames would add up to 66 bytes => 64 + 2 trash. The first trash byte normally is "08", the second "84" or "2a" or sometimes "ab", possibly some internal SJA1000 data. Better yet, if you rely on the indicators of the SJA1000 when reading the FIFO, the SJA1000 will repeat the first frame within the FIFO after the invalid frame. Mostly once or twice, and sometimes 7 times or more: inv_msg: framecnt=13, invindex=6 inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 24 04 00 00 11 40 10 22 37 55 00 37 | 4..?........$....@."7U.7 inv_msg: 25 04 00 00 0a 1b 44 ff fe 4e 01 26 | 4..?........%.....D..N.& inv_msg: 54 05 00 00 37 37 37 37 37 37 37 00 | 4..?........T...7777777. inv_msg: 56 05 00 00 31 63 14 31 53 14 31 4a | 4..?........V...1c.1S.1J inv_msg: 57 05 00 00 31 43 14 31 53 15 *08 2a* | 4..?........W...1C.1S..* ^^^^^ trashed bytes … followed by 7 repetitions of the first frame: inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n If you had occasional strange false readings leading to strange effects, this may be the explanation. Note: the SJA1000 FIFO bug may need specific circumstance to occur in addition to the overflow, maybe the overflow happening on a specific byte position in the FIFO -- I haven't tried to determine that. I've tried clearing the overflow ASAP, looping on the IR indicator and looping on the RMC, the fault is always the same. The workaround I've now done is adding up the message lengths read during a FIFO run and discard all frames exceeding the 64 byte border. Not pretty, but functional. That keeps the trashed frames from being forwarded to the OVMS application level. As with the SIMCOM UART, these overflows wouldn't happen if the interrupt handler would react fast enough. It seems we've got a general interrupt latency issue. And just as with the SIMCOM I saw a reduction of overflows when allocating the CAN IRQ on core #0, another one with raising the IRQ level to 3, so that's also part of my commit. The esp-idf CAN driver apparently has the same issue even in the latest release, I'll contact the author about this or post another issue to Espressif. So our interrupt allocation is now: Task Run# Lvl ISR# Usage Source ---------------------------------------------------------------------------------------- - 0 1 0 RTC Core ETS_RTC_CORE_INTR_SOURCE esp_timer 0 1 0 RTOS esp_timer ETS_TIMER2_INTR_SOURCE ipc0 0 1 0 RTOS Scheduler Core0 ETS_FROM_CPU_INTR0_SOURCE ipc1 1 1 1 RTOS Scheduler Core1 ETS_FROM_CPU_INTR1_SOURCE main 0 1 0 RTOS Watchdog ETS_TG0_WDT_LEVEL_INTR_SOURCE OVMS Events 1 1 1 Peripherals (GPIO ISR) ETS_GPIO_INTR_SOURCE OVMS Events 1 1 1 MAX7317 (spi_nodma) ETS_SPI3_INTR_SOURCE *esp32can init 0 3 0 ESP32 CAN ETS_CAN_INTR_SOURCE** **OVMS SIMCOM 0 2 0 SIMCOM UART ETS_UART1_INTR_SOURCE* OVMS Events 1 1 1 USB console (UART) ETS_UART0_INTR_SOURCE OVMS Events 1 1 1 SD card ETS_SDIO_HOST_INTR_SOURCE ---------------------------------------------------------------------------------------- I think we need to find the source of the interrupt latency and/or find some better scheme to distribute the tasks & interrupts over the cores. I still don't know why core #1 already got an issue with just our 6 interrupts. This may be related to the SPIRAM fix adding overhead, but I've seen the FIFO overflows before applying the fix, and the CAN issue also has been present for over a year. My working hypothesis is some task/driver disabling interrupts for too often / too long, or non-IRAM ISRs getting delayed for too often / too long. And it seems this is an issue especially on core #1. Please verify & comment. Regards, Michael PS: a little side mystery: the netmanager task starts with priority 5, then gets raised to 22 (probably a mutex priority inheritance), but never gets lowered back to 5 afterwards. A FreeRTOS bug? -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 11

File logging task
by Michael Balzer 04 Jul '20

04 Jul '20

Everyone, I've pushed the long due refactoring of the file logging into a dedicated task. This removes most of the delays involved in file logging, as the fsync() and log cycling now is done in the logging task and no longer blocks overall log message processing. I stumbled upon a bad thing (tm) though we need to resolve with logging: the esp-idf logging generally involves a semaphore wait of max 10 ms (!). I wasn't aware of this because I didn't look into the source before, there also is no warning about this in the esp-idf documentation (https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/…) so I guess I'm not the only one being taken by surprise here. // Maximum time to wait for the mutex in a logging statement. #define *MAX_MUTEX_WAIT_MS 10* #define MAX_MUTEX_WAIT_TICKS ((MAX_MUTEX_WAIT_MS + portTICK_PERIOD_MS - 1) / portTICK_PERIOD_MS) … void IRAM_ATTR *esp_log_write*(esp_log_level_t level, const char* tag, const char* format, ...) { if (!s_log_mutex) { s_log_mutex = xSemaphoreCreateMutex(); } if (*xSemaphoreTake(s_log_mutex, MAX_MUTEX_WAIT_TICKS*) == pdFALSE) { return; } … That semaphore is needed to secure the log level tag cache against parallel access, so it also affects log messages of currently disabled tags and/or levels. This means we *must not* use the standard logging from timer callbacks, at no verbosity level. But we do, partially caused by my own code serving as a bad example I assume -- sorry. But I've not only found direct log calls in most vehicle timer callbacks, there also are some hidden ones from calling the framework, for example marking a notification as read can produce a log output from Cleanup() if tracing is enabled. We need to change this short to mid term. This may be the cause of those strange timer overflow issues we've seen occasionally. It also means every debug or verbose log output currently in the code involves potentially multiple semaphore waits, that's for example the case for all hex dumps logged by the SIMCOM module. Our primary option is to remove all log calls from all low level framework functions and timer callbacks. A simple & quick workaround could be to extend the ESP_LOGx macros to check if they are executed in the timer task context, if so suppress the message or call the ESP_EARLY log function instead. That will hide these log messages from the logging framework though, if using the _EARLY_ variant they only will be sent to the USB port. For a better solution I thought about adding a dedicated queue & task for the initial log submission as well. That will need quite some rework to the esp-idf (no changes for this in the current master) but would allow to continue logging as we do now. Cons: queue submission involves some overhead & RAM, printf format expansion needs to be done before the tag/level check, messages may need to be dropped if the task starves or RAM gets tight, log output to consoles will depend on a task as well. Opinions / better ideas? Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 5

VW e-Up project
by Chris van der Meijden 19 Mar '20

19 Mar '20

Dear OVMS-devs, my name is Chris van der Meijden (devmarxx) and I just started a new project on integrating the VW e-Up into OVMS. I mailed with Michael and he provided me with usefull procedure hints. I forked OVMS V3 and the Android code into my github account https://gi thub.com/devmarxx?tab=repositories and commited my changes to the code there. I also started a pull request, but there is no hurry, because my vehicle code is just a template at the moment (like the Fiat 500 e code). The Android code is complete, but Michael is quite right that it would confuse the users to suddenly see the VW e-Up when there is still no functionality, so no hurry there too. My experience on OVMS so far is, that have been playing with the code for about 4 years. It started all because at that time I could not use the Android app on my Cyanogenmod phone due to the "google services" popup. I checked out the source code, removed the google check line and voila I could use the app with my Twizy and with my server :-) Later we bought a Nissan Leaf and I came in contact with Tom Parker. I had already set up the OVMS V2 toolchain, so it was very nice that I could use Tom's code to get the climate control of the Leaf to work. Always nice mail contact with Tom. Amazing the we can work together in New Zealand and Germany without ever have met. That where good times :- ) So now we bought a VW e-Up, that will be delivered hopefully in April. So I thought now is a good time to get started with OVMS for the VW e- Up. I decided to connect to the vehicle not over the OBD connector, but through the OCU plug of the VW e-Up. By doing so I hope to bypass the complicated VW CAN TP2.0 protocoll and I will have direct access to the the VW comfort CAN. Time will tell if that is the right decission. I started a thread at the german goingelectric forum. Here I'll post my "adventures" (in German) and I hope to find some co-coders there too ht tps://www.goingelectric.de/forum/viewtopic.php?f=50&t=54620 Well .. So far my little introduction. Looking forward to do some happy coding together. Greetinx Chris

3 10

Anyone using HTTP API?
by Mark Webb-Johnson 13 Mar '20

13 Mar '20

Is anyone here using the HTTP API at all? It seems so tied to the v2 protocol, as to not be much use. Regards, Mark.

3 18

WifiConsole V1.0
by Michael Balzer 04 Mar '20

04 Mar '20

FYI: I've just implemented an OVMS v3 style replacement for the SimpleConsole: https://github.com/dexterbg/WifiConsole V3 makes it pretty easy now to add stuff like this. It's using a script plugin to provide custom functions via the web API. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

5 28

Website changes
by Mark Webb-Johnson 29 Feb '20

29 Feb '20

I’ve released a bunch of changes to the theme and design of the www.openvehicles.com <http://www.openvehicles.com/> website. Trying to make it more informative to new users. It is about half done (still got to do the pages for existing users, links, etc), but live now. Regards, Mark.

2 1

Paranoia
by Mark Webb-Johnson 24 Feb '20

24 Feb '20

Reviewing the OVMS v3 code, I found that v2 protocol paranoid support was not there. Not sure why it was overlooked (perhaps because I hate it both from a crappy encryption, as well as support headache, point of view). I’ve added it back in today. It shouldn’t affect normal module-server-app connections at all. It has had only cursory testing, but seems to work. It is enabled by ‘config set server.v2 paranoid yes’, and ensuring that the module password is the same in the app and on the module. The default is off. I suggest to keep this as a command line option, as it really makes support even more of a headache than it already is. If users know what they are doing, they can enable it themselves (and support themselves). I do notice a bug in the ovms_server.pl code when switching from normal to paranoid; the normal messages are not removed from the database (they should be) - I’ll try to address that in the new server code. Regards, Mark.

1 0

Pushover
by Tamás Kovács 20 Feb '20

20 Feb '20

I updated my fork with the main repo and now pushover not working. I send a test message, and get the following in logs: I (548904) webserver: HTTP GET /cfg/notification I (553254) webserver: HTTP POST /cfg/notification I (553284) pushover: Sending message teszt with priority 0 E (570654) mongoose: mg_ssl_mbed_log 0x3fa44e14 mbedtls_ssl_fetch_input() returned -1 (-0x0001) E (570654) mongoose: mg_ssl_mbed_log 0x3fa44e14 ssl_get_next_record() returned -1 (-0x0001) E (570654) mongoose: mg_ssl_mbed_log 0x3fa44e14 mbedtls_ssl_read_record() returned -1 (-0x0001) E (570654) mongoose: mg_ssl_if_mbed_err 0x3fa44e14 SSL error: -1 I (570654) pushover: PushoverMongooseCallback(MG_EV_CONNECT=-3:) E (570654) pushover: Connection failed I (570664) ovms-server-v2: Send MP-0 PIConnection failed

2 1

General suggestions for adding support for a new vehicle to OVMS
by Mark Webb-Johnson 19 Feb '20

19 Feb '20

I created this forum post, which some may find helpful. Please feel free to add comments to it (at the link below) to feedback your experience and suggestions. General suggestions for adding support for a new vehicle to OVMS <https://www.openvehicles.com/node/2437> Before we start, I would like to say thank you. Your time is valuable and your contributions are appreciated. OVMS is an open source community project, and we rely on our developers to take this project forward. New vehicle support comes from the owners of those vehicles developing that support, along with the help of other experienced developers working on other vehicles or framework components. We work together to extend the project. The first step is to sign up for the developer's mailing list, and let people here know that you are working on a particular car. Perhaps other owners are also interested and can help. Next is physical hardware reconnaissance. You need to find out what sort of communications buses your car has, what vehicle modules (ECUs) are connected, what speed the buses run at, where the diagnostic connectors are, and what each pin does. The OBDII connector is pretty standard in modern cars, but is often not the only connector. It is often common to see many different types of communication buses such as CAN, LIN, K-Line, and others. Of particular interest is any communication buses connecting the car to its user visible displays or telematics modules. Google is your friend here. You can also look for service manuals, or talk to your friendly mechanic. The absolute last resort is to get out a multimeter, trace cables and measure signals. For OVMS, you are in particular looking for CAN bus networks. The next step is to look for documented signals on those communications buses. Network traffic is commonly either polled (the diagnostic tool sends a request for a particular piece of information, and the vehicle module replies with the current value), or active (the information is broadcast repeatedly, usually at a set interval). Most cars use a combination of both. Sometimes these are published as online resources on enthusiast websites, and other times the manufacturer or diagnostic tool supplier has the information. There are standardised formats for this (such as DBC for active networks, and PIDs for polled). You then need to look for existing diagnostic screens and information. As well as the general user interface of the car (temperature displays, speedometer, etc), there may also be maintenance screens and lower level diagnostics. These are incredibly useful for the work you will be doing. OK, now it is time to step away from the Internet and go to the actual car. You are going to want to record the traffic on the car communication buses, while doing different activities (such as charging the car, parked idle, driving, reversing, turning, etc). We call these "CAN bus dumps". Write down notes next to each dump, describing what you were doing, where, when, and what the key metrics shown in the car were (state of charge, odometer, speed, type of charge, charge rate, etc). The tools I use for this are OVMS (with an appropriate cable to connect it to the car networks) and SavvyCAN (a cross platform application you can run on a laptop) - you can connect the two over WiFi. It is a good idea to have the car reachable by wifi, which allows you to do a lot of work from inside your house (rather than in the driver's seat with the laptop propped up on the steering wheel). Now is the time to look for specific metrics. You can find the full list in the OVMS documentation, but we suggest you start with basic things like car on/off state, gear selection, speed, State of Charge (SOC), range displays, etc. I always go for SOC as my first target, as it gives instant gratification. At this stage, you will be looking for a particular metric, and you are going to be comparing what signals are on the car's communication buses against what is displayed on the car's diagnostic display. This may not be a 1-to-1 relationship. For example, the odometer may display to the nearest kilometre, but the metric on the car communication bus may be in 1/10ths of a mile. Other metrics are in floating point, or different byte orders. Often the change in a metric is the biggest clue - a SOC would increase during a charge and decrease during a drive, for example. SavvyCAN can help with this. When a door opens, you would expect the appropriate signal on the CAN bus to change appropriately. Once you've found a few metrics, it is time to add support. You will need some programming knowledge here, and be very familiar with computers. We use the C++ language, which is like C on steroids - vehicle modules are not complex and there are plenty of example vehicles to learn from. Setup your build environment (the developer's guide has the detail for this), clone the OVMS firmware to your own GitHub account, and try to build the standard firmware and flash it to your module. Then, add a stub for your vehicle. Just a piece of code that allows you to select the vehicle but does nothing (you will create components/vehicle_ and modify main/Kconfig to add it as an optional vehicle). Modify your vehicle module to initialise the appropriate CAN bus support, and decode the metrics you know. Flash it to your module, then fire up the App and get the instant gratification of being able to see your battery level (for example). You won't have every metric supported at this time - just get a few working and you'll be encouraged to find more. Now it is time to contribute. Add, commit, and publish your changes to your own GitHub repository, then use the GitHub tools to send in a pull request (for us to merge your code back). Please don't forget to add some documentation, so others can share in what you have achieved. From that point on, it is an iterative journey. Discovering and adding more metrics, and then starting to look for things like door lock/unlock functionality, etc. I find this a fascinating journey - the process of discovery is to me even more rewarding than the final vehicle support. Again, thank you for your contributions, and happy hacking! Mark.

2 1

SSL Support
by Mark Webb-Johnson 18 Feb '20

18 Feb '20

Given that we use the mongoose library for most of our stuff, adding SSL support should not be hard. This would finally bring strong encryption and server side authentication. Given the number of attacks now on IoT devices that would not be a bad thing. I think what we need is: A set of helper functions to make it easier for components to use SSL. Build on top of mongoose. A way to manage a list of trusted Certificate Authorities, including adding to the trusted list via: Components providing Certificate Authorities in firmware. Certificate Authorities in configuration. Extensions to ovms_server_v2 to support an SSL connection option. Extensions to ovms_server_v3 to support an SSL connection option. Extensions to the Ovms Server v2 code to support an SSL connection listener. Extensions to the iOS App to support an SSL connection option. Extensions to the Android App to support an SSL connection option. Migration of any components already supporting SSL to this new standardised approach. Then we can open up the discussion of the whole thing of passwords. We have far too many of these at the moment (user account+password, vehicle ID, server password, module password, hologram account+password, etc). Once we have an encrypted connection, we don’t need to use the password for encryption, but merely for authentication. That simplifies things, as we can perhaps just use the user account+password for most things (giving access to all vehicles registered under that user account - in a similar way to MQTT does it already for ovms_server_v3). I will take on the majority of this project. I can do #1, #2, #3, #4, #5, and #6). If anyone has any feedback on requirements, please let me know. Regards, Mark.

3 15