October 2019 - OvmsDev - Open Vehicles

ESP32 CAN controller hardware bug / interrupt issue
by Michael Balzer 02 Apr '21

02 Apr '21

Second finding & commit: TL;DR: * ESP32 CAN controller is broken, delivers trashed & duplicated RX frames * we need to optimize interrupt latency Users have been reporting occasional false BMS readings on the Twizy for more than a year. The error always trashed the reading of voltage #10 and/or temperature #8, so I initially thought that must be a variant of the SPIRAM/cache bug. But the SPIRAM fix didn't help, and the bug still occured with all BMS data in internal RAM. The issue correlated with high load situations and the SIMCOM UART issue. On testing the latest changes by Marko, my car module began showing that issue regularly, so I could take a closer look. It turned out the wrong data came from the CAN bus. Checking with a CAN logger didn't show any wrong data on the bus. So I scanned our CAN driver for potential issues, found & solved some (even a serious race condition) but that didn't fix it. I then added debug dumps of all frames read, dug into the SJA1000 spec sheet and found this: The SJA1000 RX FIFO has a capacity of 64 bytes. It's supposed to work as a ring buffer, with frames wrapping around as necessary. If no space is left in the FIFO for a new frame coming in, that frame should be discarded completely, not counted, just the overflow indicator should be set & the according interrupt be generated. Reality differs. The frame causing the overflow is added to the FIFO partially. It's also counted both in the RMC register and indicated by RBS as a valid frame when retrieving the FIFO contents. The read buffer delivers the partial frame + some trashed bytes up to the nominal frame length. In the BMS frame sequence it would normally trash bytes 7 & 8 on the sixth frame of length 8 in a sequence. A standard frame needs a 3 byte header + the data bytes in the FIFO, so 6 frames would add up to 66 bytes => 64 + 2 trash. The first trash byte normally is "08", the second "84" or "2a" or sometimes "ab", possibly some internal SJA1000 data. Better yet, if you rely on the indicators of the SJA1000 when reading the FIFO, the SJA1000 will repeat the first frame within the FIFO after the invalid frame. Mostly once or twice, and sometimes 7 times or more: inv_msg: framecnt=13, invindex=6 inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 24 04 00 00 11 40 10 22 37 55 00 37 | 4..?........$....@."7U.7 inv_msg: 25 04 00 00 0a 1b 44 ff fe 4e 01 26 | 4..?........%.....D..N.& inv_msg: 54 05 00 00 37 37 37 37 37 37 37 00 | 4..?........T...7777777. inv_msg: 56 05 00 00 31 63 14 31 53 14 31 4a | 4..?........V...1c.1S.1J inv_msg: 57 05 00 00 31 43 14 31 53 15 *08 2a* | 4..?........W...1C.1S..* ^^^^^ trashed bytes … followed by 7 repetitions of the first frame: inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n If you had occasional strange false readings leading to strange effects, this may be the explanation. Note: the SJA1000 FIFO bug may need specific circumstance to occur in addition to the overflow, maybe the overflow happening on a specific byte position in the FIFO -- I haven't tried to determine that. I've tried clearing the overflow ASAP, looping on the IR indicator and looping on the RMC, the fault is always the same. The workaround I've now done is adding up the message lengths read during a FIFO run and discard all frames exceeding the 64 byte border. Not pretty, but functional. That keeps the trashed frames from being forwarded to the OVMS application level. As with the SIMCOM UART, these overflows wouldn't happen if the interrupt handler would react fast enough. It seems we've got a general interrupt latency issue. And just as with the SIMCOM I saw a reduction of overflows when allocating the CAN IRQ on core #0, another one with raising the IRQ level to 3, so that's also part of my commit. The esp-idf CAN driver apparently has the same issue even in the latest release, I'll contact the author about this or post another issue to Espressif. So our interrupt allocation is now: Task Run# Lvl ISR# Usage Source ---------------------------------------------------------------------------------------- - 0 1 0 RTC Core ETS_RTC_CORE_INTR_SOURCE esp_timer 0 1 0 RTOS esp_timer ETS_TIMER2_INTR_SOURCE ipc0 0 1 0 RTOS Scheduler Core0 ETS_FROM_CPU_INTR0_SOURCE ipc1 1 1 1 RTOS Scheduler Core1 ETS_FROM_CPU_INTR1_SOURCE main 0 1 0 RTOS Watchdog ETS_TG0_WDT_LEVEL_INTR_SOURCE OVMS Events 1 1 1 Peripherals (GPIO ISR) ETS_GPIO_INTR_SOURCE OVMS Events 1 1 1 MAX7317 (spi_nodma) ETS_SPI3_INTR_SOURCE *esp32can init 0 3 0 ESP32 CAN ETS_CAN_INTR_SOURCE** **OVMS SIMCOM 0 2 0 SIMCOM UART ETS_UART1_INTR_SOURCE* OVMS Events 1 1 1 USB console (UART) ETS_UART0_INTR_SOURCE OVMS Events 1 1 1 SD card ETS_SDIO_HOST_INTR_SOURCE ---------------------------------------------------------------------------------------- I think we need to find the source of the interrupt latency and/or find some better scheme to distribute the tasks & interrupts over the cores. I still don't know why core #1 already got an issue with just our 6 interrupts. This may be related to the SPIRAM fix adding overhead, but I've seen the FIFO overflows before applying the fix, and the CAN issue also has been present for over a year. My working hypothesis is some task/driver disabling interrupts for too often / too long, or non-IRAM ISRs getting delayed for too often / too long. And it seems this is an issue especially on core #1. Please verify & comment. Regards, Michael PS: a little side mystery: the netmanager task starts with priority 5, then gets raised to 22 (probably a mutex priority inheritance), but never gets lowered back to 5 afterwards. A FreeRTOS bug? -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 11

WifiConsole V1.0
by Michael Balzer 04 Mar '20

04 Mar '20

FYI: I've just implemented an OVMS v3 style replacement for the SimpleConsole: https://github.com/dexterbg/WifiConsole V3 makes it pretty easy now to add stuff like this. It's using a script plugin to provide custom functions via the web API. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

5 28

Scripting - missing integrations
by Mark Webb-Johnson 10 Jan '20

10 Jan '20

Michael’s work has brought up an issue that we seem to be missing some javascript integrations. We have: OvmsCommand OvmsLocation OvmsMetrics OvmsVehicle PubSub (to ovms events) Obvious ones missing are: OvmsConfig OvmsWebServer I can do OvmsConfig, but one question: Do we allow scripts access to protected config values? I think not (as that would void the entire protection system with a ’script eval JSON.print(…)', but is a limitation). Comments? It would be interesting to see scripts extend the OvmsWebServer, so urls could be mapped to scripts - but not sure of the practical use of that vs plugins. Regards, Mark.

2 3

Locations and scripts
by ovms 03 Dec '19

03 Dec '19

Hi I would like to run scripts based on location. The scripts work fine, but the location service stops working. I guess the location service requires CAN1 to be working? Since CAN1 stops receiving after a short period of time, the location service also stops. Is there a way to bypass this so the location service does not depend on CAN1 working? Kind regards, Stein Arne Sordal

7 61

Android app updated, lost my car
by Greg D. 25 Nov '19

25 Nov '19

Last night the Android app updated on my phone. I'm still getting alerts from the car, but the only car shown on the app is the "Demonstration car", which of course, isn't active. I've added the car back, and it seems to be working. So, just FYI. Greg

9 15

SIMCOM UART interrupt issue & optimization
by Michael Balzer 01 Nov '19

01 Nov '19

Everyone, I've pushed another optimization for the SIMCOM component that reduces the UART hardware FIFO overflows (https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/274) to nearly zero by changing core associations. TL;DR: * you also should apply the sdkconfig changes from the commit * we need to optimize interrupt latency Analyzing our current core usage I found we've been running nearly all interrupt handlers on core #1. That's because interrupts are bound to the core that allocates them, and after system init, the housekeeper created all peripheral components on core #1. This has been our interrupt allocation up to now: Task Run# Lvl ISR# Usage Source ---------------------------------------------------------------------------------------- - 0 1 0 RTC Core ETS_RTC_CORE_INTR_SOURCE esp_timer 0 1 0 RTOS esp_timer ETS_TIMER2_INTR_SOURCE ipc0 0 1 0 RTOS Scheduler Core0 ETS_FROM_CPU_INTR0_SOURCE ipc1 1 1 1 RTOS Scheduler Core1 ETS_FROM_CPU_INTR1_SOURCE main 0 1 0 RTOS Watchdog ETS_TG0_WDT_LEVEL_INTR_SOURCE OVMS Events 1 1 1 Peripherals (GPIO ISR) ETS_GPIO_INTR_SOURCE OVMS Events 1 1 1 MAX7317 (spi_nodma) ETS_SPI3_INTR_SOURCE OVMS Events 1 1 1 ESP32 CAN ETS_CAN_INTR_SOURCE OVMS Events 1 1 1 SIMCOM UART ETS_UART1_INTR_SOURCE OVMS Events 1 1 1 USB console (UART) ETS_UART0_INTR_SOURCE OVMS Events 1 1 1 SD card ETS_SDIO_HOST_INTR_SOURCE ---------------------------------------------------------------------------------------- I had some advances on the FIFO overflows by allocating a level 3 interrupt for the SIMCOM UART, but not much. I then tried moving the SIMCOM interrupt handler to core #0, and that immediately reduced the overflows drastically -- most of the time. I read through the Espressif forums and issues on interrupt latency optimization and found a precious hint here: https://www.esp32.com/viewtopic.php?t=11462#p46437 /"Finally, by default the LWIP task is allowed to run on either core, and will sometimes take a// //spinlock which requires disabling interrupts. Try pinning this task to CPU 0, only"/ I did so, and the FIFO overflows disappeared completely on my bench. In my car there still are some, but very few. I also tested pinning LWIP to core #1, that also worked and may be an alternative. I then searched for more config candidates affecting interrupt performance and also disabled the interrupt backtrace feature. That saves just 4 instructions on interrupt entry, so has little effect and can be re-enabled in case we need it. I've reduced the UART IRQ level to 2 again as a support for my second finding regarding the CAN controller, will describe that in the next post. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 2

Roadster: Some alerts not showing up
by Greg D. 31 Oct '19

31 Oct '19

Hi folks (Mark, probably), Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3. The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening. Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out? Thanks, Greg

5 9

New Joiner: Jaunius
by Jaunius Kapkan 30 Oct '19

30 Oct '19

3 4

ReadTheDocs plugin needed
by Michael Balzer 30 Oct '19

30 Oct '19

Mark, I've pushed the change to include the README.md as the introduction page into the user guide. This change needs the sphinx plugin "m2r", which seems to require a change to the RTD configuration -- the build currently fails: https://readthedocs.org/projects/ovms/builds/9887081/ It seems you need to add the plugin manually in the RTD web interface. To avoid that, the RTD systems supports build configurations in the source tree, maybe that's a better option: https://docs.readthedocs.io/en/latest/config-file/index.html Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 1

Relais extension / first(?) use of prototype PCB
by Michael Balzer 30 Oct '19

30 Oct '19

…work in progress for Renault Zoe to control additional devices, currently mapped to the homelink call: https://www.youtube.com/watch?v=cqyFZ_GoE-U Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

1 0