December 2020 - OvmsDev - Open Vehicles

Netmanager priority issue solved
by Michael Balzer 05 Jul '23

05 Jul '23

TL;DR: you need to pull my latest esp-idf changes. I've finally found & solved the strange priority changes for our netmanager task (i.e. being raised suddenly from origin 5 to 18/22): the bug was in the esp-idf posix threads mutex implementation. I had suspected the mutex priority inheritance for a while, so added a way to retrieve the internal mutex hold count for our task list. Using this I noticed the hold count would always & only raise by 2 whenever any kind of mongoose connection was closed. That lead me to checking the thread concurrency protection for mongoose mbufs, because every mongoose connection has two mbufs associated (rx & tx), each having a posix mutex. The bug was: posix mutexes were deleted after locking (taking) them. FreeRTOS mutexes must not be deleted while being taken, that breaks the priority inheritance (more precisely the disinheritance), with the visible effect being the mutex hold count not returning to zero. As a side effect, this may also solve the strange event task starvations (hope so…). I was investigating this as I suspected some busy loop in the netmanager context. With the netman running at prio 22, that would effectively block almost all other processing including the timer service. I've found & fixed one potential busy loop trigger in the netman that would have been caused by the netman task still running while all interfaces had been lost -- not sure if that could happen, but it would explain the effects. So please watch your crash debug info & report if the issue still turns up. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

5 19

ESP32 CAN controller hardware bug / interrupt issue
by Michael Balzer 03 Apr '21

03 Apr '21

Second finding & commit: TL;DR: * ESP32 CAN controller is broken, delivers trashed & duplicated RX frames * we need to optimize interrupt latency Users have been reporting occasional false BMS readings on the Twizy for more than a year. The error always trashed the reading of voltage #10 and/or temperature #8, so I initially thought that must be a variant of the SPIRAM/cache bug. But the SPIRAM fix didn't help, and the bug still occured with all BMS data in internal RAM. The issue correlated with high load situations and the SIMCOM UART issue. On testing the latest changes by Marko, my car module began showing that issue regularly, so I could take a closer look. It turned out the wrong data came from the CAN bus. Checking with a CAN logger didn't show any wrong data on the bus. So I scanned our CAN driver for potential issues, found & solved some (even a serious race condition) but that didn't fix it. I then added debug dumps of all frames read, dug into the SJA1000 spec sheet and found this: The SJA1000 RX FIFO has a capacity of 64 bytes. It's supposed to work as a ring buffer, with frames wrapping around as necessary. If no space is left in the FIFO for a new frame coming in, that frame should be discarded completely, not counted, just the overflow indicator should be set & the according interrupt be generated. Reality differs. The frame causing the overflow is added to the FIFO partially. It's also counted both in the RMC register and indicated by RBS as a valid frame when retrieving the FIFO contents. The read buffer delivers the partial frame + some trashed bytes up to the nominal frame length. In the BMS frame sequence it would normally trash bytes 7 & 8 on the sixth frame of length 8 in a sequence. A standard frame needs a 3 byte header + the data bytes in the FIFO, so 6 frames would add up to 66 bytes => 64 + 2 trash. The first trash byte normally is "08", the second "84" or "2a" or sometimes "ab", possibly some internal SJA1000 data. Better yet, if you rely on the indicators of the SJA1000 when reading the FIFO, the SJA1000 will repeat the first frame within the FIFO after the invalid frame. Mostly once or twice, and sometimes 7 times or more: inv_msg: framecnt=13, invindex=6 inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 24 04 00 00 11 40 10 22 37 55 00 37 | 4..?........$....@."7U.7 inv_msg: 25 04 00 00 0a 1b 44 ff fe 4e 01 26 | 4..?........%.....D..N.& inv_msg: 54 05 00 00 37 37 37 37 37 37 37 00 | 4..?........T...7777777. inv_msg: 56 05 00 00 31 63 14 31 53 14 31 4a | 4..?........V...1c.1S.1J inv_msg: 57 05 00 00 31 43 14 31 53 15 *08 2a* | 4..?........W...1C.1S..* ^^^^^ trashed bytes … followed by 7 repetitions of the first frame: inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n inv_msg: 55 01 00 00 07 98 50 54 54 20 00 6e | 4..?........U.....PTT .n If you had occasional strange false readings leading to strange effects, this may be the explanation. Note: the SJA1000 FIFO bug may need specific circumstance to occur in addition to the overflow, maybe the overflow happening on a specific byte position in the FIFO -- I haven't tried to determine that. I've tried clearing the overflow ASAP, looping on the IR indicator and looping on the RMC, the fault is always the same. The workaround I've now done is adding up the message lengths read during a FIFO run and discard all frames exceeding the 64 byte border. Not pretty, but functional. That keeps the trashed frames from being forwarded to the OVMS application level. As with the SIMCOM UART, these overflows wouldn't happen if the interrupt handler would react fast enough. It seems we've got a general interrupt latency issue. And just as with the SIMCOM I saw a reduction of overflows when allocating the CAN IRQ on core #0, another one with raising the IRQ level to 3, so that's also part of my commit. The esp-idf CAN driver apparently has the same issue even in the latest release, I'll contact the author about this or post another issue to Espressif. So our interrupt allocation is now: Task Run# Lvl ISR# Usage Source ---------------------------------------------------------------------------------------- - 0 1 0 RTC Core ETS_RTC_CORE_INTR_SOURCE esp_timer 0 1 0 RTOS esp_timer ETS_TIMER2_INTR_SOURCE ipc0 0 1 0 RTOS Scheduler Core0 ETS_FROM_CPU_INTR0_SOURCE ipc1 1 1 1 RTOS Scheduler Core1 ETS_FROM_CPU_INTR1_SOURCE main 0 1 0 RTOS Watchdog ETS_TG0_WDT_LEVEL_INTR_SOURCE OVMS Events 1 1 1 Peripherals (GPIO ISR) ETS_GPIO_INTR_SOURCE OVMS Events 1 1 1 MAX7317 (spi_nodma) ETS_SPI3_INTR_SOURCE *esp32can init 0 3 0 ESP32 CAN ETS_CAN_INTR_SOURCE** **OVMS SIMCOM 0 2 0 SIMCOM UART ETS_UART1_INTR_SOURCE* OVMS Events 1 1 1 USB console (UART) ETS_UART0_INTR_SOURCE OVMS Events 1 1 1 SD card ETS_SDIO_HOST_INTR_SOURCE ---------------------------------------------------------------------------------------- I think we need to find the source of the interrupt latency and/or find some better scheme to distribute the tasks & interrupts over the cores. I still don't know why core #1 already got an issue with just our 6 interrupts. This may be related to the SPIRAM fix adding overhead, but I've seen the FIFO overflows before applying the fix, and the CAN issue also has been present for over a year. My working hypothesis is some task/driver disabling interrupts for too often / too long, or non-IRAM ISRs getting delayed for too often / too long. And it seems this is an issue especially on core #1. Please verify & comment. Regards, Michael PS: a little side mystery: the netmanager task starts with priority 5, then gets raised to 22 (probably a mutex priority inheritance), but never gets lowered back to 5 afterwards. A FreeRTOS bug? -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 11

Hi from a new comer
by didier＠ernotte.com 18 Jan '21

18 Jan '21

Hi, I am new here, just received my OVMS device, and started to play with it.I have a couple of questions : 1) Is there anyone around that has started the integration with the Jaguar IPace ? or any JLR products ? 2) I can see that there is multiple CAN bus on the plug (can1, can2, can3). Does the car use the 3 of them, only one, or a mix and we have to listen to all of them ?3) Is there a detailed step-by-step process to start investigating the data on the bus ? I just plug the cable and expected to see "something" in the RE console, in "discover" mode, out of the box, without results. No a good start... :-( Didier

2 7

Metrics: Cabin Fan
by jaunius＠gmx.com 06 Jan '21

06 Jan '21

Hi, I would like to bring back this topic. Back in 2019 we discussed on how the standard metrics for climate and cabin should look like. I remember we initially discussed of adding cabinfan as number of steps, but latter decided to go with percentage. However as it is still better for the user to show the same scale as in car, we need to relay the number of steps available. I added this metric a while to Nissan Leaf. And it works great for me. So would suggest to add it to the standard metrics as well like this. v.e.cabinfansteps [integer] Number of cabin fan speed steps. Kind Regards, Jaunius Sent using the GMX mail app

2 2

Addition to OVMS Developer Guide
by Stephen Casner 05 Jan '21

05 Jan '21

At Mark's request I have added a description of how to add commands to the console command interpreter including the usage string syntax. This should be considered a draft at this point; questions and comments are welcome. I added this under Vehicle Firmware Overview, but that might not be the right place. There is no general description of the core firmware. -- Steve

3 22

Investigating crashes. And, OVMS stopped collecting data
by Steve Davies 01 Jan '21

01 Jan '21

Hi, I went for a scenic drive partly to test my OVMS I3 stuff and feeding data to ABRP. When I got home I found that ABRP had my drive ending say 15km from home. And when I checked OVMS it also thought I was in the same spot. [6315S] v.p.altitude 65.4m [6315S] v.p.direction 97.5° [6315S] v.p.gpshdop 1 [9999S] v.p.gpslock yes [6315S] v.p.gpsmode AN [6315S] v.p.gpsspeed 18.7052km/h [6315S] v.p.latitude -34.0808 [-----] v.p.location [6315S] v.p.longitude 18.4327 [5282S] v.p.odometer 14901km [6315S] v.p.satcount 9 [5195S] v.p.speed 0km/h [5195S] v.p.trip 85km You can see that the metrics are very old (I show more digits in my version). I then looked further and no metrics seemed to be updating - for instance the m.time.utc wasn't updating. I was starting to look at it when my OVMS rebooted (which seems to happen sometimes. I've also had the logging stop - I noticed one less task so I presume that's related). Last boot was 127 second(s) ago Time at boot: 2020-12-31 19:10:09 SAST This is reset #8 since last power cycle Detected boot reason: Crash (12/12) Reset reason: Exception/panic (4) Crash counters: 1 total, 0 early Last crash: LoadProhibited exception on core 1 Registers: PC : 0x400f0161 PS : 0x00060a30 A0 : 0x800f023a A1 : 0x3ffe5920 A2 : 0x3ffb5738 A3 : 0x00000000 A4 : 0x3f8566a8 A5 : 0x00000001 A6 : 0x3ffe5990 A7 : 0x3f8567d0 A8 : 0x800f015c A9 : 0x3ffe5910 A10 : 0x00000000 A11 : 0x3f414cb1 A12 : 0x00000002 A13 : 0x00000001 A14 : 0x000000fe A15 : 0x3ffd4f90 SAR : 0x00000004 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000000 LBEG : 0x4008bcf1 LEND : 0x4008bd01 LCOUNT : 0xfffffffd Backtrace: 0x400f0161 0x400f0237 0x400fe87f 0x40103d4b 0x40103f02 0x400fe8af 0x4013a432 Version: 3.2.015-267-g39dba375-dirty/ota_1/main (build idf v3.3.4-845-gd59ed8bba-dirty Dec 26 2020 15:37:52) After this crash and restart m.time.utc was empty. I rebooted and its still missing - does it only appear once NTP syncs or something? Also I have no GPS position, gpsmode or anything like that (but I have a 3G session with the hologram sim and roaming on one of the local cellular operators). I've made sure the GPS puck is out in the open and I guess I'll wait - does the time come from the GPS? And any ideas why I'm not getting a GPS lock suddenly? Apart from that - how do I debug crashes? I went looking for some sort of loader memory map in the build directory to try turn the PC and backtrace addresses into something more useful but didn't pick up anything. Thanks Steve

3 6

Curious message about GreyPeter/Open-Vehicle-Monitoring-System-3
by Stephen Casner 30 Dec '20

30 Dec '20

This morning I received the below automated email from GitGuardian, which is another thing I didn't know existed. The warning does not identify a real problem since the RSA Private Key in question is an example server key included as part of the WolfSSL package that I imported into our OVMS repository in 2017. However, the thing that's curious is that this is in reference to repository GreyPeter/Open-Vehicle-Monitoring-System-3 which does not appear as a fork of the openvehicles repository but rather some other kind of copy. If GreyPeter is listening here, was that intentional? -- Steve ---------- Forwarded message ---------- Date: Tue, 29 Dec 2020 05:57:23 +0000 From: GitGuardian <security(a)mail.gitguardian.com> To: casner(a)acm.org Subject: [GreyPeter/Open-Vehicle-Monitoring-System-3] RSA Private Key exposed on GitHub GitGuardian security alert GitGuardian has detected the following RSA Private Key exposed within your GitHub account. Details - Secret type: RSA Private Key - Repository: GreyPeter/Open-Vehicle-Monitoring-System-3 - Pushed date: 2020-12-29T05:51:58+0000 Protect your GitHub repos GitGuardian is an automated secrets detection service trusted by 150,000 developers worldwide. Unsubscribe from these alerts. [eJyVUk2P3CAM_TWTyyojvgmHHHalTg9V1VW36jUyYBLUfIlhRpp_X2dH6r0IkI0xfu-Z2DOluG9gz3_wMUx5 nGZaFeNwK3M_1bpfT_L1JC40x1ynmz-HbSHna8HHO1YsZP_YcW1_45TDjO33bc11K3kd24_HteLSSrpCSUuu hwECjQ3RgEwmRCud7mQnpI5SuRQ1dJEr59xJyJhTahFTsJYZsIx5B8iC9UZHSL4znhntlA8U5ZJxB96HJFxg 3nIZvGLKYJN7wQTjQjimtRXyzM-GfRHGvimrO3PR_HJSbIE8n4ngeIMSM6wHy4bA7zNUHFZYsP_n7WUZ7qLB O651iMdJ2soClVT7rNVy0Qr3i2lSTvNj607ijdFoYMZShxx7RBG9Zp2JILwLB3BOaDiyqDtOhLpE6IXVUYAP OnLHuETiq7kLEXxT-gDXleRXDMJy3srY1H4vWxzA37n6dPL9AHc0tlyhmXpBiaAAjbGOaimdkiYQ3EYVwDpo IjU0UPOGmK_E9fEk_vPj9eX9-djLN3w0BfftGfmfT_AXbLjFZw]

1 0

Decoding "OBD/UDS" errors
by Steve Davies 29 Dec '20

29 Dec '20

Hi, For some of my requests I get errors: steve@maximus ovms-logs % grep 'process.*error' log.20201228 | sed -e 's/^.*vehicle/vehicle/' | sort | uniq -c 1 vehicle: PollerReceive[607F1]: process OBD/UDS error 22(6335) code=21 4 vehicle: PollerReceive[607F1]: process OBD/UDS error 22(DD68) code=21 4 vehicle: PollerReceive[607F1]: process OBD/UDS error 22(DD69) code=21 285 vehicle: PollerReceive[614F1]: process OBD/UDS error 22(DEF7) code=12 304 vehicle: PollerReceive[614F1]: process OBD/UDS error 22(DEFA) code=31 39 vehicle: PollerReceive[61AF1]: process OBD/UDS error 22(DE8A) code=31 148 vehicle: PollerReceive[640F1]: process OBD/UDS error 22(D130) code=22 I found a list here: https://automotive.wiki/index.php/ISO_14229 If so: 0x21: Busy, try again 0x12: sub function not supported (problem with my PID?) 0x31: "request out of range". 0x22: "conditions not correct". Does that sound right? Thanks, Steve D

2 1

Android App V3.17.3
by Michael Balzer 29 Dec '20

29 Dec '20

…is now being processed by the Play Store. Release info: https://github.com/openvehicles/Open-Vehicle-Android/wiki/History Note: you can download the APK files of the current and previous releases from:https://github.com/openvehicles/Open-Vehicle-Android/tree/master/relea… <https://github.com/openvehicles/Open-Vehicle-Android/tree/master/release> To install an App from an APK file, you need to allow installing from "unknown sources" in your Android security configuration. <https://github.com/openvehicles/Open-Vehicle-Android/wiki/History#version-3…>Version 3.17.3 (2020-12-28) * Overall image size optimizations * Volt/Ampera: images for doors, lighting, charge port * Nicer light images for VW e-Up / Mii / Citigo * Add "car_hvac_on" flag to Update Broadcast Intent * Add car service interval info if available <https://github.com/openvehicles/Open-Vehicle-Android/wiki/History#version-3…> -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 1

Fireplace app
by Chris van der Meijden 29 Dec '20

29 Dec '20

I'm writing a little Android app with Cordova to do something similar as Tesla's "Romance mode". The app will turn climate control on or off and will show a burning log fire while the car is heating. As cordova uses HTML and Javascript you can see an online preview here: http://www.arachnon.de/fire/test.html When you click or tap on the fireplace the countdown for turning "climatecontrol" on ticks down for 17 seconds. After that the fire starts "moving" and on webkit browsers you will hear a tickling sound. When you click or tap again the tickling will stop and you will get a static image again. You will also get a red bar for 10 seconds, turning "climatecontrol" off and blocking any new commands for those 10 seconds. When you click or tap on the yellow bar you will be able to store your OVMS credentials (vehicle ID and password). Cave: this is done in plain text with window.localstorage so the password will be plain readable from the local storage within the browser on your device. The credentials are needed to communicate with the Android OVMS app, as I am planning to use the "SendCommand Broadcast Intent data" functionality of the OVMS app. I should be able to do that via this cordova plugin: https://github.com/cordova-misc/cordova-webintent I read the OVMS "SendCommand Broadcast Intent data" documentation here: https://github.com/openvehicles/Open-Vehicle-Android/wiki/SendCommand-B roadcast-Intent-data but it is not clear to me, how I can initiate the "climatecontrol on" shell command over the broadcast. So would like to ask for help on that :-) Once the app is up and running it should be usable for any vehicle with OVMS climate control implemented. Just a little gimmick. But fun :-)) Regards Chris

3 12