- OvmsDev - Open Vehicles

AutoInit Inhibit
by Mark Webb-Johnson 29 Aug '18

29 Aug '18

Last week’s 3.1.009 issue with Tesla Roadsters was interesting. Looking at the car I saw with the problem, it was booting, crashing when the vehicle module received a certain CAN bus message (triggering the issue), and rebooting. It would do this 5 times, until it his the AUTO_INIT_INHIBIT_CRASHCOUNT count, and then AutoInit inhibit would kick in, and it would end up sitting idle with nothing loaded (and no network connection). The approach worked very well, and prevented an endless reboot loop. However, we ended up with a car unable to connect to the network, and requiring a console cable and laptop to recover. I’ve been thinking about bluetooth (which is intended to provide a smartphone connection irrespective of wifi configuration), how that might work with a completely unconfigured module (for initial configuration), and I can suggest a workaround to try to improve this… At the moment, the order of autoinit is: External 12V Wifi Modem SIMCOM Vehicle OBD2ECU Server V2 Server V3 None of those run if the early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT (coded as a constant 5). My suggestion is to change this as follows: Wifi Bluetooth Server V2 Server V3 Modem SIMCOM External 12V Vehicle ODB2ECU But also to change that logic that: #8 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT #7 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+1 #6 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+2 etc. This will mean that once we hit the AUTO_INIT_INHIBIT_CRASHCOUNT limit, we turn off the least required module (OBD2ECU), then continue to see if that solves the problem. If not (ie; we crash and reboot), then we try turning off the Vehicle module and OBD2ECU, then continue to see if that solves the problem. etc. In the example case of the Tesla Roadster vehicle module causing the issue, this revised approach would leave us up and running with Wifi and a Server Connection (but no vehicle data). Checking the module would identify the problem quite quickly (and remotely). Starting the vehicle module manually would presumably result in a crash and we would pretty quickly get an idea of the cause (especially if we added a command to show the status of AutoInit, what was started, and what wasn’t because it is causing a crash). I think we could even issue an Alert notification in the case where autoinit recovered after inhibiting certain modules (but not all). The disadvantage is that we would crash more (up to 13 times, vs 5) in the case the system is badly messed up. What do people think? Does this make sense? Regards, Mark.

3 8

OT: A possible insight on how Tesla may have been doing telematics
by Nikolay Shishkov 27 Aug '18

27 Aug '18

This a bit off-topic and a bit questionable, but I found interesting the comment about Node.js vs Go for the backend implementation:ato̧̕m̀͡i̴̷̛c̨͝t͝҉͡h̷҉u̵̶m͜͞b͏͝s̀́ on Twitter | | | | | | | | | | | ato̧̕m̀͡i̴̷̛c̨͝t͝҉͡h̷҉u̵̶m͜͞b͏͝s̀́ on Twitter “A former Tesla employee, who worked on their IT infrastructure, is posting in a subforum of a subforum, a littl... | | | On Monday, August 27, 2018, 2:53:52 AM GMT+2, Mark Webb-Johnson <mark(a)webb-johnson.net> wrote: Last week’s 3.1.009 issue with Tesla Roadsters was interesting. Looking at the car I saw with the problem, it was booting, crashing when the vehicle module received a certain CAN bus message (triggering the issue), and rebooting. It would do this 5 times, until it his the AUTO_INIT_INHIBIT_CRASHCOUNT count, and then AutoInit inhibit would kick in, and it would end up sitting idle with nothing loaded (and no network connection). The approach worked very well, and prevented an endless reboot loop. However, we ended up with a car unable to connect to the network, and requiring a console cable and laptop to recover. I’ve been thinking about bluetooth (which is intended to provide a smartphone connection irrespective of wifi configuration), how that might work with a completely unconfigured module (for initial configuration), and I can suggest a workaround to try to improve this… At the moment, the order of autoinit is: - External 12V - Wifi - Modem SIMCOM - Vehicle - OBD2ECU - Server V2 - Server V3 None of those run if the early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT (coded as a constant 5). My suggestion is to change this as follows: - Wifi - Bluetooth - Server V2 - Server V3 - Modem SIMCOM - External 12V - Vehicle - ODB2ECU But also to change that logic that: - #8 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT - #7 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+1 - #6 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+2 - etc. This will mean that once we hit the AUTO_INIT_INHIBIT_CRASHCOUNT limit, we turn off the least required module (OBD2ECU), then continue to see if that solves the problem. If not (ie; we crash and reboot), then we try turning off the Vehicle module and OBD2ECU, then continue to see if that solves the problem. etc. In the example case of the Tesla Roadster vehicle module causing the issue, this revised approach would leave us up and running with Wifi and a Server Connection (but no vehicle data). Checking the module would identify the problem quite quickly (and remotely). Starting the vehicle module manually would presumably result in a crash and we would pretty quickly get an idea of the cause (especially if we added a command to show the status of AutoInit, what was started, and what wasn’t because it is causing a crash). I think we could even issue an Alert notification in the case where autoinit recovered after inhibiting certain modules (but not all). The disadvantage is that we would crash more (up to 13 times, vs 5) in the case the system is badly messed up. What do people think? Does this make sense? Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev(a)lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

1 0

Config store mount failure / reformatting
by Michael Balzer 23 Aug '18

23 Aug '18

I've pulled the latest esp-idf updates, merges, builds and runs without issues. Btw, the idf now includes a CAN driver, may be worth a look. On the second app-flashing, this happened: … Wrote 2399584 bytes (1382879 compressed) at 0x00010000 in 26.1 seconds (effective 736.0 kbit/s)... Hash of data verified. Leaving... Hard resetting via RTS pin... … I (452) ovms_main: Mounting CONFIG... W (722) vfs_fat_spiflash: f_mount failed (13) I (722) vfs_fat_spiflash: Formatting FATFS partition, allocation unit size=4096 I (1152) vfs_fat_spiflash: Mounting again Initialising OVMS CONFIG within STORE :-( The first flash was perfectly OK and I haven't been able to reproduce this afterwards. I don't think this is related to idf changes, as we had some reports on lost configs before. Issue #145 also seems to be solved by the new idf, I had no more crashes during reboots. I wonder if there could be a better strategy than immediately formatting the config filesystem on a mount error. Is there any chance the mount fails due to some race condition, i.e. would it make sense to first retry mounting after a short delay? Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

3 7

SIMCom diagnostics
by Tom Parker 22 Aug '18

22 Aug '18

Hi, I've been experiencing poor cellular connectivity with my v3.0 prototype. The changes Mark made a while ago to recover from various corner cases improved things a lot but outside my house it is often disconnected even though my cell phone has reasonable signal strength. I tried to extract extra information from the modem and also force it to use 2G since my OVMS v2 is rock solid in the same location on the same carrier. I learnt some things: I'm using the possibly slightly out of date https://simcom.ee/documents/SIM5360/SIMCOM_SIM5360_ATC_EN_V0.24.pdf which doesn't require a login to get. Sending AT+CNMP=13 is documented to force it to use 2G, but instead it puts the modem into a boot loop. It does the START +CPIN: READY dance ending with PB DONE and then goes back to START. It never responds to the first AT commands we send it, and power cycling does not fix it. I went down many rabbit holes to fix this, it looks a lot like not enough power (I'm powering via USB) but adding 13V to the CAN bus port didn't fix it. Removing the simcard fixed it, but this supported the power theory which turned out be wrong. Sending AT+CNMP=2 as soon as it sends START does result in OK and pust it back into auto mode and stops the boot loop. I spent a while hacking simcom.cpp to send commands of my choice to the modem, but got tired of that and thought, there _must_ be a way to do this interactively! I traced the tx code paths and found that there are two ways: simcom tx AT+CNMP? simcom muxtx 3 AT+CNSMOD? You probably need at least verbose logging to see the responses. You need to use muxtx when in mux mode. I'm not 100% sure what muxtx does not in mux mode. After I sorted out the modem boot loop I added polling of AT+CPSI? and hooked the result up to a metric (and also logged it). I can see which cell tower I am connected two and on what frequency and protocol, below is it switching cell id and frequency: I (26562395) simcom: CPSI: +CPSI: WCDMA,Online,530-01,0x0778,xxx,WCDMA IMT 2000,306,10588,0,9.0,95,24,20,500 I (26592445) simcom: CPSI: +CPSI: WCDMA,Online,530-01,0x0778,yyy,WCDMA 900,62,3086,0,5.0,94,27,21,500 Obviously smashing all of that into one metric isn't a great thing to do, and probably not all of it is needed in metrics? The cell identifier, protocol and frequency look useful. Simcom's documentation for this says If camping on a 2G cell: +CPSI:<System Mode>,<Operation Mode>,<MCC>-<MNC><LAC>,<Cell ID>,<Absolute RF Ch Num>,<RxLev>,<Track LO Adjust>,<C1-C2> If camping on a 3G cell: +CPSI:<System Mode>,<Operation Mode>,<MCC>-<MNC>,<LAC>,<Cell ID>,<Frequency Band>,<PSC>,<Freq>,<SSC>,<EC/IO>,< RSCP >,<Qual>,<RxLev>,<TXPWR> I'm not sure what else is useful to parse into metrics? Maybe the network numbers for roaming? See page 140 of the pdf mentioned above for more details. I haven't yet used this information to try to tell what is wrong with my poor connectivity problem. I also discovered that the modem is not stable powered by USB on my desk. It disconnects from the cell network or reboots every few minutes. It has been stable for many hours since I turned the 13.5V power supply back on.

2 1

OvmsMetricString memory lifecycle
by Tom Parker 21 Aug '18

21 Aug '18

I have what I hope is a stupid question. The additional cellular information code I wrote about yesterday worked for most of a day and then the ovms module stopped communicating with the modem. I didn't have a data logger connected so I don't know what exactly happened. When I got on the console the ovms was going through the modem power cycle loop, but never receiving data from the modem. I tried a few things and then I did a simcom status and it hung after printing a few garbage characters instead of the line of information I added to the status report yesterday. This got me to looking at how that memory is handled. I'm doing the following in simcom::StandardLineHandler() which copies the similar pattern for +CGMR and +ICCID in the same function: else if (line.compare(0, 7, "+CPSI: ") == 0) { StandardMetrics.ms_m_net_mdm_cpsi->SetValue(line.substr(7)); ESP_LOGI(TAG, "CPSI: %s", line.c_str()); } I know very little about C++ but from what I gather and reading the code * std:string.substr() creates an object on the stack pointing to memory on the heap * OvmsMetricString::SetValue(std::string value) copies the object into it's internal m_value but it still points to the memory allocated on the heap It's not clear when the heap memory for the std:string created in substr() is freed. I think it is considered for freeing at the end of simcom::StandardLineHandler as it goes out of scope. But it escapes the scope when it is passed to OvmsMetricString::SetValue() so if it were freed on function exit, we would have a use after free bug. Is it possible the compiler notices the reference escapes the scope and so doesn't free the string? If it does, then when does it get freed? It surely can't be freed when it replaced by the new value inside OvmsMetricString::SetValue() -- at first thought, this pattern without further hints from the programmer would make it too easy to end up with double free bugs or memory leaks. If it isn't freed automatically and it isn't freed manually then it looks like we have a memory leak. Looking at the memory logs that the housekeeping message gives, it doesn't look like I have a memory leak, so is there a use after free? If there is, why doesn't it cause problems all the time? There are a lot of string metrics and at least 2 that use exactly the same pattern I used. My changes are at https://github.com/carrott/Open-Vehicle-Monitoring-System-3/tree/cell-info

3 3

esp-idf update
by Michael Balzer 21 Aug '18

21 Aug '18

I've pushed the merge with the latest esp-idf master to our repository. I've been running this build for about a day without issues. It contains some new configurations, all defaults can be accepted. Don't forget to do a "git submodule update" after pulling. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 1

Lost configuration
by Tom Parker 18 Aug '18

18 Aug '18

Hi, My OVMS v3 prototype appears to have lost it's configuration. It stopped talking to the server a few days ago and power cycling wasn't helping. I had a chance to see what was on the serial console today. The OVMS prompt and command interface is working normally, but there doesn't appear to be anything configured. I'm still using a startup script I wrote before we had the configuration gui, and that doesn't appear to execute at startup. If I do a vfs ls /store then the command line interface hangs and I have to power cycle it to get it back. I don't have the wifi turned on so I don't know if the web interface works after doing this. This behaviour is repeatable, I if I power cycle the module the config is still missing and vfs ls /store still hangs. What is the best way to debug it? Michael appears to have had a similar problem in https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/151 how do I dump the flash to investigate why listing the directory hangs the command line interface?

3 3

Fwd: OVMS Nightly Firmware Build 3.1.009
by Mark Webb-Johnson 15 Aug '18

15 Aug '18

Enjoy > Begin forwarded message: > > From: mark(a)openvehicles.com > Subject: OVMS Nightly Firmware Build 3.1.009 > Date: 15 August 2018 at 8:11:30 PM HKT > To: mark(a)openvehicles.com > > 3.1.009 > Wed Aug 15 12:11:30 UTC 2018 Automated build (markhk8) > > Total sizes: > DRAM .data size: 16452 bytes > DRAM .bss size: 32864 bytes > Used static DRAM: 49316 bytes ( 131420 available, 27.3% used) > Used static IRAM: 107932 bytes ( 23140 available, 82.3% used) > Flash code: 1328694 bytes > Flash rodata: 612052 bytes > Total image size:~2065130 bytes (.bin may be padded larger) > > * 92d8501 (HEAD, tag: 3.1.009, origin/master, origin/HEAD, master) Known Issues, ready for 3.1.009 > * 9dd6131 Changes and TODO, ready for 3.1.009 > * 8724c1d Merge pull request #152 from caederus-ovms/units > |\ > | * e0aa195 Use correct SI unit abbreviations for distance (km), speed (km/h), acceleration (km/h/s - as an alternative to the usual m/s²). > | * 51ce2a2 If converting value to different unit, AsUnitString() should also convert unit name. > |/ > * 3c05772 TeslaModelS: Add support for CAN2 and CAN3, and decode MCU GPS lat, lon, heading, and HDOP > * 0e3c6d4 Twizy: fix missing charge records in RT-PWR-Log > * 7192be1 Minor command description changes to sync with user guide > * 3704933 Wifi: add RSSI level to status output > * 4cf3dba Merge pull request #150 from bobbo3/master > |\

1 0

units
by Robin O'Leary 15 Aug '18

15 Aug '18

I noticed that changing the config option for preferred units caused the display of distance on the status page to show the wrong units. AsUnitString() correctly converted the value to the desired units, but still appended the native unit name. While looking at that code, I also noticed that the metric unit abbreviations don't conform to standard SI notation: - kilometer should be 'km' (not 'Km' because capital-K means Kelvin). - kilometers per hour is better as 'km/h' (not 'Kph' because p means pico-). Here is a pull request to fix both: https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/152

2 1

Roadmap and team info
by Robert Sharpe (Electric Vehicle Consultant) 14 Aug '18

14 Aug '18

Hi, In response to a request for a road map there is now an easy to use Google Sheet where people can see the current road map, share what you are doing and what future features you would like to see. While doing that it's also a good opportunity to help new comers by pulling together a summary of the team and any (non feature) tasks people might be working on. Can you please therefore take a few minutes of your time in the next few days to fill in all three sheets below Thanks, Robert Sharpe Electric Vehicle Consultant 07711 252971 Your Plugged-in Partner Evergreen Consulting is the trading name of Sharpe Consultants Ltd

2 1