August 2018 - OvmsDev - Open Vehicles

v3.1.010 OTA
by Mark Webb-Johnson 12 Sep '18

12 Sep '18

I just released v3.1.010 OTA to ‘main’. Everyone gets it. Regards, Mark.

2 2

MQTT and Ovms Server v3
by Mark Webb-Johnson 31 Aug '18

31 Aug '18

I am far from an expert in MQTT. Not even a novice. So, the work below is ‘best efforts’. Any help / comments / suggestions would be most appreciated. In particular, a big thanks to Jakob for his contributions to this so far. With yesterday’s merge, and commits, we have a very very basic OVMS server v3 implementation. It sends the metrics, and doesn’t crash (much). The overall design is as follows: We use the mongoose mqtt library. We don’t do anything special, and everything is following the mqtt 3.1 standard. MQTT has the concept of topics. Our default prefix for everything is: ovms/<mqtt-username>/<vehicleid>/ (that can be customised with config server.v3 topic.prefix) Metrics are sent as topics. The metric name is added to the topic prefix + “m/“ suffix, and “.” characters converted to “/“ to match the MQTT conventions. The value is simply the metric value (as a string). With our current arrangement, this adds m/m/, m/s/, and m/v/ sub-trees to the the MQTT topic hierarchy. Clients can subscribe to <prefix>/m/# to receive all metrics. The ‘retained’ flag is set on these metrics, at QOS level 0 (so subscribing clients will get a copy of the last known values for these metrics, even with a disconnected vehicle). The metric s.v3.connected is maintained by the ServerV3 code. When a successful MQTT connection is made, and login successful, that is set true (yes). A MQTT last-will-testament is set so that if the OVMS module disconnects the server will automatically update that to false (no). The ‘retained’ flag is set on this metric, at QOS level 0 (so subscribing clients will get a copy of the last known state). Clients can use this to see if the vehicle is connected to the server. Connecting clients are expected to write a “1” value to the <prefix>/c/<clientid> topic, and to repeat that write once a minute. They are also expected to use a last-will-testament on that same topic with value “0”. QOS 1 should be used, and these topics should not be retained. The server V3 code subscribes to this <prefix>/c/# topic, so it gets informed of all connected clients. It can then update the s.v3.peers metric appropriately. Clients are expected to monitor the <prefix>/s/v3/connected topic, so that if it becomes ‘yes’ (true) the client should send <prefix>/c/<clientid> “1” immediately. This mechanism allows a newly connected vehicle to immediately know if one or more clients is connected. The Server v3 sets a timeout for <prefix>/c/<clientid> connections of 2 minutes. If that mqtt topic is not sent again within that time, it is expired (and that client is treated as disconnected). Similar to the v2 code, the server v3 transmits modified metrics once a minute if there are one or more clients connected, or once every ten minutes if there are no clients connected. All the above has been implemented. To reach parity with v2, and exceed it’s functionality in places, we have a few more things to do: Notifications On the vehicle side, I am proposing to use the <prefix>/notify/<type> namespace for these, using QOS 2 messages without retention. Clients can listen to those, if necessary. We can also have a central daemon running that listens to ovms/+/+/n/# topic pattern to receive these notifications and handle appropriately. Using QOS 2 we can confirm the reception of the notification / historical data, and mark it as delivered, appropriately. However, that would only confirm delivery to the MQTT server, not to the central daemon; if the daemon was not running, the notification would be lost. Textual Commands I am proposing to use the <prefix>/cmd/<clientid>/c/ and <prefix>/cmd/<clientid>/r/ namespaces for this, using QOS 2 messages without retention. The value in the /c/ would be the command, and the response would be written to matching the /r/ topic. The commands and responses could be prefixed by an identifier (like in imap protocol) so responses can be matched to commands by the clients). The client side can simply subscribe to itself, and the vehicle side subscribes to <prefix>/cmd/#. In this way, commands cannot be duplicated, and clients don’t see responses to commands they didn’t initiate (which was an issue with v2). Numeric Commands I am not sure if we need to implement the numeric commands, as used in the v2 protocol. It seems to me that we can use textual commands. Config Access I am not sure if we need this, beyond the command processor. If we do, we could expose a /config/ namespace. Events It would be nice to expose events (except for the ticker.* ones, of course). This could be done by a <prefix>/events topic, using QOS 2 and no retention. Logs It would be nice to expose logs. This could be done by a <prefix>/logs topic, using QOS 1 and no retention. Security We need to add SSL support. I am trying to get an authentication plugin for mosquitto / vernemq written so that we can authenticate straight from the OVMS database that is already running on the servers, and give each user their own ovms/<userid>/# namespace. That way, the configuration for v3 on the vehicle/apps/clients is simple - just put in the server username and password (no separate vehicle passwords necessary). I think that is it. The above would form the basis of the specification for this. As this is the basis for future work and direction in OVMS, it is important that we get it right, so all comments / suggestions most welcome. Regards, Mark.

4 15

Production batch 2 and onwards
by Mark Webb-Johnson 30 Aug '18

30 Aug '18

FYI The order for production batch #2 has just been placed: 50x US kits (module + US 3G modem + Hologram sim) 150x EU kits (module + EU 3G modem + Hologram sim) 20x modules only (board + case, but no modem) 5x EU 3G modems (incl. Hologram sim) 5x US 3G modems (incl. Hologram sim) No change to specification and board layout from the first production batch, except to re-iterate the points that came in from feedback. We are arranging a foam box to ship the modules in, with our logo on it, to avoid the issues with sticky glue on those padded bags. Availability at Fasttech perhaps in about 3 weeks time. Our deadline for firmware is about 10 days. This is a lot of $$$ upfront payment, and the project is pretty tapped financially (I’ve had to pay for most of this myself). Once the money from sales starts to come in, we’ll be better off. I have quotes from certification labs for CE and FCC certs, and it seems possible. They say we may have some small changes to make (ESD primarily), but overall our design seems fine and they are confident we can achieve the certification. The process is expensive, but is just a one off upfront payment. Current plan is to start with that process once we have this production batch built and at Fasttech. We did consider doing this before this production batch #2, but that would have just delayed things several months. So long as this certification process doesn’t drag on, we should be able to keep OVMS v3 in stock and available from now. Regards, Mark.

3 15

Nissan Leaf, GPIO and "Stop Charging", "Lock/Unlock" commands.
by Brian Greenberg 30 Aug '18

30 Aug '18

I picked up a 2015 Leaf about 3 weeks ago, and found this project a few days later. My 3.1 hardware should arrive from FastTech this week. I've been browsing through the git repos and manuals, and hope to make some small contributions in the future. How interested are you devs in code contributions that require hardware modifications to the vehicle? :) Two things that I am interested in are remotely toggling the door locks, and stopping/disabling an ongoing charge. Neither are currently supported functions in the Leaf, as (if it is possible at all) the codes are unknown. However, both functions are trivially available outside of the car network. To disable charging, it should be quite easy to fake the charge computer into thinking that the J1772 connector is about to be removed -- there is a connector under the hood, just past the L1/L2 socket that carries the socket ground and (low voltage) proximity pin signal to the VCM (Or BCM?). Adding a 330 ohm resistance to the proximity pin tells the car that power is about to be removed, and to stop charging. Door locks will have a fairly easy to locate pair of wires in the front doors that will trigger locking and unlocking. Three GPIO assignments and three opto isolated 12V relays should easily enable the desired functionality. Assuming I can walk this walk, would there be interest in the code? This would be a bit more involved than the single wire connection between the OVMS module and the TCU that is needed to enable the vehicle wake up signal on 2011 and 2012 models..... I'm very impressed with this project, BTW. I've been around open source for years, but I've never seen open hardware development at this level. Brian. -- Brian Greenberg grnbrg(a)grnbrg.org

3 4

AutoInit Inhibit
by Mark Webb-Johnson 29 Aug '18

29 Aug '18

Last week’s 3.1.009 issue with Tesla Roadsters was interesting. Looking at the car I saw with the problem, it was booting, crashing when the vehicle module received a certain CAN bus message (triggering the issue), and rebooting. It would do this 5 times, until it his the AUTO_INIT_INHIBIT_CRASHCOUNT count, and then AutoInit inhibit would kick in, and it would end up sitting idle with nothing loaded (and no network connection). The approach worked very well, and prevented an endless reboot loop. However, we ended up with a car unable to connect to the network, and requiring a console cable and laptop to recover. I’ve been thinking about bluetooth (which is intended to provide a smartphone connection irrespective of wifi configuration), how that might work with a completely unconfigured module (for initial configuration), and I can suggest a workaround to try to improve this… At the moment, the order of autoinit is: External 12V Wifi Modem SIMCOM Vehicle OBD2ECU Server V2 Server V3 None of those run if the early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT (coded as a constant 5). My suggestion is to change this as follows: Wifi Bluetooth Server V2 Server V3 Modem SIMCOM External 12V Vehicle ODB2ECU But also to change that logic that: #8 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT #7 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+1 #6 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+2 etc. This will mean that once we hit the AUTO_INIT_INHIBIT_CRASHCOUNT limit, we turn off the least required module (OBD2ECU), then continue to see if that solves the problem. If not (ie; we crash and reboot), then we try turning off the Vehicle module and OBD2ECU, then continue to see if that solves the problem. etc. In the example case of the Tesla Roadster vehicle module causing the issue, this revised approach would leave us up and running with Wifi and a Server Connection (but no vehicle data). Checking the module would identify the problem quite quickly (and remotely). Starting the vehicle module manually would presumably result in a crash and we would pretty quickly get an idea of the cause (especially if we added a command to show the status of AutoInit, what was started, and what wasn’t because it is causing a crash). I think we could even issue an Alert notification in the case where autoinit recovered after inhibiting certain modules (but not all). The disadvantage is that we would crash more (up to 13 times, vs 5) in the case the system is badly messed up. What do people think? Does this make sense? Regards, Mark.

3 8

OT: A possible insight on how Tesla may have been doing telematics
by Nikolay Shishkov 27 Aug '18

27 Aug '18

This a bit off-topic and a bit questionable, but I found interesting the comment about Node.js vs Go for the backend implementation:ato̧̕m̀͡i̴̷̛c̨͝t͝҉͡h̷҉u̵̶m͜͞b͏͝s̀́ on Twitter | | | | | | | | | | | ato̧̕m̀͡i̴̷̛c̨͝t͝҉͡h̷҉u̵̶m͜͞b͏͝s̀́ on Twitter “A former Tesla employee, who worked on their IT infrastructure, is posting in a subforum of a subforum, a littl... | | | On Monday, August 27, 2018, 2:53:52 AM GMT+2, Mark Webb-Johnson <mark(a)webb-johnson.net> wrote: Last week’s 3.1.009 issue with Tesla Roadsters was interesting. Looking at the car I saw with the problem, it was booting, crashing when the vehicle module received a certain CAN bus message (triggering the issue), and rebooting. It would do this 5 times, until it his the AUTO_INIT_INHIBIT_CRASHCOUNT count, and then AutoInit inhibit would kick in, and it would end up sitting idle with nothing loaded (and no network connection). The approach worked very well, and prevented an endless reboot loop. However, we ended up with a car unable to connect to the network, and requiring a console cable and laptop to recover. I’ve been thinking about bluetooth (which is intended to provide a smartphone connection irrespective of wifi configuration), how that might work with a completely unconfigured module (for initial configuration), and I can suggest a workaround to try to improve this… At the moment, the order of autoinit is: - External 12V - Wifi - Modem SIMCOM - Vehicle - OBD2ECU - Server V2 - Server V3 None of those run if the early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT (coded as a constant 5). My suggestion is to change this as follows: - Wifi - Bluetooth - Server V2 - Server V3 - Modem SIMCOM - External 12V - Vehicle - ODB2ECU But also to change that logic that: - #8 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT - #7 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+1 - #6 will start if early crash count > AUTO_INIT_INHIBIT_CRASHCOUNT+2 - etc. This will mean that once we hit the AUTO_INIT_INHIBIT_CRASHCOUNT limit, we turn off the least required module (OBD2ECU), then continue to see if that solves the problem. If not (ie; we crash and reboot), then we try turning off the Vehicle module and OBD2ECU, then continue to see if that solves the problem. etc. In the example case of the Tesla Roadster vehicle module causing the issue, this revised approach would leave us up and running with Wifi and a Server Connection (but no vehicle data). Checking the module would identify the problem quite quickly (and remotely). Starting the vehicle module manually would presumably result in a crash and we would pretty quickly get an idea of the cause (especially if we added a command to show the status of AutoInit, what was started, and what wasn’t because it is causing a crash). I think we could even issue an Alert notification in the case where autoinit recovered after inhibiting certain modules (but not all). The disadvantage is that we would crash more (up to 13 times, vs 5) in the case the system is badly messed up. What do people think? Does this make sense? Regards, Mark. _______________________________________________ OvmsDev mailing list OvmsDev(a)lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev

1 0

Config store mount failure / reformatting
by Michael Balzer 23 Aug '18

23 Aug '18

I've pulled the latest esp-idf updates, merges, builds and runs without issues. Btw, the idf now includes a CAN driver, may be worth a look. On the second app-flashing, this happened: … Wrote 2399584 bytes (1382879 compressed) at 0x00010000 in 26.1 seconds (effective 736.0 kbit/s)... Hash of data verified. Leaving... Hard resetting via RTS pin... … I (452) ovms_main: Mounting CONFIG... W (722) vfs_fat_spiflash: f_mount failed (13) I (722) vfs_fat_spiflash: Formatting FATFS partition, allocation unit size=4096 I (1152) vfs_fat_spiflash: Mounting again Initialising OVMS CONFIG within STORE :-( The first flash was perfectly OK and I haven't been able to reproduce this afterwards. I don't think this is related to idf changes, as we had some reports on lost configs before. Issue #145 also seems to be solved by the new idf, I had no more crashes during reboots. I wonder if there could be a better strategy than immediately formatting the config filesystem on a mount error. Is there any chance the mount fails due to some race condition, i.e. would it make sense to first retry mounting after a short delay? Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

3 7

SIMCom diagnostics
by Tom Parker 22 Aug '18

22 Aug '18

Hi, I've been experiencing poor cellular connectivity with my v3.0 prototype. The changes Mark made a while ago to recover from various corner cases improved things a lot but outside my house it is often disconnected even though my cell phone has reasonable signal strength. I tried to extract extra information from the modem and also force it to use 2G since my OVMS v2 is rock solid in the same location on the same carrier. I learnt some things: I'm using the possibly slightly out of date https://simcom.ee/documents/SIM5360/SIMCOM_SIM5360_ATC_EN_V0.24.pdf which doesn't require a login to get. Sending AT+CNMP=13 is documented to force it to use 2G, but instead it puts the modem into a boot loop. It does the START +CPIN: READY dance ending with PB DONE and then goes back to START. It never responds to the first AT commands we send it, and power cycling does not fix it. I went down many rabbit holes to fix this, it looks a lot like not enough power (I'm powering via USB) but adding 13V to the CAN bus port didn't fix it. Removing the simcard fixed it, but this supported the power theory which turned out be wrong. Sending AT+CNMP=2 as soon as it sends START does result in OK and pust it back into auto mode and stops the boot loop. I spent a while hacking simcom.cpp to send commands of my choice to the modem, but got tired of that and thought, there _must_ be a way to do this interactively! I traced the tx code paths and found that there are two ways: simcom tx AT+CNMP? simcom muxtx 3 AT+CNSMOD? You probably need at least verbose logging to see the responses. You need to use muxtx when in mux mode. I'm not 100% sure what muxtx does not in mux mode. After I sorted out the modem boot loop I added polling of AT+CPSI? and hooked the result up to a metric (and also logged it). I can see which cell tower I am connected two and on what frequency and protocol, below is it switching cell id and frequency: I (26562395) simcom: CPSI: +CPSI: WCDMA,Online,530-01,0x0778,xxx,WCDMA IMT 2000,306,10588,0,9.0,95,24,20,500 I (26592445) simcom: CPSI: +CPSI: WCDMA,Online,530-01,0x0778,yyy,WCDMA 900,62,3086,0,5.0,94,27,21,500 Obviously smashing all of that into one metric isn't a great thing to do, and probably not all of it is needed in metrics? The cell identifier, protocol and frequency look useful. Simcom's documentation for this says If camping on a 2G cell: +CPSI:<System Mode>,<Operation Mode>,<MCC>-<MNC><LAC>,<Cell ID>,<Absolute RF Ch Num>,<RxLev>,<Track LO Adjust>,<C1-C2> If camping on a 3G cell: +CPSI:<System Mode>,<Operation Mode>,<MCC>-<MNC>,<LAC>,<Cell ID>,<Frequency Band>,<PSC>,<Freq>,<SSC>,<EC/IO>,< RSCP >,<Qual>,<RxLev>,<TXPWR> I'm not sure what else is useful to parse into metrics? Maybe the network numbers for roaming? See page 140 of the pdf mentioned above for more details. I haven't yet used this information to try to tell what is wrong with my poor connectivity problem. I also discovered that the modem is not stable powered by USB on my desk. It disconnects from the cell network or reboots every few minutes. It has been stable for many hours since I turned the 13.5V power supply back on.

2 1

OvmsMetricString memory lifecycle
by Tom Parker 21 Aug '18

21 Aug '18

I have what I hope is a stupid question. The additional cellular information code I wrote about yesterday worked for most of a day and then the ovms module stopped communicating with the modem. I didn't have a data logger connected so I don't know what exactly happened. When I got on the console the ovms was going through the modem power cycle loop, but never receiving data from the modem. I tried a few things and then I did a simcom status and it hung after printing a few garbage characters instead of the line of information I added to the status report yesterday. This got me to looking at how that memory is handled. I'm doing the following in simcom::StandardLineHandler() which copies the similar pattern for +CGMR and +ICCID in the same function: else if (line.compare(0, 7, "+CPSI: ") == 0) { StandardMetrics.ms_m_net_mdm_cpsi->SetValue(line.substr(7)); ESP_LOGI(TAG, "CPSI: %s", line.c_str()); } I know very little about C++ but from what I gather and reading the code * std:string.substr() creates an object on the stack pointing to memory on the heap * OvmsMetricString::SetValue(std::string value) copies the object into it's internal m_value but it still points to the memory allocated on the heap It's not clear when the heap memory for the std:string created in substr() is freed. I think it is considered for freeing at the end of simcom::StandardLineHandler as it goes out of scope. But it escapes the scope when it is passed to OvmsMetricString::SetValue() so if it were freed on function exit, we would have a use after free bug. Is it possible the compiler notices the reference escapes the scope and so doesn't free the string? If it does, then when does it get freed? It surely can't be freed when it replaced by the new value inside OvmsMetricString::SetValue() -- at first thought, this pattern without further hints from the programmer would make it too easy to end up with double free bugs or memory leaks. If it isn't freed automatically and it isn't freed manually then it looks like we have a memory leak. Looking at the memory logs that the housekeeping message gives, it doesn't look like I have a memory leak, so is there a use after free? If there is, why doesn't it cause problems all the time? There are a lot of string metrics and at least 2 that use exactly the same pattern I used. My changes are at https://github.com/carrott/Open-Vehicle-Monitoring-System-3/tree/cell-info

3 3

esp-idf update
by Michael Balzer 21 Aug '18

21 Aug '18

I've pushed the merge with the latest esp-idf master to our repository. I've been running this build for about a day without issues. It contains some new configurations, all defaults can be accepted. Don't forget to do a "git submodule update" after pulling. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

2 1