[Ovmsdev] Roadster: Some alerts not showing up

Thu Oct 31 11:41:19 HKT 2019

> Wait, your understanding is that the 1146 is a false positive?

My understanding is that both 1144 and 1146 may false positive. Tesla classify them as DMC (debug only messages) and don’t display them on the car VDS screen (although they are stored in the logs). When looking at vehicle logs, I have seen these two errors back to the birth of the cars. Almost from when they left the factory floor.

> The folks on TMC generally conclude that the error is related to the under-spec'd connector under the PEM that drives the motor, and that the connector pins need to be replaced (both sides).

I tend to agree with the root cause, although not necessarily the suggested action. The best guess is that the car is measuring current flow through fans, and if it is too high (indicating increased resistance in the wiring/connectors, or the fans themselves), it alerts. IMHO, the issue is that the thresholds in the firmware are too aggressive.

> To be fair, I was also getting some 1116 events in the log a while ago, and they seemed to have amounted to nothing and mostly have gone away on their own.  I've also gotten a number of 104s and 3001s, which nobody seems to think are important.

Have a look at ovms_server_tr.vece in the Open-Vehicles-Server project. That contains textual messages for the error codes we know of. Tesla don’t announce on the CAN bus whether the alert is debug only, or real, and we can only use the DMC prefix we have seen to guess. The VDS firmware appears to contain a lookup table (code -> message and debug/user flag), but efforts to decode that in an automated way have failed. So, the current server logic is to not notify DMC prefixed alerts.

We don’t have 1116, 104, or 3001 in the ovms_server_tr.vece, so for those we would just notify the alert code without a message. We also cannot know if they are debug only DMC or not.

> When was the server code updated? 

I think a week or so ago. The git change was October 16th (Hong Kong time).

> In my opinion, the 1144 and 1146 do need to be passed through

Quite frankly, over the years they have caused me a large support workload (with emails going back and forth, trying to explain each new case). Certainly the #1 support question for roadsters. If I had a dollar for every time a roadster owner has asked what these are and what they can do about them… Sad thing is that the only option is the spend $$$ to replace the connectors and fans, and even then that may not silence the alerts.

I did a quick check of last month’s server logs - in 20 days, the api.openvehicles.com <http://api.openvehicles.com/> server saw 2,592 of these (1144, 1146) alerts coming from about 1/4 of the Roadsters connecting in that time period.

I’ve also had some upset users. They sold the car, then the new owner sees these alerts in OVMS and complains. My response is ’the car is raising those alerts’, but I can’t easily explain why the car is not showing these debugging-only alerts on the VDS; It is only OVMS that is showing them.

Putting the alert message lookup table in the OVMS module firmware (which wasn’t possible in v2 due to size constraints) would solve this, and make it configurable on the module, but wouldn’t solve the problem for cars running older OVMS firmware. I would still get the support calls.

> Matching the limitations of the in-car VMS and decade-old diagnostic code is not high on my list of project objectives.

Actually, it is. We have gone to huge lengths (for example, matching integer only km/miles conversion algorithms) to make sure we display the same, or as close as possible, to the VDS. Otherwise, again the support call comes in “why does OVMS say 112km range, but the car says 111km?).

> If nothing else, we really should ask the Roadster community about silencing these alerts. 

I am happy to revert and let these through, so long as somebody else steps forward and takes on Roadster support. I am also happy to modify the server code, and make it kludgy configurable on a vehicle-by-vehicle basis (by changing ovms_server.conf to list the exception vehicle IDs). But I would rather the default (including for those with older firmware) be to not show alerts that are not shown on the VDS.

Regards, Mark.

> On 31 Oct 2019, at 10:20 AM, Greg D. <gregd2350 at gmail.com> wrote:
> 
> Wait, your understanding is that the 1146 is a false positive?  That's not the impression I was given by the service center crew and others.
> 
> After consulting with the folks in Fremont, the SC is recommending that I replace the fan motor assembly (USD $998.00).  The folks on TMC generally conclude that the error is related to the under-spec'd connector under the PEM that drives the motor, and that the connector pins need to be replaced (both sides).  It is insertion-cycled every year with the Annual Maintenance, but is only rated for a few dozen cycles, and it's being operated at nearly full current (no de-rating).  A failure certain to happen.
> 
> A few years ago I was getting 1144 alerts (in the cabin!), and eventually found that the contacts on said connector had in fact burned, and I had to replace the cable AND PEM.  Normally a $10k repair, but they found a refurbished PEM and swapped them for cost of the pins and labor.  Still an expensive repair at nearly a grand.  So you can see why I'm a little concerned about those events in particular.  
> 
> To be fair, I was also getting some 1116 events in the log a while ago, and they seemed to have amounted to nothing and mostly have gone away on their own.  I've also gotten a number of 104s and 3001s, which nobody seems to think are important.
> 
> When was the server code updated?  The last 1146 alert here was on 10/15.  Interestingly, I've had both 104 and 3001 alerts since then.  They appear to be ignorable, and those are not shown in the car, same as the 1146.  How are these alerts categorized?  I think we have some of the logic backwards.  In my opinion, the 1144 and 1146 do need to be passed through.  I can sleep without the 104 and 3001, but it's good to keep a pulse on their frequency, just in case there's a sudden up-tick.  I'm actually a little more annoyed about the Possible Theft / Flatbed alerts - they seem to have increased in frequency since this past spring (before the roll-over), and wonder if the GPS module is starting to fail.
> 
> If nothing else, we really should ask the Roadster community about silencing these alerts.  I really want to know what's going on with the car.  Matching the limitations of the in-car VMS and decade-old diagnostic code is not high on my list of project objectives.  We can do a lot better than that.
> 
> Greg
> 
> 
> Mark Webb-Johnson wrote:
>> 
>> The server doesn’t have any per-vehicle configuration (other than authentication), so not sure how this could be easily done server side.
>> 
>> A kludge might be to pickup the notification event in a javascript on the module, and then raise a new different textual event for specific IDs - but not sure if the vehicle module javascript framework gets enough about the raised notification in order to do that.
>> 
>> An alternate kludge would be to manually list vehicle IDs to be exempted from the DMC policy in the ovms_server.conf file, and have code to check that in ovms_server.pl.
>> 
>> But, that said, I’m not really sure why we would want this. Our goal has always been to mirror what it says on the little VDS in the car, and that doesn’t show these debug-only alerts. The solution we have is not perfect, but it does avoid the worst of them. I have received so many support questions and concerns from roadster users of OVMS asking about these over the years - Tesla is aware that they false positive so labelled them as debug-only, so why would we want to bombard the user with these alerts every time he drives/charges the car?
>> 
>> Regards, Mark.
>> 
>>> On 31 Oct 2019, at 7:39 AM, Greg D. <gregd2350 at gmail.com <mailto:gregd2350 at gmail.com>> wrote:
>>> 
>>> Bummer.
>>> 
>>> Can we make this configurable?  While I understand that debug messages aren't intended to be seen by the user, in the 10 years since these cars launched we have come to know a lot more about some of the, um, "quirks" in their design.  These alerts can be important!  
>>> 
>>> Greg
>>> 
>>> 
>>> Mark Webb-Johnson wrote:
>>>> Greg,
>>>> 
>>>> Yes, this change to the server code:
>>>> 
>>>> * bcc02c27 Ignore DMC push alert messages
>>>> 
>>>> That won’t notify debug-only messages, but will still continue to notify the user-visible ones (to better mimick the behaviour in the car itself). Including, for the roadster:
>>>> 
>>>> 1144=DMC: Powertrain Problem Service Required
>>>> 1146=DMC: Motor Fan Problem
>>>> 
>>>> Regards, Mark.
>>>> 
>>>>> On 31 Oct 2019, at 7:01 AM, Greg D. <gregd2350 at gmail.com <mailto:gregd2350 at gmail.com>> wrote:
>>>>> 
>>>>> Hi folks (Mark, probably),
>>>>> 
>>>>> Interesting...  My Tesla Roadster has been having trouble with its PEM
>>>>> fan, causing a bunch of 1146 Alerts to be generated while driving and/or
>>>>> charging.  I've been monitoring the situation via OVMSv3. 
>>>>> 
>>>>> The alerts suddenly stopped a bit over a week ago.  Other events
>>>>> continue to be received (charge start, stop, key in / door open, etc.),
>>>>> but not the 1146's.  I do a monthly download of the car logs, and to my
>>>>> surprise (and disappointment), the 1146s are still happening.
>>>>> 
>>>>> Was there a change to the plumbing somewhere such that these diagnostic
>>>>> alerts (perhaps versus user-visible events) are now being filtered out? 
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Greg
>>>>> 
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>> 
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>> 
>> 
>> 
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
> 
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20191031/b6d0fa6c/attachment.htm>