Roadster: Some alerts not showing up
Hi folks (Mark, probably), Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3. The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening. Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out? Thanks, Greg
Greg, Yes, this change to the server code: * bcc02c27 Ignore DMC push alert messages That won’t notify debug-only messages, but will still continue to notify the user-visible ones (to better mimick the behaviour in the car itself). Including, for the roadster: 1144=DMC: Powertrain Problem Service Required 1146=DMC: Motor Fan Problem Regards, Mark.
On 31 Oct 2019, at 7:01 AM, Greg D. <gregd2350@gmail.com> wrote:
Hi folks (Mark, probably),
Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3.
The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening.
Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out?
Thanks,
Greg
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
The server doesn’t have any per-vehicle configuration (other than authentication), so not sure how this could be easily done server side. A kludge might be to pickup the notification event in a javascript on the module, and then raise a new different textual event for specific IDs - but not sure if the vehicle module javascript framework gets enough about the raised notification in order to do that. An alternate kludge would be to manually list vehicle IDs to be exempted from the DMC policy in the ovms_server.conf file, and have code to check that in ovms_server.pl. But, that said, I’m not really sure why we would want this. Our goal has always been to mirror what it says on the little VDS in the car, and that doesn’t show these debug-only alerts. The solution we have is not perfect, but it does avoid the worst of them. I have received so many support questions and concerns from roadster users of OVMS asking about these over the years - Tesla is aware that they false positive so labelled them as debug-only, so why would we want to bombard the user with these alerts every time he drives/charges the car? Regards, Mark.
On 31 Oct 2019, at 7:39 AM, Greg D. <gregd2350@gmail.com> wrote:
Bummer.
Can we make this configurable? While I understand that debug messages aren't intended to be seen by the user, in the 10 years since these cars launched we have come to know a lot more about some of the, um, "quirks" in their design. These alerts can be important!
Greg
Mark Webb-Johnson wrote:
Greg,
Yes, this change to the server code:
* bcc02c27 Ignore DMC push alert messages
That won’t notify debug-only messages, but will still continue to notify the user-visible ones (to better mimick the behaviour in the car itself). Including, for the roadster:
1144=DMC: Powertrain Problem Service Required 1146=DMC: Motor Fan Problem
Regards, Mark.
On 31 Oct 2019, at 7:01 AM, Greg D. <gregd2350@gmail.com <mailto:gregd2350@gmail.com>> wrote:
Hi folks (Mark, probably),
Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3.
The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening.
Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out?
Thanks,
Greg
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Wait, your understanding is that the 1146 is a false positive?
My understanding is that both 1144 and 1146 may false positive. Tesla classify them as DMC (debug only messages) and don’t display them on the car VDS screen (although they are stored in the logs). When looking at vehicle logs, I have seen these two errors back to the birth of the cars. Almost from when they left the factory floor.
The folks on TMC generally conclude that the error is related to the under-spec'd connector under the PEM that drives the motor, and that the connector pins need to be replaced (both sides).
I tend to agree with the root cause, although not necessarily the suggested action. The best guess is that the car is measuring current flow through fans, and if it is too high (indicating increased resistance in the wiring/connectors, or the fans themselves), it alerts. IMHO, the issue is that the thresholds in the firmware are too aggressive.
To be fair, I was also getting some 1116 events in the log a while ago, and they seemed to have amounted to nothing and mostly have gone away on their own. I've also gotten a number of 104s and 3001s, which nobody seems to think are important.
Have a look at ovms_server_tr.vece in the Open-Vehicles-Server project. That contains textual messages for the error codes we know of. Tesla don’t announce on the CAN bus whether the alert is debug only, or real, and we can only use the DMC prefix we have seen to guess. The VDS firmware appears to contain a lookup table (code -> message and debug/user flag), but efforts to decode that in an automated way have failed. So, the current server logic is to not notify DMC prefixed alerts. We don’t have 1116, 104, or 3001 in the ovms_server_tr.vece, so for those we would just notify the alert code without a message. We also cannot know if they are debug only DMC or not.
When was the server code updated?
I think a week or so ago. The git change was October 16th (Hong Kong time).
In my opinion, the 1144 and 1146 do need to be passed through
Quite frankly, over the years they have caused me a large support workload (with emails going back and forth, trying to explain each new case). Certainly the #1 support question for roadsters. If I had a dollar for every time a roadster owner has asked what these are and what they can do about them… Sad thing is that the only option is the spend $$$ to replace the connectors and fans, and even then that may not silence the alerts. I did a quick check of last month’s server logs - in 20 days, the api.openvehicles.com <http://api.openvehicles.com/> server saw 2,592 of these (1144, 1146) alerts coming from about 1/4 of the Roadsters connecting in that time period. I’ve also had some upset users. They sold the car, then the new owner sees these alerts in OVMS and complains. My response is ’the car is raising those alerts’, but I can’t easily explain why the car is not showing these debugging-only alerts on the VDS; It is only OVMS that is showing them. Putting the alert message lookup table in the OVMS module firmware (which wasn’t possible in v2 due to size constraints) would solve this, and make it configurable on the module, but wouldn’t solve the problem for cars running older OVMS firmware. I would still get the support calls.
Matching the limitations of the in-car VMS and decade-old diagnostic code is not high on my list of project objectives.
Actually, it is. We have gone to huge lengths (for example, matching integer only km/miles conversion algorithms) to make sure we display the same, or as close as possible, to the VDS. Otherwise, again the support call comes in “why does OVMS say 112km range, but the car says 111km?).
If nothing else, we really should ask the Roadster community about silencing these alerts.
I am happy to revert and let these through, so long as somebody else steps forward and takes on Roadster support. I am also happy to modify the server code, and make it kludgy configurable on a vehicle-by-vehicle basis (by changing ovms_server.conf to list the exception vehicle IDs). But I would rather the default (including for those with older firmware) be to not show alerts that are not shown on the VDS. Regards, Mark.
On 31 Oct 2019, at 10:20 AM, Greg D. <gregd2350@gmail.com> wrote:
Wait, your understanding is that the 1146 is a false positive? That's not the impression I was given by the service center crew and others.
After consulting with the folks in Fremont, the SC is recommending that I replace the fan motor assembly (USD $998.00). The folks on TMC generally conclude that the error is related to the under-spec'd connector under the PEM that drives the motor, and that the connector pins need to be replaced (both sides). It is insertion-cycled every year with the Annual Maintenance, but is only rated for a few dozen cycles, and it's being operated at nearly full current (no de-rating). A failure certain to happen.
A few years ago I was getting 1144 alerts (in the cabin!), and eventually found that the contacts on said connector had in fact burned, and I had to replace the cable AND PEM. Normally a $10k repair, but they found a refurbished PEM and swapped them for cost of the pins and labor. Still an expensive repair at nearly a grand. So you can see why I'm a little concerned about those events in particular.
To be fair, I was also getting some 1116 events in the log a while ago, and they seemed to have amounted to nothing and mostly have gone away on their own. I've also gotten a number of 104s and 3001s, which nobody seems to think are important.
When was the server code updated? The last 1146 alert here was on 10/15. Interestingly, I've had both 104 and 3001 alerts since then. They appear to be ignorable, and those are not shown in the car, same as the 1146. How are these alerts categorized? I think we have some of the logic backwards. In my opinion, the 1144 and 1146 do need to be passed through. I can sleep without the 104 and 3001, but it's good to keep a pulse on their frequency, just in case there's a sudden up-tick. I'm actually a little more annoyed about the Possible Theft / Flatbed alerts - they seem to have increased in frequency since this past spring (before the roll-over), and wonder if the GPS module is starting to fail.
If nothing else, we really should ask the Roadster community about silencing these alerts. I really want to know what's going on with the car. Matching the limitations of the in-car VMS and decade-old diagnostic code is not high on my list of project objectives. We can do a lot better than that.
Greg
Mark Webb-Johnson wrote:
The server doesn’t have any per-vehicle configuration (other than authentication), so not sure how this could be easily done server side.
A kludge might be to pickup the notification event in a javascript on the module, and then raise a new different textual event for specific IDs - but not sure if the vehicle module javascript framework gets enough about the raised notification in order to do that.
An alternate kludge would be to manually list vehicle IDs to be exempted from the DMC policy in the ovms_server.conf file, and have code to check that in ovms_server.pl.
But, that said, I’m not really sure why we would want this. Our goal has always been to mirror what it says on the little VDS in the car, and that doesn’t show these debug-only alerts. The solution we have is not perfect, but it does avoid the worst of them. I have received so many support questions and concerns from roadster users of OVMS asking about these over the years - Tesla is aware that they false positive so labelled them as debug-only, so why would we want to bombard the user with these alerts every time he drives/charges the car?
Regards, Mark.
On 31 Oct 2019, at 7:39 AM, Greg D. <gregd2350@gmail.com <mailto:gregd2350@gmail.com>> wrote:
Bummer.
Can we make this configurable? While I understand that debug messages aren't intended to be seen by the user, in the 10 years since these cars launched we have come to know a lot more about some of the, um, "quirks" in their design. These alerts can be important!
Greg
Mark Webb-Johnson wrote:
Greg,
Yes, this change to the server code:
* bcc02c27 Ignore DMC push alert messages
That won’t notify debug-only messages, but will still continue to notify the user-visible ones (to better mimick the behaviour in the car itself). Including, for the roadster:
1144=DMC: Powertrain Problem Service Required 1146=DMC: Motor Fan Problem
Regards, Mark.
On 31 Oct 2019, at 7:01 AM, Greg D. <gregd2350@gmail.com <mailto:gregd2350@gmail.com>> wrote:
Hi folks (Mark, probably),
Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3.
The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening.
Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out?
Thanks,
Greg
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Watching this thread with some interest as I, like Greg, would like to see the “Debug” alerts over OVMS. I would note that if the goal is to have OVMS show what the VDS shows, if you have the Roadster set to “Show Debug”, then OVMS will not show what VDS shows. Is it possible to track the status of “Debug reporting to VDS” in the car, and have the OVMS server relay accordingly? Thanks, and apologies for jumping in. Chip Sent from my tablet On Thu, Oct 31, 2019 at 1:30 AM +0100, "Mark Webb-Johnson" <mark@webb-johnson.net> wrote: The server doesn’t have any per-vehicle configuration (other than authentication), so not sure how this could be easily done server side. A kludge might be to pickup the notification event in a javascript on the module, and then raise a new different textual event for specific IDs - but not sure if the vehicle module javascript framework gets enough about the raised notification in order to do that. An alternate kludge would be to manually list vehicle IDs to be exempted from the DMC policy in the ovms_server.conf file, and have code to check that in ovms_server.pl. But, that said, I’m not really sure why we would want this. Our goal has always been to mirror what it says on the little VDS in the car, and that doesn’t show these debug-only alerts. The solution we have is not perfect, but it does avoid the worst of them. I have received so many support questions and concerns from roadster users of OVMS asking about these over the years - Tesla is aware that they false positive so labelled them as debug-only, so why would we want to bombard the user with these alerts every time he drives/charges the car? Regards, Mark. On 31 Oct 2019, at 7:39 AM, Greg D. <gregd2350@gmail.com> wrote: Bummer. Can we make this configurable? While I understand that debug messages aren't intended to be seen by the user, in the 10 years since these cars launched we have come to know a lot more about some of the, um, "quirks" in their design. These alerts can be important! Greg Mark Webb-Johnson wrote: Greg, Yes, this change to the server code: * bcc02c27 Ignore DMC push alert messages That won’t notify debug-only messages, but will still continue to notify the user-visible ones (to better mimick the behaviour in the car itself). Including, for the roadster: 1144=DMC: Powertrain Problem Service Required 1146=DMC: Motor Fan Problem Regards, Mark. On 31 Oct 2019, at 7:01 AM, Greg D. <gregd2350@gmail.com> wrote: Hi folks (Mark, probably), Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3. The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening. Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out? Thanks, Greg _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Chip, Good suggestion, But unfortunately we haven’t been able to find that signal on the can bus. I suspect it is never sent, but just entered and stored in the VDS; as the vms sends debug alerts regardless of the setting. The alert message strings are in the vds firmware, and vms only sends codes. To do this properly would require ovms doing what the vds does - have an alert code -> debug/user,message table and do the conversion and filtering there (not in the server). But that is quite a bit of code flash memory. Regards, Mark
On 31 Oct 2019, at 4:50 PM, Chip Cangialose <chip@cangmag.com> wrote:
Watching this thread with some interest as I, like Greg, would like to see the “Debug” alerts over OVMS. I would note that if the goal is to have OVMS show what the VDS shows, if you have the Roadster set to “Show Debug”, then OVMS will not show what VDS shows. Is it possible to track the status of “Debug reporting to VDS” in the car, and have the OVMS server relay accordingly?
Thanks, and apologies for jumping in.
Chip
Sent from my tablet
On Thu, Oct 31, 2019 at 1:30 AM +0100, "Mark Webb-Johnson" <mark@webb-johnson.net> wrote:
The server doesn’t have any per-vehicle configuration (other than authentication), so not sure how this could be easily done server side.
A kludge might be to pickup the notification event in a javascript on the module, and then raise a new different textual event for specific IDs - but not sure if the vehicle module javascript framework gets enough about the raised notification in order to do that.
An alternate kludge would be to manually list vehicle IDs to be exempted from the DMC policy in the ovms_server.conf file, and have code to check that in ovms_server.pl.
But, that said, I’m not really sure why we would want this. Our goal has always been to mirror what it says on the little VDS in the car, and that doesn’t show these debug-only alerts. The solution we have is not perfect, but it does avoid the worst of them. I have received so many support questions and concerns from roadster users of OVMS asking about these over the years - Tesla is aware that they false positive so labelled them as debug-only, so why would we want to bombard the user with these alerts every time he drives/charges the car?
Regards, Mark.
On 31 Oct 2019, at 7:39 AM, Greg D. <gregd2350@gmail.com> wrote:
Bummer.
Can we make this configurable? While I understand that debug messages aren't intended to be seen by the user, in the 10 years since these cars launched we have come to know a lot more about some of the, um, "quirks" in their design. These alerts can be important!
Greg
Mark Webb-Johnson wrote:
Greg,
Yes, this change to the server code:
* bcc02c27 Ignore DMC push alert messages
That won’t notify debug-only messages, but will still continue to notify the user-visible ones (to better mimick the behaviour in the car itself). Including, for the roadster:
1144=DMC: Powertrain Problem Service Required 1146=DMC: Motor Fan Problem
Regards, Mark.
On 31 Oct 2019, at 7:01 AM, Greg D. <gregd2350@gmail.com> wrote:
Hi folks (Mark, probably),
Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3.
The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening.
Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out?
Thanks,
Greg
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
A suggestion: use error notifications for visible/real errors, and a history record table for all errors/debug messages? Regards, Michael Am 31.10.19 um 09:57 schrieb Mark Webb-Johnson:
Chip,
Good suggestion, But unfortunately we haven’t been able to find that signal on the can bus. I suspect it is never sent, but just entered and stored in the VDS; as the vms sends debug alerts regardless of the setting. The alert message strings are in the vds firmware, and vms only sends codes.
To do this properly would require ovms doing what the vds does - have an alert code -> debug/user,message table and do the conversion and filtering there (not in the server). But that is quite a bit of code flash memory.
Regards, Mark
On 31 Oct 2019, at 4:50 PM, Chip Cangialose <chip@cangmag.com> wrote:
Watching this thread with some interest as I, like Greg, would like to see the “Debug” alerts over OVMS. I would note that if the goal is to have OVMS show what the VDS shows, if you have the Roadster set to “Show Debug”, then OVMS _will not_ show what VDS shows. Is it possible to track the status of “Debug reporting to VDS” in the car, and have the OVMS server relay accordingly?
Thanks, and apologies for jumping in.
Chip
Sent from my tablet
On Thu, Oct 31, 2019 at 1:30 AM +0100, "Mark Webb-Johnson" <mark@webb-johnson.net <mailto:mark@webb-johnson.net>> wrote:
The server doesn’t have any per-vehicle configuration (other than authentication), so not sure how this could be easily done server side.
A kludge might be to pickup the notification event in a javascript on the module, and then raise a new different textual event for specific IDs - but not sure if the vehicle module javascript framework gets enough about the raised notification in order to do that.
An alternate kludge would be to manually list vehicle IDs to be exempted from the DMC policy in the ovms_server.conf file, and have code to check that in ovms_server.pl.
But, that said, I’m not really sure why we would want this. Our goal has always been to mirror what it says on the little VDS in the car, and that doesn’t show these debug-only alerts. The solution we have is not perfect, but it does avoid the worst of them. I have received so many support questions and concerns from roadster users of OVMS asking about these over the years - Tesla is aware that they false positive so labelled them as debug-only, so why would we want to bombard the user with these alerts every time he drives/charges the car?
Regards, Mark.
On 31 Oct 2019, at 7:39 AM, Greg D. <gregd2350@gmail.com <mailto:gregd2350@gmail.com>> wrote:
Bummer.
Can we make this configurable? While I understand that debug messages aren't intended to be seen by the user, in the 10 years since these cars launched we have come to know a lot more about some of the, um, "quirks" in their design. These alerts can be important!
Greg
Mark Webb-Johnson wrote:
Greg,
Yes, this change to the server code:
* bcc02c27 Ignore DMC push alert messages
That won’t notify debug-only messages, but will still continue to notify the user-visible ones (to better mimick the behaviour in the car itself). Including, for the roadster:
* 1144=DMC: Powertrain Problem Service Required * 1146=DMC: Motor Fan Problem
Regards, Mark.
On 31 Oct 2019, at 7:01 AM, Greg D. <gregd2350@gmail.com <mailto:gregd2350@gmail.com>> wrote:
Hi folks (Mark, probably),
Interesting... My Tesla Roadster has been having trouble with its PEM fan, causing a bunch of 1146 Alerts to be generated while driving and/or charging. I've been monitoring the situation via OVMSv3.
The alerts suddenly stopped a bit over a week ago. Other events continue to be received (charge start, stop, key in / door open, etc.), but not the 1146's. I do a monthly download of the car logs, and to my surprise (and disappointment), the 1146s are still happening.
Was there a change to the plumbing somewhere such that these diagnostic alerts (perhaps versus user-visible events) are now being filtered out?
Thanks,
Greg
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
On Thu, 31 Oct 2019, Mark Webb-Johnson wrote:
To do this properly would require ovms doing what the vds does - have an alert code -> debug/user,message table and do the conversion and filtering there (not in the server). But that is quite a bit of code flash memory.
It would not be necessary to convert to text in OVMS, just have a bit array to indicate debug/user. That wouldn't take so much space. (I don't know whether the code to support bit arrays is already included from the existing libraries.) -- Steve
participants (5)
-
Chip Cangialose -
Greg D. -
Mark Webb-Johnson -
Michael Balzer -
Stephen Casner