<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Dmitry found out we had a bad timing with our merge. The bug was
fixed by him just five days later:<br>
<br>
<a class="moz-txt-link-freetext" href="https://github.com/espressif/esp-idf/commit/82eca97300410dd743de7f116836879cf3cdc6ba">https://github.com/espressif/esp-idf/commit/82eca97300410dd743de7f116836879cf3cdc6ba</a><br>
<br>
I have merged the latest upstream HEAD locally now. It builds
correctly, with just one little issue in the Kconfig for Smart ED,
and seems to work correctly as well. I'll do some testing now.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 27.12.18 um 11:03 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:f5e8a4fa-e23d-9a5d-1459-fa8053d74fd9@expeedo.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Update: I managed to reproduce the bug, and Dmitry's assumption is
correct.<br>
<br>
The wear leveling system attempts to do an update when mounting a
partition formatted with the old version. That update fails and
leads to the next reboot failing to mount the partition.<br>
<br>
So the WL update process needs to be fixed.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 27.11.18 um 22:35 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:b31dd300-d7b3-0850-295d-19f7e34eb736@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
@all: we need some help on this.<br>
<br>
Dmitry has asked us to verify an assumption about the source of
the problem. Unfortunately, my module has decided to not show
the bug anymore.<br>
<br>
Can anyone a) reproduce the bug and b) check Dmitry's code
change? The code change is very simple:<br>
<blockquote><a class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779"
moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779</a><br>
</blockquote>
…that's in file <tt>esp-idf/components/wear_levelling/WL_Flash.cpp</tt>
line 183.<br>
<br>
Chances are the bug is more frequently present on V3.1 first
batch modules.<br>
<br>
<u>To test a build including the bug, do</u>:<br>
<blockquote><tt>OVMS# config backup /sd/cfgbackup.zip</tt><br>
<br>
<tt>~/…/esp-idf> git checkout
5afc1b0cbbcd5903744cd2948ee235b7016eeede</tt><br>
<tt> ~/…/esp-idf> git merge --no-commit
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8</tt><br>
<tt> ~/…/esp-idf> git submodule update --recursive</tt><br>
<tt> </tt><br>
<tt>~/…/OVMS.V3> git checkout
c3c908c00a09c4a841d1fec115be785c8f486270</tt><br>
<tt>~/…/OVMS.V3> make app-flash</tt><br>
</blockquote>
Then do some reboots of the module.<br>
<br>
If the bug shows up, usually the second reboot already will
result in a lost configuration. You'll see some warnings in the
boot process (mount failure), and the prompt will directly be
"#" as no module password is in place.<br>
<br>
If that happens, rebuild after applying the change as described
by Dmitry, restore your config (see below), check if the bug is
gone. If it isn't, the change <u>definitely doesn't work</u>.<br>
<br>
If it's gone, as the bug can stop showing up, you need to revert
Dmitry's change and check once again if the bug reappears. If it
does, the change <u>definitely works</u>.<br>
<br>
If the bug doesn't reappear, you need to switch off the module
for some hours, then redo the above steps.<br>
<br>
<br>
<u>Note on restoring the config backup</u>: as my later restore
fix isn't included in that version, you may need to create the
events directory manually:<br>
<blockquote><tt>OVMS# vfs mkdir /store/events</tt><tt><br>
</tt><tt> OVMS# config restore /sd/cfgbackup.zip
<password></tt><tt><br>
</tt></blockquote>
To return to normal build setup:<br>
<blockquote><tt>~/…/esp-idf> git checkout -f master</tt><tt><br>
</tt><tt> ~/…/esp-idf> git submodule update --recursive</tt><tt><br>
</tt><tt><br>
</tt><tt>~/…/OVMS.V3> git checkout master</tt><br>
</blockquote>
<br>
Please report definitive results directly to Dmitry in the issue
#2730.<br>
<br>
Thanks,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 27.11.18 um 02:02 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite"
cite="mid:5C2B0ED3-BFA4-4629-9892-2D1B50B65EF4@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
I think unmount should be on the system.shutdown event. That
would be after system.shuttingdown and the
MyBoot.RestartPending/RestartReady control steps.
<div class=""><br class="">
</div>
<div class="">That means if a component really wants to write
to config (or elsewhere on /store), it should do it during
system.shuttingdown event (and optionally do
the MyBoot.RestartPending(TAG) and MyBoot.RestartReady(TAG)
thing, like simcom).</div>
<div class=""><br class="">
</div>
<div class="">As you say, that doesn’t address the
crash-during-shuttingdown issue, but can be done in a very
component manner. No changes to MyBoot - all done in
ovms_config. Or just hard-code the /store unmount before
esp_restart().</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 27 Nov 2018, at 12:47 AM, Michael
Balzer <<a href="mailto:dexter@expeedo.de" class=""
moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
Dimitris question about unclean shutdowns let me
check our reboot code, and we actually do not take
care of unmounting the config store.<br class="">
<br class="">
While my crash observations do not look like
crash/clean makes a difference to this bug, I still
think we should do that.<br class="">
<br class="">
The simple solution would be to do this last, in
Boot::boot_shutdown_done() just before the
esp_restart(), but that does not take care of
crashes during shutdown, which still are too
frequent.<br class="">
<br class="">
How about unmounting /store ASAP after the
shuttingdown event (i.e. normal shutdown handling)?<br
class="">
Writing to the config would then have no effect
during shutdown, but do we actually need (want) that
to be possible?<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 25.11.18 um 03:12
schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:AD733748-1B40-447A-BC16-32B037A7227A@webb-johnson.net"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
I second this. A fantastic effort, Michael.
<div class=""><br class="">
</div>
<div class="">The log you provided on the GitHub
issue looks really helpful, and I see Dimitri
has replied:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px;
border:
 none;
 padding:
 0px;"
class="">
<div class="">
<div class="clearfix timeline-comment-header"
style="box-sizing: border-box;

background-color:
 rgb(246,

248,
 250); border-bottom-width:

1px;
 border-bottom-style:

solid;

 border-bottom-color:
rgb(209, 213,
 218);


border-top-left-radius:
 3px;

border-top-right-radius: 3px;


color:
 rgb(88, 96, 105);
padding-left:
 15px;

padding-right:
 15px;

font-family:
 -apple-system,

BlinkMacSystemFont,


"Segoe
 UI", Helvetica,
Arial,

 sans-serif,

"Apple Color
 Emoji",

"Segoe UI
 Emoji",

"Segoe UI

 Symbol";
font-size: 14px;">
<h3
class="timeline-comment-header-text


text-normal f5" style="box-sizing:

border-box;
 margin-bottom:
0px;

 margin-top: 0px;

font-size: 14px
 !important;
font-weight:

 400

!important; max-width: 78%;


padding-bottom: 10px;

padding-top:
 10px;"><span
class="css-truncate"
style="box-sizing:
 border-box;

font-weight: 600;"><a
class="css-truncate-target
author

 text-inherit"
data-hovercard-type="user"
data-hovercard-url="/hovercards?user_id=32667452"
data-octo-click="hovercard-link-click"
data-octo-dimensions="link_type:self"
href="https://github.com/dmitry1945"
aria-describedby="hovercard-aria-description"
style="box-sizing: border-box;

color:
 rgb(88, 96,

105);
 text-decoration:
 none;
display:
 inline-block;


max-width:
 125px; overflow:
hidden;

 text-overflow:

ellipsis;

 vertical-align:
top; white-space:
 nowrap;"
moz-do-not-send="true">dmitry1945</a> </span>commented <a
href="https://github.com/espressif/esp-idf#issuecomment-441391665"
id="issuecomment-441391665-permalink"
class="js-timestamp timestamp"
style="box-sizing:
 border-box;

color:
 inherit;
text-decoration:
 none;


white-space: nowrap;"
moz-do-not-send="true">6 hours ago</a><span
class="js-comment-fragment"
style="box-sizing:
 border-box;"><include-fragment
class="js-comment-edit-history
d-inline" style="box-sizing:
border-box;
 display:

inline
 !important;"></include-fragment></span></h3>
</div>
<div class="edit-comment-hide"
style="box-sizing:

border-box;

 caret-color: rgb(36,
41, 46);
 color:
 rgb(36, 41,
46);
 font-family:


-apple-system, BlinkMacSystemFont,


"Segoe
 UI", Helvetica,

Arial,
 sans-serif, "Apple
Color

 Emoji",

"Segoe UI Emoji",
 "Segoe
UI

 Symbol";
 font-size:
14px; background-color:


rgb(255,
 255, 255);"><task-lists
disabled="" sortable=""
style="box-sizing:
 border-box;"
class="">
<table class="d-block"
style="box-sizing:

border-box;

 border-collapse:
collapse;

 border-spacing: 0px;
display:

 block

!important;">
<tbody class="d-block"
style="box-sizing:

border-box;

 display:
block
 !important;">
<tr class="d-block"
style="box-sizing:


border-box;
 display:
block

 !important;">
<td class=" markdown-body

d-block


comment-body
 js-comment-body"
style="box-sizing:

border-box;

 padding:
15px;
 font-size: 14px;

line-height:
 1.5;

word-wrap:

 break-word;
overflow: visible;


width:
 698px; display:

block
 !important;">
<div style="box-sizing:

border-box;


margin-bottom:
 0px

!important;
 margin-top:
0px
 !important;" class="">Thank
you <a class="user-mention"
data-hovercard-type="user"
data-hovercard-url="/hovercards?user_id=2706753"
data-octo-click="hovercard-link-click"
data-octo-dimensions="link_type:self"
href="https://github.com/dexterbg"
aria-describedby="hovercard-aria-description"
style="box-sizing:


border-box; color:

rgb(36,

 41,
46);
 text-decoration:
none;


font-weight:

600;

 white-space:
nowrap;"
moz-do-not-send="true">@dexterbg</a>,
the place is clear.</div>
</td>
</tr>
</tbody>
</table>
</task-lists></div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">Hopefully Dimitri can find the
issue from here. So many little bugs in wifi
and bluetooth stacks causing us random issues
- it would be helpful to be able to update to
the latest IDF.</div>
<div class=""><br class="">
</div>
<div class="">Thanks, and Regards,</div>
<div class="">Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 25 Nov 2018, at 3:22 AM,
Stephen Casner <<a
href="mailto:casner@acm.org" class=""
moz-do-not-send="true">casner@acm.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Michael -- Kudos for this
Herculean effort -- Steve<br class="">
<br class="">
On Sat, 24 Nov 2018, Michael Balzer
wrote:<br class="">
<br class="">
<blockquote type="cite" class="">Narrowing
this down is a real PITA. The effect
sometimes stops to occur, I then need
to leave the module powered down for a
while to get the effect back.<br
class="">
Sometimes 1 hour is sufficient,
currently it's been off for 2 hours
and still works. I had a test window
yesterday evening, one this morning
and one in the<br class="">
afternoon. Temperature is most
probably irrelevant, as the last power
downs were outside in ~4-5 °C to rule
this out.<br class="">
<br class="">
So as a "passed" can always be a false
positive I need to validate every
passed step by switching back to a
failing version and see if that still
fails.<br class="">
<br class="">
It seems I now have to wait until
tomorrow for the next test window, but
I have bisected down to this range of
just 8 commits:<br class="">
<br class="">
balzer@leela:~/esp/esp-idf> git
rev-list
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
^9d2f7c60d9aef9860c61c2756318ada68c80fddf<br class="">
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8<br class="">
f392727abf7d56490c2f33127a59bfac42c937e0<br class="">
e834d6fffc23a6fcfc0d2e871c9235417a7fb48f<br class="">
35842d02abb5f574aaab466d46081a232fdd20a6<br class="">
f05f3fbde87a9ce45c6818f71b49cd13888fd457<br class="">
a6d6c58ecadb9759a0bacf35cd7332ac641e598d<br class="">
321b1e02052de95db60ddce87eecce5f9e04e9b8<br class="">
40486c872345584d34949b3ce83f9e956a7eea13<br class="">
<br class="">
...with
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
being the last identified bad commit,
and
9d2f7c60d9aef9860c61c2756318ada68c80fddf
being the last good.<br class="">
<br class="">
If I should guess now, it's probably
one of Dmitry's commits on the wear
leveling code.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 23.11.18 um 17:20 schrieb Michael
Balzer:<br class="">
<blockquote type="cite" class="">It's
not a timing issue, I've let it
reboot about 30 times without any
successful mount after the first
failure.<br class="">
<br class="">
Going into bisecting now...<br
class="">
<br class="">
<br class="">
Am 23.11.18 um 15:54 schrieb Mark
Webb-Johnson:<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">It
may actually not be a corruption
of the filesystem but some
timing issue on the mount
procedure. To test that we could
disable the auto formatting on<br
class="">
mount failures.<br class="">
</blockquote>
<br class="">
True.<br class="">
<br class="">
A couple of Espressif guys have
jumped on the issue, and I have
provided some more information for
them. I think key will be
reproducing it.<br class="">
<br class="">
<blockquote type="cite" class="">The
issue may also be dependant on
the hardware version, i.e. it
could be caused by the bug that
caused the SD speed issue on the
first 3.1 batch.<br class="">
</blockquote>
<br class="">
That was definitely a hardware
issue with the CP2102 chip. I
don't think related to ESP in any
way.<br class="">
<br class="">
Regards, Mark.<br class="">
<br class="">
<blockquote type="cite" class="">On
23 Nov 2018, at 10:34 PM,
Michael Balzer <<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">dexter@expeedo.de</a>
<<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
wrote:<br class="">
<br class="">
It may actually not be a
corruption of the filesystem but
some timing issue on the mount
procedure. To test that we could
disable the auto formatting on<br
class="">
mount failures.<br class="">
<br class="">
The issue may also be dependant
on the hardware version, i.e. it
could be caused by the bug that
caused the SD speed issue on the
first 3.1 batch.<br class="">
<br class="">
I only have tried the idf update
on my batch 1 module (my bench /
development module). I think
most of our edge testers also
have that version.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 23.11.18 um 02:32 schrieb
Mark Webb-Johnson:<br class="">
<blockquote type="cite" class="">I
have raised the following
github issue to Espressif:<br
class="">
<br class="">
<a
href="https://github.com/espressif/esp-idf/issues/2730"
class=""
moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730</a><br
class="">
<br class="">
<br class="">
Environment<br class="">
<br class="">
* Development Kit: none<br
class="">
* Kit version (for
WroverKit/PicoKit/DevKitC):
none<br class="">
* Module or chip used:
ESP32-WROVER 16MB<br class="">
* IDF version (run |git
describe --tags| to find it):
v3.2-beta1-208-g0d7f2d77c<br
class="">
* Build System: make<br
class="">
* Compiler version (run
|xtensa-esp32-elf-gcc
--version| to find it):
(crosstool-NG
crosstool-ng-1.22.0-80-g6c4433a)
5.2.0<br class="">
* Operating System: macOS<br
class="">
* Power Supply: USB<br
class="">
<br class="">
<br class="">
Problem Description<br
class="">
<br class="">
TLDR: Between May and July
2018 a change was made to esp
idf master that is causing
corruption on FAT filesystems
mounted on SPI flash.<br
class="">
<br class="">
Our project uses a
partitions.csv as follows:<br
class="">
<br class="">
|# Name, Type, SubType,
Offset, Size nvs, data, nvs,
0x9000, 0x4000 otadata, data,
ota, 0xd000, 0x2000 phy_init,
data, phy, 0xf000, 0x1000
factory,<br class="">
app, factory, 0x10000, 4M
ota_0, app, ota_0, , 4M ota_1,
app, ota_1, , 4M store, data,
fat, , 1M |<br class="">
<br class="">
The 'store' partition is
formatted as FAT, as follows:<br
class="">
<br class="">
esp_vfs_fat_mount_config_t
m_store_fat;<br class="">
wl_handle_t m_store_wlh;<br
class="">
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
class="">
m_store_fat.format_if_mount_failed = true;<br class="">
m_store_fat.max_files = 5;<br
class="">
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);<br class="">
<br class="">
We have previously used a
clone of esp idf master, dated
around May 22 2018, without
issues. The partition is very
reliable.<br class="">
<br class="">
However, on Jul 6 2018, we
updated our clone to use the
latest esp idf master at that
time. Shortly afterwards,
users started to report that
their<br class="">
'store' filesystem contents
were corrupted. We rolled
back.<br class="">
<br class="">
We have now tried again
(updating on Oct 20 2018 to
v3.2-beta1-208-g0d7f2d77c) and
immediately had the same
issue. Random corruption of
FAT filesystem<br class="">
in SPI flash.<br class="">
<br class="">
<br class="">
Expected Behavior<br
class="">
<br class="">
No corruption of FAT
filesystem.<br class="">
<br class="">
<br class="">
Actual Behavior<br
class="">
<br class="">
Corruption of FAT
filesystem.<br class="">
<br class="">
<br class="">
Steps to reproduce<br
class="">
<br class="">
1. Create a partition in
SPI flash, and mount FAT
filesystem<br class="">
2. Read and write to files
on FAT filesystem<br class="">
3. Reboot<br class="">
4. Observe random
corruption and unmountable
filesystem<br class="">
<br class="">
<br class="">
Code to reproduce
this issue<br class="">
<br class="">
esp_vfs_fat_mount_config_t
m_store_fat;<br class="">
wl_handle_t m_store_wlh;<br
class="">
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
class="">
m_store_fat.format_if_mount_failed = true;<br class="">
m_store_fat.max_files = 5;<br
class="">
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);<br class="">
<br class="">
<br class="">
Debug Logs<br class="">
<br class="">
n/a<br class="">
<br class="">
<br class="">
Other items if possible<br
class="">
<br class="">
Please advise if you need
anything further.<br class="">
<br class="">
<br class="">
I think the timeline is
correct (the issue is in esp
idf master some time between
May and July 2018), but please
let me know if you know
differently (or<br class="">
update the github issue with
your comments).<br class="">
<br class="">
Regards, Mark<br class="">
<br class="">
<blockquote type="cite"
class="">On 23 Nov 2018, at
6:19 AM, Michael Balzer <<a
href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>
<<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
wrote:<br class="">
<br class="">
esp-idf and OVMS branches
are back to the working
version.<br class="">
<br class="">
In case you also lost your
config: I also just fixed a
bug on restoring into an
empty /store partition.<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 22.11.18 um 22:34 schrieb
Michael Balzer:<br class="">
<blockquote type="cite"
class="">See <a
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165"
class=""
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165</a><br
class="">
<br class="">
I'll reset both master
branches now.<br class="">
<br class="">
If you're about to pull,
please wait until I've
reverted the branches.<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer *
Helkenberger Weg 9 * D-58256
Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy
0176 / 206 989 26<br
class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class=""
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a
class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger
Weg 9 * D-58256 Ennepetal<br
class="">
Fon 02333 / 833 5735 * Handy
0176 / 206 989 26<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class=""
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger Weg 9
* D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176 /
206 989 26<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger Weg 9 *
D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176 /
206 989 26<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
--
Steve_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</body>
</html>