<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I tried to replicate the fault on both my desk and car modules. Neither could repeat the fault (using IDF version you requested). I guess in that case, no point trying the fix?<div class=""><br class=""></div><div class="">I am wondering if completely wiping flash, and re-installing from scratch, would work to recreate the problem?</div><div class=""><br class=""></div><div class="">Anyway, I will try again tonight on a couple of brand new modules, to see if we can see the problem there. Probably good to take a raw backup of the flash partition first.</div><div class=""><br class=""></div><div class="">Regards, Mark.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 28 Nov 2018, at 5:35 AM, Michael Balzer <<a href="mailto:dexter@expeedo.de" class="">dexter@expeedo.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
@all: we need some help on this.<br class="">
<br class="">
Dmitry has asked us to verify an assumption about the source of the
problem. Unfortunately, my module has decided to not show the bug
anymore.<br class="">
<br class="">
Can anyone a) reproduce the bug and b) check Dmitry's code change?
The code change is very simple:<br class="">
<blockquote class=""><a class="moz-txt-link-freetext" href="https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779">https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779</a><br class="">
</blockquote>
…that's in file <tt class="">esp-idf/components/wear_levelling/WL_Flash.cpp</tt>
line 183.<br class="">
<br class="">
Chances are the bug is more frequently present on V3.1 first batch
modules.<br class="">
<br class="">
<u class="">To test a build including the bug, do</u>:<br class="">
<blockquote class=""><tt class="">OVMS# config backup /sd/cfgbackup.zip</tt><br class="">
<br class="">
<tt class="">~/…/esp-idf> git checkout
5afc1b0cbbcd5903744cd2948ee235b7016eeede</tt><br class="">
<tt class="">
~/…/esp-idf> git merge --no-commit
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8</tt><br class="">
<tt class="">
~/…/esp-idf> git submodule update --recursive</tt><br class="">
<tt class="">
</tt><br class="">
<tt class="">~/…/OVMS.V3> git checkout
c3c908c00a09c4a841d1fec115be785c8f486270</tt><br class="">
<tt class="">~/…/OVMS.V3> make app-flash</tt><br class="">
</blockquote>
Then do some reboots of the module.<br class="">
<br class="">
If the bug shows up, usually the second reboot already will result
in a lost configuration. You'll see some warnings in the boot
process (mount failure), and the prompt will directly be "#" as no
module password is in place.<br class="">
<br class="">
If that happens, rebuild after applying the change as described by
Dmitry, restore your config (see below), check if the bug is gone.
If it isn't, the change <u class="">definitely doesn't work</u>.<br class="">
<br class="">
If it's gone, as the bug can stop showing up, you need to revert
Dmitry's change and check once again if the bug reappears. If it
does, the change <u class="">definitely works</u>.<br class="">
<br class="">
If the bug doesn't reappear, you need to switch off the module for
some hours, then redo the above steps.<br class="">
<br class="">
<br class="">
<u class="">Note on restoring the config backup</u>: as my later restore fix
isn't included in that version, you may need to create the events
directory manually:<br class="">
<blockquote class=""><tt class="">OVMS# vfs mkdir /store/events</tt><tt class=""><br class="">
</tt><tt class="">
OVMS# config restore /sd/cfgbackup.zip <password></tt><tt class=""><br class="">
</tt></blockquote>
To return to normal build setup:<br class="">
<blockquote class=""><tt class="">~/…/esp-idf> git checkout -f master</tt><tt class=""><br class="">
</tt><tt class="">
~/…/esp-idf> git submodule update --recursive</tt><tt class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">~/…/OVMS.V3> git checkout master</tt><br class="">
</blockquote>
<br class="">
Please report definitive results directly to Dmitry in the issue
#2730.<br class="">
<br class="">
Thanks,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 27.11.18 um 02:02 schrieb Mark
Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:5C2B0ED3-BFA4-4629-9892-2D1B50B65EF4@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
I think unmount should be on the system.shutdown event. That would
be after system.shuttingdown and the
MyBoot.RestartPending/RestartReady control steps.
<div class=""><br class="">
</div>
<div class="">That means if a component really wants to write to
config (or elsewhere on /store), it should do it during
system.shuttingdown event (and optionally do
the MyBoot.RestartPending(TAG) and MyBoot.RestartReady(TAG)
thing, like simcom).</div>
<div class=""><br class="">
</div>
<div class="">As you say, that doesn’t address the
crash-during-shuttingdown issue, but can be done in a very
component manner. No changes to MyBoot - all done in
ovms_config. Or just hard-code the /store unmount before
esp_restart().</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 27 Nov 2018, at 12:47 AM, Michael Balzer
<<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""> Dimitris
question about unclean shutdowns let me check our reboot
code, and we actually do not take care of unmounting the
config store.<br class="">
<br class="">
While my crash observations do not look like crash/clean
makes a difference to this bug, I still think we should
do that.<br class="">
<br class="">
The simple solution would be to do this last, in
Boot::boot_shutdown_done() just before the
esp_restart(), but that does not take care of crashes
during shutdown, which still are too frequent.<br class="">
<br class="">
How about unmounting /store ASAP after the shuttingdown
event (i.e. normal shutdown handling)?<br class="">
Writing to the config would then have no effect during
shutdown, but do we actually need (want) that to be
possible?<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 25.11.18 um 03:12
schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite" cite="mid:AD733748-1B40-447A-BC16-32B037A7227A@webb-johnson.net" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
I second this. A fantastic effort, Michael.
<div class=""><br class="">
</div>
<div class="">The log you provided on the GitHub issue
looks really helpful, and I see Dimitri has replied:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none;
padding:
0px;" class="">
<div class="">
<div class="clearfix timeline-comment-header" style="box-sizing: border-box; background-color:
rgb(246,
248, 250); border-bottom-width:
1px; border-bottom-style:
solid;
border-bottom-color: rgb(209, 213, 218);
border-top-left-radius: 3px;
border-top-right-radius: 3px;
color:
rgb(88, 96, 105); padding-left: 15px;
padding-right:
15px; font-family:
-apple-system, BlinkMacSystemFont,
"Segoe UI", Helvetica, Arial,
sans-serif,
"Apple Color Emoji",
"Segoe UI Emoji",
"Segoe UI
Symbol"; font-size: 14px;">
<h3 class=" text-normal timeline-comment-header-text
f5" style="box-sizing: border-box;
margin-bottom: 0px;
margin-top: 0px;
font-size: 14px !important; font-weight:
400 !important; max-width: 78%;
padding-bottom: 10px;
padding-top: 10px;"><span class="css-truncate" style="box-sizing:
border-box; font-weight: 600;"><a class=" text-inherit author
css-truncate-target" data-hovercard-type="user" data-hovercard-url="/hovercards?user_id=32667452" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dmitry1945" aria-describedby="hovercard-aria-description" style="box-sizing: border-box; color:
rgb(88, 96,
105); text-decoration:
none; display: inline-block;
max-width: 125px; overflow: hidden;
text-overflow:
ellipsis;
vertical-align: top; white-space: nowrap;" moz-do-not-send="true">dmitry1945</a> </span>commented <a href="https://github.com/espressif/esp-idf#issuecomment-441391665" id="issuecomment-441391665-permalink" class="js-timestamp timestamp" style="box-sizing:
border-box; color:
inherit; text-decoration: none;
white-space: nowrap;" moz-do-not-send="true">6
hours ago</a><span class="js-comment-fragment" style="box-sizing:
border-box;"><include-fragment class="js-comment-edit-history d-inline" style="box-sizing: border-box; display:
inline
!important;"></include-fragment></span></h3>
</div>
<div class="edit-comment-hide" style="box-sizing:
border-box;
caret-color: rgb(36, 41, 46);
color: rgb(36, 41, 46);
font-family:
-apple-system, BlinkMacSystemFont,
"Segoe
UI", Helvetica, Arial,
sans-serif, "Apple Color
Emoji",
"Segoe UI Emoji", "Segoe UI
Symbol"; font-size: 14px; background-color:
rgb(255,
255, 255);"><task-lists disabled="" sortable="" style="box-sizing:
border-box;" class="">
<table class="d-block" style="box-sizing:
border-box;
border-collapse: collapse;
border-spacing: 0px; display:
block
!important;">
<tbody class="d-block" style="box-sizing:
border-box;
display: block
!important;">
<tr class="d-block" style="box-sizing:
border-box;
display: block
!important;">
<td class=" markdown-body comment-body js-comment-body d-block
" style="box-sizing: border-box;
padding: 15px; font-size: 14px;
line-height: 1.5;
word-wrap:
break-word; overflow: visible;
width:
698px; display: block
!important;">
<div style="box-sizing: border-box;
margin-bottom:
0px !important;
margin-top: 0px !important;" class="">Thank you <a class="user-mention" data-hovercard-type="user" data-hovercard-url="/hovercards?user_id=2706753" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dexterbg" aria-describedby="hovercard-aria-description" style="box-sizing:
border-box; color: rgb(36,
41, 46); text-decoration: none;
font-weight:
600;
white-space: nowrap;" moz-do-not-send="true">@dexterbg</a>,
the place is clear.</div>
</td>
</tr>
</tbody>
</table>
</task-lists></div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">Hopefully Dimitri can find the issue
from here. So many little bugs in wifi and
bluetooth stacks causing us random issues - it
would be helpful to be able to update to the
latest IDF.</div>
<div class=""><br class="">
</div>
<div class="">Thanks, and Regards,</div>
<div class="">Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 25 Nov 2018, at 3:22 AM,
Stephen Casner <<a href="mailto:casner@acm.org" class="" moz-do-not-send="true">casner@acm.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Michael -- Kudos for this
Herculean effort -- Steve<br class="">
<br class="">
On Sat, 24 Nov 2018, Michael Balzer wrote:<br class="">
<br class="">
<blockquote type="cite" class="">Narrowing
this down is a real PITA. The effect
sometimes stops to occur, I then need to
leave the module powered down for a while
to get the effect back.<br class="">
Sometimes 1 hour is sufficient, currently
it's been off for 2 hours and still works.
I had a test window yesterday evening, one
this morning and one in the<br class="">
afternoon. Temperature is most probably
irrelevant, as the last power downs were
outside in ~4-5 °C to rule this out.<br class="">
<br class="">
So as a "passed" can always be a false
positive I need to validate every passed
step by switching back to a failing
version and see if that still fails.<br class="">
<br class="">
It seems I now have to wait until tomorrow
for the next test window, but I have
bisected down to this range of just 8
commits:<br class="">
<br class="">
balzer@leela:~/esp/esp-idf> git
rev-list
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
^9d2f7c60d9aef9860c61c2756318ada68c80fddf<br class="">
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8<br class="">
f392727abf7d56490c2f33127a59bfac42c937e0<br class="">
e834d6fffc23a6fcfc0d2e871c9235417a7fb48f<br class="">
35842d02abb5f574aaab466d46081a232fdd20a6<br class="">
f05f3fbde87a9ce45c6818f71b49cd13888fd457<br class="">
a6d6c58ecadb9759a0bacf35cd7332ac641e598d<br class="">
321b1e02052de95db60ddce87eecce5f9e04e9b8<br class="">
40486c872345584d34949b3ce83f9e956a7eea13<br class="">
<br class="">
...with
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
being the last identified bad commit, and
9d2f7c60d9aef9860c61c2756318ada68c80fddf
being the last good.<br class="">
<br class="">
If I should guess now, it's probably one
of Dmitry's commits on the wear leveling
code.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 23.11.18 um 17:20 schrieb Michael
Balzer:<br class="">
<blockquote type="cite" class="">It's not
a timing issue, I've let it reboot about
30 times without any successful mount
after the first failure.<br class="">
<br class="">
Going into bisecting now...<br class="">
<br class="">
<br class="">
Am 23.11.18 um 15:54 schrieb Mark
Webb-Johnson:<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">It
may actually not be a corruption of
the filesystem but some timing issue
on the mount procedure. To test that
we could disable the auto formatting
on<br class="">
mount failures.<br class="">
</blockquote>
<br class="">
True.<br class="">
<br class="">
A couple of Espressif guys have jumped
on the issue, and I have provided some
more information for them. I think key
will be reproducing it.<br class="">
<br class="">
<blockquote type="cite" class="">The
issue may also be dependant on the
hardware version, i.e. it could be
caused by the bug that caused the SD
speed issue on the first 3.1 batch.<br class="">
</blockquote>
<br class="">
That was definitely a hardware issue
with the CP2102 chip. I don't think
related to ESP in any way.<br class="">
<br class="">
Regards, Mark.<br class="">
<br class="">
<blockquote type="cite" class="">On 23
Nov 2018, at 10:34 PM, Michael
Balzer <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>
<<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
wrote:<br class="">
<br class="">
It may actually not be a corruption
of the filesystem but some timing
issue on the mount procedure. To
test that we could disable the auto
formatting on<br class="">
mount failures.<br class="">
<br class="">
The issue may also be dependant on
the hardware version, i.e. it could
be caused by the bug that caused the
SD speed issue on the first 3.1
batch.<br class="">
<br class="">
I only have tried the idf update on
my batch 1 module (my bench /
development module). I think most of
our edge testers also have that
version.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 23.11.18 um 02:32 schrieb Mark
Webb-Johnson:<br class="">
<blockquote type="cite" class="">I
have raised the following github
issue to Espressif:<br class="">
<br class="">
<a href="https://github.com/espressif/esp-idf/issues/2730" class="" moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730</a><br class="">
<br class="">
<br class="">
Environment<br class="">
<br class="">
* Development Kit: none<br class="">
* Kit version (for
WroverKit/PicoKit/DevKitC): none<br class="">
* Module or chip used:
ESP32-WROVER 16MB<br class="">
* IDF version (run |git
describe --tags| to find it):
v3.2-beta1-208-g0d7f2d77c<br class="">
* Build System: make<br class="">
* Compiler version (run
|xtensa-esp32-elf-gcc --version|
to find it): (crosstool-NG
crosstool-ng-1.22.0-80-g6c4433a)
5.2.0<br class="">
* Operating System: macOS<br class="">
* Power Supply: USB<br class="">
<br class="">
<br class="">
Problem Description<br class="">
<br class="">
TLDR: Between May and July 2018
a change was made to esp idf
master that is causing corruption
on FAT filesystems mounted on SPI
flash.<br class="">
<br class="">
Our project uses a
partitions.csv as follows:<br class="">
<br class="">
|# Name, Type, SubType, Offset,
Size nvs, data, nvs, 0x9000,
0x4000 otadata, data, ota, 0xd000,
0x2000 phy_init, data, phy,
0xf000, 0x1000 factory,<br class="">
app, factory, 0x10000, 4M
ota_0, app, ota_0, , 4M ota_1,
app, ota_1, , 4M store, data, fat,
, 1M |<br class="">
<br class="">
The 'store' partition is
formatted as FAT, as follows:<br class="">
<br class="">
esp_vfs_fat_mount_config_t
m_store_fat;<br class="">
wl_handle_t m_store_wlh;<br class="">
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br class="">
m_store_fat.format_if_mount_failed = true;<br class="">
m_store_fat.max_files = 5;<br class="">
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);<br class="">
<br class="">
We have previously used a clone
of esp idf master, dated around
May 22 2018, without issues. The
partition is very reliable.<br class="">
<br class="">
However, on Jul 6 2018, we
updated our clone to use the
latest esp idf master at that
time. Shortly afterwards, users
started to report that their<br class="">
'store' filesystem contents
were corrupted. We rolled back.<br class="">
<br class="">
We have now tried again
(updating on Oct 20 2018 to
v3.2-beta1-208-g0d7f2d77c) and
immediately had the same issue.
Random corruption of FAT
filesystem<br class="">
in SPI flash.<br class="">
<br class="">
<br class="">
Expected Behavior<br class="">
<br class="">
No corruption of FAT
filesystem.<br class="">
<br class="">
<br class="">
Actual Behavior<br class="">
<br class="">
Corruption of FAT filesystem.<br class="">
<br class="">
<br class="">
Steps to reproduce<br class="">
<br class="">
1. Create a partition in SPI
flash, and mount FAT filesystem<br class="">
2. Read and write to files on
FAT filesystem<br class="">
3. Reboot<br class="">
4. Observe random corruption
and unmountable filesystem<br class="">
<br class="">
<br class="">
Code to reproduce this
issue<br class="">
<br class="">
esp_vfs_fat_mount_config_t
m_store_fat;<br class="">
wl_handle_t m_store_wlh;<br class="">
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br class="">
m_store_fat.format_if_mount_failed = true;<br class="">
m_store_fat.max_files = 5;<br class="">
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);<br class="">
<br class="">
<br class="">
Debug Logs<br class="">
<br class="">
n/a<br class="">
<br class="">
<br class="">
Other items if possible<br class="">
<br class="">
Please advise if you need
anything further.<br class="">
<br class="">
<br class="">
I think the timeline is correct
(the issue is in esp idf master
some time between May and July
2018), but please let me know if
you know differently (or<br class="">
update the github issue with your
comments).<br class="">
<br class="">
Regards, Mark<br class="">
<br class="">
<blockquote type="cite" class="">On
23 Nov 2018, at 6:19 AM, Michael
Balzer <<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">dexter@expeedo.de</a>
<<a href="mailto:dexter@expeedo.de" class="" moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
wrote:<br class="">
<br class="">
esp-idf and OVMS branches are
back to the working version.<br class="">
<br class="">
In case you also lost your
config: I also just fixed a bug
on restoring into an empty
/store partition.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 22.11.18 um 22:34 schrieb
Michael Balzer:<br class="">
<blockquote type="cite" class="">See
<a href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165" class="" moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165</a><br class="">
<br class="">
I'll reset both master
branches now.<br class="">
<br class="">
If you're about to pull,
please wait until I've
reverted the branches.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger
Weg 9 * D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy
0176 / 206 989 26<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br class="">
<a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</blockquote>
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger Weg 9
* D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176 /
206 989 26<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br class="">
<a href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" class="" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</blockquote>
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger Weg 9 *
D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176 / 206
989 26<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger Weg 9 *
D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176 / 206
989 26<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
--
Steve_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br class="">
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.openvehicles.com" class="">OvmsDev@lists.openvehicles.com</a><br class="">http://lists.openvehicles.com/mailman/listinfo/ovmsdev<br class=""></div></blockquote></div><br class=""></div></body></html>