<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Wifi AP mode is broken. A fix should be available in some days:<br>
<a class="moz-txt-link-freetext" href="https://github.com/espressif/esp-idf/issues/2915#issuecomment-452606571">https://github.com/espressif/esp-idf/issues/2915#issuecomment-452606571</a><br>
<br>
I haven't been able to test the modem yet. It seems to work but my
test SIM ran out of balance, so I need to wait for the payment
process to finish.<br>
<br>
Apart from that it works pretty well so far.<br>
<br>
<br>
<div class="moz-cite-prefix">Am 22.01.19 um 17:43 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:8483fdbb-36cc-6bf6-0f2f-ab07600fa9cd@expeedo.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Dmitry found out we had a bad timing with our merge. The bug was
fixed by him just five days later:<br>
<br>
<a class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/commit/82eca97300410dd743de7f116836879cf3cdc6ba"
moz-do-not-send="true">https://github.com/espressif/esp-idf/commit/82eca97300410dd743de7f116836879cf3cdc6ba</a><br>
<br>
I have merged the latest upstream HEAD locally now. It builds
correctly, with just one little issue in the Kconfig for Smart ED,
and seems to work correctly as well. I'll do some testing now.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 27.12.18 um 11:03 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:f5e8a4fa-e23d-9a5d-1459-fa8053d74fd9@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Update: I managed to reproduce the bug, and Dmitry's assumption
is correct.<br>
<br>
The wear leveling system attempts to do an update when mounting
a partition formatted with the old version. That update fails
and leads to the next reboot failing to mount the partition.<br>
<br>
So the WL update process needs to be fixed.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 27.11.18 um 22:35 schrieb
Michael Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:b31dd300-d7b3-0850-295d-19f7e34eb736@expeedo.de">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
@all: we need some help on this.<br>
<br>
Dmitry has asked us to verify an assumption about the source
of the problem. Unfortunately, my module has decided to not
show the bug anymore.<br>
<br>
Can anyone a) reproduce the bug and b) check Dmitry's code
change? The code change is very simple:<br>
<blockquote><a class="moz-txt-link-freetext"
href="https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779"
moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730#issuecomment-441928779</a><br>
</blockquote>
…that's in file <tt>esp-idf/components/wear_levelling/WL_Flash.cpp</tt>
line 183.<br>
<br>
Chances are the bug is more frequently present on V3.1 first
batch modules.<br>
<br>
<u>To test a build including the bug, do</u>:<br>
<blockquote><tt>OVMS# config backup /sd/cfgbackup.zip</tt><br>
<br>
<tt>~/…/esp-idf> git checkout
5afc1b0cbbcd5903744cd2948ee235b7016eeede</tt><br>
<tt> ~/…/esp-idf> git merge --no-commit
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8</tt><br>
<tt> ~/…/esp-idf> git submodule update --recursive</tt><br>
<tt> </tt><br>
<tt>~/…/OVMS.V3> git checkout
c3c908c00a09c4a841d1fec115be785c8f486270</tt><br>
<tt>~/…/OVMS.V3> make app-flash</tt><br>
</blockquote>
Then do some reboots of the module.<br>
<br>
If the bug shows up, usually the second reboot already will
result in a lost configuration. You'll see some warnings in
the boot process (mount failure), and the prompt will directly
be "#" as no module password is in place.<br>
<br>
If that happens, rebuild after applying the change as
described by Dmitry, restore your config (see below), check if
the bug is gone. If it isn't, the change <u>definitely
doesn't work</u>.<br>
<br>
If it's gone, as the bug can stop showing up, you need to
revert Dmitry's change and check once again if the bug
reappears. If it does, the change <u>definitely works</u>.<br>
<br>
If the bug doesn't reappear, you need to switch off the module
for some hours, then redo the above steps.<br>
<br>
<br>
<u>Note on restoring the config backup</u>: as my later
restore fix isn't included in that version, you may need to
create the events directory manually:<br>
<blockquote><tt>OVMS# vfs mkdir /store/events</tt><tt><br>
</tt><tt> OVMS# config restore /sd/cfgbackup.zip
<password></tt><tt><br>
</tt></blockquote>
To return to normal build setup:<br>
<blockquote><tt>~/…/esp-idf> git checkout -f master</tt><tt><br>
</tt><tt> ~/…/esp-idf> git submodule update --recursive</tt><tt><br>
</tt><tt><br>
</tt><tt>~/…/OVMS.V3> git checkout master</tt><br>
</blockquote>
<br>
Please report definitive results directly to Dmitry in the
issue #2730.<br>
<br>
Thanks,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 27.11.18 um 02:02 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite"
cite="mid:5C2B0ED3-BFA4-4629-9892-2D1B50B65EF4@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
I think unmount should be on the system.shutdown event. That
would be after system.shuttingdown and the
MyBoot.RestartPending/RestartReady control steps.
<div class=""><br class="">
</div>
<div class="">That means if a component really wants to
write to config (or elsewhere on /store), it should do it
during system.shuttingdown event (and optionally do
the MyBoot.RestartPending(TAG)
and MyBoot.RestartReady(TAG) thing, like simcom).</div>
<div class=""><br class="">
</div>
<div class="">As you say, that doesn’t address the
crash-during-shuttingdown issue, but can be done in a very
component manner. No changes to MyBoot - all done in
ovms_config. Or just hard-code the /store unmount before
esp_restart().</div>
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 27 Nov 2018, at 12:47 AM, Michael
Balzer <<a href="mailto:dexter@expeedo.de"
class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
Dimitris question about unclean shutdowns let me
check our reboot code, and we actually do not take
care of unmounting the config store.<br class="">
<br class="">
While my crash observations do not look like
crash/clean makes a difference to this bug, I
still think we should do that.<br class="">
<br class="">
The simple solution would be to do this last, in
Boot::boot_shutdown_done() just before the
esp_restart(), but that does not take care of
crashes during shutdown, which still are too
frequent.<br class="">
<br class="">
How about unmounting /store ASAP after the
shuttingdown event (i.e. normal shutdown
handling)?<br class="">
Writing to the config would then have no effect
during shutdown, but do we actually need (want)
that to be possible?<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 25.11.18 um 03:12
schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:AD733748-1B40-447A-BC16-32B037A7227A@webb-johnson.net"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
I second this. A fantastic effort, Michael.
<div class=""><br class="">
</div>
<div class="">The log you provided on the GitHub
issue looks really helpful, and I see Dimitri
has replied:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px;

border:
 none;
 padding:
 0px;"
class="">
<div class="">
<div class="clearfix
timeline-comment-header"
style="box-sizing: border-box;


background-color:
 rgb(246,


248,
 250);
border-bottom-width:

 1px;

border-bottom-style:


solid;

 border-bottom-color:

rgb(209, 213,
 218);



border-top-left-radius:

3px;

 border-top-right-radius:
3px;


 color:
 rgb(88,
96, 105);
 padding-left:

15px;

 padding-right:

15px;

 font-family:

-apple-system,


BlinkMacSystemFont,



"Segoe
 UI", Helvetica,

Arial,

 sans-serif,


"Apple Color

Emoji",

 "Segoe UI

Emoji",

 "Segoe
UI

 Symbol";
 font-size:
14px;">
<h3
class="timeline-comment-header-text



text-normal f5"
style="box-sizing:


border-box;
 margin-bottom:

0px;

 margin-top:
0px;

 font-size: 14px

!important;
 font-weight:


400

 !important; max-width:
78%;


 padding-bottom:
10px;

 padding-top:
 10px;"><span
class="css-truncate"
style="box-sizing:

border-box;

 font-weight:
600;"><a
class="css-truncate-target

author

 text-inherit"
data-hovercard-type="user"
data-hovercard-url="/hovercards?user_id=32667452"
data-octo-click="hovercard-link-click"
data-octo-dimensions="link_type:self"
href="https://github.com/dmitry1945"
aria-describedby="hovercard-aria-description" style="box-sizing:
border-box;

 color:

rgb(88, 96,

 105);

text-decoration:
 none;

display:

inline-block;



max-width:
 125px;
overflow:
 hidden;


text-overflow:


ellipsis;


vertical-align:
 top;
white-space:
 nowrap;"
moz-do-not-send="true">dmitry1945</a> </span>commented <a
href="https://github.com/espressif/esp-idf#issuecomment-441391665"
id="issuecomment-441391665-permalink"
class="js-timestamp timestamp"
style="box-sizing:

border-box;

 color:

inherit;
 text-decoration:

none;


 white-space:
nowrap;" moz-do-not-send="true">6
hours ago</a><span
class="js-comment-fragment"
style="box-sizing:
 border-box;"><include-fragment
class="js-comment-edit-history

d-inline" style="box-sizing:

border-box;
 display:


inline
 !important;"></include-fragment></span></h3>
</div>
<div class="edit-comment-hide"
style="box-sizing:


border-box;

 caret-color:
rgb(36,
 41, 46);
 color:

rgb(36, 41,
 46);

font-family:


 -apple-system,
BlinkMacSystemFont,



"Segoe
 UI",
Helvetica,

 Arial,

sans-serif, "Apple

Color

 Emoji",


"Segoe UI Emoji",

"Segoe
 UI


Symbol";
 font-size:
 14px;
background-color:



rgb(255,
 255, 255);"><task-lists
disabled="" sortable=""
style="box-sizing:
 border-box;"
class="">
<table class="d-block"
style="box-sizing:


border-box;


border-collapse:

collapse;

 border-spacing:
0px;
 display:


block

 !important;">
<tbody class="d-block"
style="box-sizing:


border-box;

 display:

block
 !important;">
<tr class="d-block"
style="box-sizing:



border-box;
 display:

block

 !important;">
<td class="
markdown-body


d-block



comment-body

js-comment-body"
style="box-sizing:


border-box;


padding:
 15px;

font-size: 14px;


line-height:
 1.5;


word-wrap:


break-word;
 overflow:
visible;



width:
 698px;
display:

 block

!important;">
<div
style="box-sizing:


border-box;



margin-bottom:

0px

 !important;

margin-top:
 0px

!important;" class="">Thank
you <a class="user-mention"
data-hovercard-type="user"
data-hovercard-url="/hovercards?user_id=2706753"
data-octo-click="hovercard-link-click"
data-octo-dimensions="link_type:self"
href="https://github.com/dexterbg"
aria-describedby="hovercard-aria-description"
style="box-sizing:


 border-box; color:


rgb(36,

 41,

46);

text-decoration:

none;



font-weight:


600;


white-space:
 nowrap;"
moz-do-not-send="true">@dexterbg</a>,
the place is clear.</div>
</td>
</tr>
</tbody>
</table>
</task-lists></div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">Hopefully Dimitri can find the
issue from here. So many little bugs in wifi
and bluetooth stacks causing us random
issues - it would be helpful to be able to
update to the latest IDF.</div>
<div class=""><br class="">
</div>
<div class="">Thanks, and Regards,</div>
<div class="">Mark.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 25 Nov 2018, at 3:22 AM,
Stephen Casner <<a
href="mailto:casner@acm.org" class=""
moz-do-not-send="true">casner@acm.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Michael -- Kudos for this
Herculean effort -- Steve<br class="">
<br class="">
On Sat, 24 Nov 2018, Michael Balzer
wrote:<br class="">
<br class="">
<blockquote type="cite" class="">Narrowing
this down is a real PITA. The effect
sometimes stops to occur, I then
need to leave the module powered
down for a while to get the effect
back.<br class="">
Sometimes 1 hour is sufficient,
currently it's been off for 2 hours
and still works. I had a test window
yesterday evening, one this morning
and one in the<br class="">
afternoon. Temperature is most
probably irrelevant, as the last
power downs were outside in ~4-5 °C
to rule this out.<br class="">
<br class="">
So as a "passed" can always be a
false positive I need to validate
every passed step by switching back
to a failing version and see if that
still fails.<br class="">
<br class="">
It seems I now have to wait until
tomorrow for the next test window,
but I have bisected down to this
range of just 8 commits:<br class="">
<br class="">
balzer@leela:~/esp/esp-idf> git
rev-list
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
^9d2f7c60d9aef9860c61c2756318ada68c80fddf<br class="">
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8<br class="">
f392727abf7d56490c2f33127a59bfac42c937e0<br class="">
e834d6fffc23a6fcfc0d2e871c9235417a7fb48f<br class="">
35842d02abb5f574aaab466d46081a232fdd20a6<br class="">
f05f3fbde87a9ce45c6818f71b49cd13888fd457<br class="">
a6d6c58ecadb9759a0bacf35cd7332ac641e598d<br class="">
321b1e02052de95db60ddce87eecce5f9e04e9b8<br class="">
40486c872345584d34949b3ce83f9e956a7eea13<br class="">
<br class="">
...with
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
being the last identified bad
commit, and
9d2f7c60d9aef9860c61c2756318ada68c80fddf
being the last good.<br class="">
<br class="">
If I should guess now, it's probably
one of Dmitry's commits on the wear
leveling code.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 23.11.18 um 17:20 schrieb Michael
Balzer:<br class="">
<blockquote type="cite" class="">It's
not a timing issue, I've let it
reboot about 30 times without any
successful mount after the first
failure.<br class="">
<br class="">
Going into bisecting now...<br
class="">
<br class="">
<br class="">
Am 23.11.18 um 15:54 schrieb Mark
Webb-Johnson:<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">It
may actually not be a
corruption of the filesystem
but some timing issue on the
mount procedure. To test that
we could disable the auto
formatting on<br class="">
mount failures.<br class="">
</blockquote>
<br class="">
True.<br class="">
<br class="">
A couple of Espressif guys have
jumped on the issue, and I have
provided some more information
for them. I think key will be
reproducing it.<br class="">
<br class="">
<blockquote type="cite" class="">The
issue may also be dependant on
the hardware version, i.e. it
could be caused by the bug
that caused the SD speed issue
on the first 3.1 batch.<br
class="">
</blockquote>
<br class="">
That was definitely a hardware
issue with the CP2102 chip. I
don't think related to ESP in
any way.<br class="">
<br class="">
Regards, Mark.<br class="">
<br class="">
<blockquote type="cite" class="">On
23 Nov 2018, at 10:34 PM,
Michael Balzer <<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">dexter@expeedo.de</a>
<<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
wrote:<br class="">
<br class="">
It may actually not be a
corruption of the filesystem
but some timing issue on the
mount procedure. To test that
we could disable the auto
formatting on<br class="">
mount failures.<br class="">
<br class="">
The issue may also be
dependant on the hardware
version, i.e. it could be
caused by the bug that caused
the SD speed issue on the
first 3.1 batch.<br class="">
<br class="">
I only have tried the idf
update on my batch 1 module
(my bench / development
module). I think most of our
edge testers also have that
version.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 23.11.18 um 02:32 schrieb
Mark Webb-Johnson:<br class="">
<blockquote type="cite"
class="">I have raised the
following github issue to
Espressif:<br class="">
<br class="">
<a
href="https://github.com/espressif/esp-idf/issues/2730"
class=""
moz-do-not-send="true">https://github.com/espressif/esp-idf/issues/2730</a><br
class="">
<br class="">
<br class="">
Environment<br
class="">
<br class="">
* Development Kit: none<br
class="">
* Kit version (for
WroverKit/PicoKit/DevKitC):
none<br class="">
* Module or chip used:
ESP32-WROVER 16MB<br
class="">
* IDF version (run |git
describe --tags| to find
it):
v3.2-beta1-208-g0d7f2d77c<br
class="">
* Build System: make<br
class="">
* Compiler version (run
|xtensa-esp32-elf-gcc
--version| to find it):
(crosstool-NG
crosstool-ng-1.22.0-80-g6c4433a)
5.2.0<br class="">
* Operating System:
macOS<br class="">
* Power Supply: USB<br
class="">
<br class="">
<br class="">
Problem Description<br
class="">
<br class="">
TLDR: Between May and
July 2018 a change was made
to esp idf master that is
causing corruption on FAT
filesystems mounted on SPI
flash.<br class="">
<br class="">
Our project uses a
partitions.csv as follows:<br
class="">
<br class="">
|# Name, Type, SubType,
Offset, Size nvs, data, nvs,
0x9000, 0x4000 otadata,
data, ota, 0xd000, 0x2000
phy_init, data, phy, 0xf000,
0x1000 factory,<br class="">
app, factory, 0x10000, 4M
ota_0, app, ota_0, , 4M
ota_1, app, ota_1, , 4M
store, data, fat, , 1M |<br
class="">
<br class="">
The 'store' partition is
formatted as FAT, as
follows:<br class="">
<br class="">
esp_vfs_fat_mount_config_t m_store_fat;<br class="">
wl_handle_t m_store_wlh;<br
class="">
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
class="">
m_store_fat.format_if_mount_failed = true;<br class="">
m_store_fat.max_files =
5;<br class="">
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);<br
class="">
<br class="">
We have previously used a
clone of esp idf master,
dated around May 22 2018,
without issues. The
partition is very reliable.<br
class="">
<br class="">
However, on Jul 6 2018,
we updated our clone to use
the latest esp idf master at
that time. Shortly
afterwards, users started to
report that their<br
class="">
'store' filesystem
contents were corrupted. We
rolled back.<br class="">
<br class="">
We have now tried again
(updating on Oct 20 2018 to
v3.2-beta1-208-g0d7f2d77c)
and immediately had the same
issue. Random corruption of
FAT filesystem<br class="">
in SPI flash.<br class="">
<br class="">
<br class="">
Expected Behavior<br
class="">
<br class="">
No corruption of FAT
filesystem.<br class="">
<br class="">
<br class="">
Actual Behavior<br
class="">
<br class="">
Corruption of FAT
filesystem.<br class="">
<br class="">
<br class="">
Steps to reproduce<br
class="">
<br class="">
1. Create a partition in
SPI flash, and mount FAT
filesystem<br class="">
2. Read and write to
files on FAT filesystem<br
class="">
3. Reboot<br class="">
4. Observe random
corruption and unmountable
filesystem<br class="">
<br class="">
<br class="">
Code to reproduce
this issue<br class="">
<br class="">
esp_vfs_fat_mount_config_t m_store_fat;<br class="">
wl_handle_t m_store_wlh;<br
class="">
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));<br
class="">
m_store_fat.format_if_mount_failed = true;<br class="">
m_store_fat.max_files =
5;<br class="">
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);<br
class="">
<br class="">
<br class="">
Debug Logs<br
class="">
<br class="">
n/a<br class="">
<br class="">
<br class="">
Other items if
possible<br class="">
<br class="">
Please advise if you need
anything further.<br
class="">
<br class="">
<br class="">
I think the timeline is
correct (the issue is in esp
idf master some time between
May and July 2018), but
please let me know if you
know differently (or<br
class="">
update the github issue with
your comments).<br class="">
<br class="">
Regards, Mark<br class="">
<br class="">
<blockquote type="cite"
class="">On 23 Nov 2018,
at 6:19 AM, Michael Balzer
<<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">dexter@expeedo.de</a>
<<a
href="mailto:dexter@expeedo.de"
class=""
moz-do-not-send="true">mailto:dexter@expeedo.de</a>>>
wrote:<br class="">
<br class="">
esp-idf and OVMS branches
are back to the working
version.<br class="">
<br class="">
In case you also lost your
config: I also just fixed
a bug on restoring into an
empty /store partition.<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 22.11.18 um 22:34
schrieb Michael Balzer:<br
class="">
<blockquote type="cite"
class="">See <a
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165"
class=""
moz-do-not-send="true">https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165</a><br
class="">
<br class="">
I'll reset both master
branches now.<br
class="">
<br class="">
If you're about to pull,
please wait until I've
reverted the branches.<br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer *
Helkenberger Weg 9 *
D-58256 Ennepetal<br
class="">
Fon 02333 / 833 5735 *
Handy 0176 / 206 989 26<br
class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class=""
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a
class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger
Weg 9 * D-58256 Ennepetal<br
class="">
Fon 02333 / 833 5735 * Handy
0176 / 206 989 26<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br
class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">mailto:OvmsDev@lists.openvehicles.com</a>><br
class="">
<a
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
class=""
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger Weg
9 * D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176
/ 206 989 26<br class="">
<br class="">
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</blockquote>
<br class="">
--<br class="">
Michael Balzer * Helkenberger Weg 9
* D-58256 Ennepetal<br class="">
Fon 02333 / 833 5735 * Handy 0176 /
206 989 26<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
--
Steve_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</body>
</html>