Dimitris
question about unclean shutdowns let me check our reboot
code, and we actually do not take care of unmounting the
config store.
While my crash observations do not look like crash/clean
makes a difference to this bug, I still think we should
do that.
The simple solution would be to do this last, in
Boot::boot_shutdown_done() just before the
esp_restart(), but that does not take care of crashes
during shutdown, which still are too frequent.
How about unmounting /store ASAP after the shuttingdown
event (i.e. normal shutdown handling)?
Writing to the config would then have no effect during
shutdown, but do we actually need (want) that to be
possible?
Regards,
Michael
Am 25.11.18 um 03:12
schrieb Mark Webb-Johnson:
I second this. A fantastic effort, Michael.
The log you provided on the GitHub issue
looks really helpful, and I see Dimitri has replied:
Hopefully Dimitri can find the issue
from here. So many little bugs in wifi and
bluetooth stacks causing us random issues - it
would be helpful to be able to update to the
latest IDF.
Thanks, and Regards,
Mark.
Michael -- Kudos for this
Herculean effort -- Steve
On Sat, 24 Nov 2018, Michael Balzer wrote:
Narrowing
this down is a real PITA. The effect
sometimes stops to occur, I then need to
leave the module powered down for a while
to get the effect back.
Sometimes 1 hour is sufficient, currently
it's been off for 2 hours and still works.
I had a test window yesterday evening, one
this morning and one in the
afternoon. Temperature is most probably
irrelevant, as the last power downs were
outside in ~4-5 °C to rule this out.
So as a "passed" can always be a false
positive I need to validate every passed
step by switching back to a failing
version and see if that still fails.
It seems I now have to wait until tomorrow
for the next test window, but I have
bisected down to this range of just 8
commits:
balzer@leela:~/esp/esp-idf> git
rev-list
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
^9d2f7c60d9aef9860c61c2756318ada68c80fddf
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
f392727abf7d56490c2f33127a59bfac42c937e0
e834d6fffc23a6fcfc0d2e871c9235417a7fb48f
35842d02abb5f574aaab466d46081a232fdd20a6
f05f3fbde87a9ce45c6818f71b49cd13888fd457
a6d6c58ecadb9759a0bacf35cd7332ac641e598d
321b1e02052de95db60ddce87eecce5f9e04e9b8
40486c872345584d34949b3ce83f9e956a7eea13
...with
9d609af54c63e7f949a4fbc43d4f1c13b57f49d8
being the last identified bad commit, and
9d2f7c60d9aef9860c61c2756318ada68c80fddf
being the last good.
If I should guess now, it's probably one
of Dmitry's commits on the wear leveling
code.
Regards,
Michael
Am 23.11.18 um 17:20 schrieb Michael
Balzer:
It's not
a timing issue, I've let it reboot about
30 times without any successful mount
after the first failure.
Going into bisecting now...
Am 23.11.18 um 15:54 schrieb Mark
Webb-Johnson:
It
may actually not be a corruption of
the filesystem but some timing issue
on the mount procedure. To test that
we could disable the auto formatting
on
mount failures.
True.
A couple of Espressif guys have jumped
on the issue, and I have provided some
more information for them. I think key
will be reproducing it.
The
issue may also be dependant on the
hardware version, i.e. it could be
caused by the bug that caused the SD
speed issue on the first 3.1 batch.
That was definitely a hardware issue
with the CP2102 chip. I don't think
related to ESP in any way.
Regards, Mark.
On 23
Nov 2018, at 10:34 PM, Michael
Balzer <dexter@expeedo.de
<mailto:dexter@expeedo.de>>
wrote:
It may actually not be a corruption
of the filesystem but some timing
issue on the mount procedure. To
test that we could disable the auto
formatting on
mount failures.
The issue may also be dependant on
the hardware version, i.e. it could
be caused by the bug that caused the
SD speed issue on the first 3.1
batch.
I only have tried the idf update on
my batch 1 module (my bench /
development module). I think most of
our edge testers also have that
version.
Regards,
Michael
Am 23.11.18 um 02:32 schrieb Mark
Webb-Johnson:
I
have raised the following github
issue to Espressif:
https://github.com/espressif/esp-idf/issues/2730
Environment
* Development Kit: none
* Kit version (for
WroverKit/PicoKit/DevKitC): none
* Module or chip used:
ESP32-WROVER 16MB
* IDF version (run |git
describe --tags| to find it):
v3.2-beta1-208-g0d7f2d77c
* Build System: make
* Compiler version (run
|xtensa-esp32-elf-gcc --version|
to find it): (crosstool-NG
crosstool-ng-1.22.0-80-g6c4433a)
5.2.0
* Operating System: macOS
* Power Supply: USB
Problem Description
TLDR: Between May and July 2018
a change was made to esp idf
master that is causing corruption
on FAT filesystems mounted on SPI
flash.
Our project uses a
partitions.csv as follows:
|# Name, Type, SubType, Offset,
Size nvs, data, nvs, 0x9000,
0x4000 otadata, data, ota, 0xd000,
0x2000 phy_init, data, phy,
0xf000, 0x1000 factory,
app, factory, 0x10000, 4M
ota_0, app, ota_0, , 4M ota_1,
app, ota_1, , 4M store, data, fat,
, 1M |
The 'store' partition is
formatted as FAT, as follows:
esp_vfs_fat_mount_config_t
m_store_fat;
wl_handle_t m_store_wlh;
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));
m_store_fat.format_if_mount_failed = true;
m_store_fat.max_files = 5;
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);
We have previously used a clone
of esp idf master, dated around
May 22 2018, without issues. The
partition is very reliable.
However, on Jul 6 2018, we
updated our clone to use the
latest esp idf master at that
time. Shortly afterwards, users
started to report that their
'store' filesystem contents
were corrupted. We rolled back.
We have now tried again
(updating on Oct 20 2018 to
v3.2-beta1-208-g0d7f2d77c) and
immediately had the same issue.
Random corruption of FAT
filesystem
in SPI flash.
Expected Behavior
No corruption of FAT
filesystem.
Actual Behavior
Corruption of FAT filesystem.
Steps to reproduce
1. Create a partition in SPI
flash, and mount FAT filesystem
2. Read and write to files on
FAT filesystem
3. Reboot
4. Observe random corruption
and unmountable filesystem
Code to reproduce this
issue
esp_vfs_fat_mount_config_t
m_store_fat;
wl_handle_t m_store_wlh;
memset(&m_store_fat,0,sizeof(esp_vfs_fat_sdmmc_mount_config_t));
m_store_fat.format_if_mount_failed = true;
m_store_fat.max_files = 5;
esp_vfs_fat_spiflash_mount("/store", "store", &m_store_fat,
&m_store_wlh);
Debug Logs
n/a
Other items if possible
Please advise if you need
anything further.
I think the timeline is correct
(the issue is in esp idf master
some time between May and July
2018), but please let me know if
you know differently (or
update the github issue with your
comments).
Regards, Mark
On
23 Nov 2018, at 6:19 AM, Michael
Balzer <dexter@expeedo.de
<mailto:dexter@expeedo.de>>
wrote:
esp-idf and OVMS branches are
back to the working version.
In case you also lost your
config: I also just fixed a bug
on restoring into an empty
/store partition.
Regards,
Michael
Am 22.11.18 um 22:34 schrieb
Michael Balzer:
See
https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/pull/165
I'll reset both master
branches now.
If you're about to pull,
please wait until I've
reverted the branches.
Regards,
Michael
--
Michael Balzer * Helkenberger
Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy
0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
<mailto:OvmsDev@lists.openvehicles.com>
http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9
* D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 /
206 989 26
_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
<mailto:OvmsDev@lists.openvehicles.com>
http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 *
D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206
989 26
_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 *
D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206
989 26
--
Steve_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
dmitry1945 commented 6 hours ago