First test result: I've performed the full upgrade&migrate procedure on my test module, starting from the old partition scheme and running from ota, so with the initial copy to factory.

It all worked without any issue.

Migrate copied 94 files in 22 directories with a total of 620 kB.

OVMS# vfs df 
/store:  Size:    976 kB  Avail:    356 kB  Used:  64%

→ migrate →

OVMS# vfs df
/store:  Size:   1936 kB  Avail:   1316 kB  Used:  33%

OVMS# ota
Hardware:          OVMS WIFI BLE BT cores=2 rev=ESP32/3; MODEM SIM7600
Firmware:          3.3.005-826-gb7d9ad0d9/ota_0/edge (build idf v3.3.4-854-g9063c8662c Apr 17 2026 13:10:34)
Partition type:    v3-35 (ota1, ota2, no factory, maximized store)
Partition table:   0x8000
Running partition: ota_0
Boot partition:    ota_0
OTA_O image:       3.3.005-826-gb7d9ad0d9

OVMS# ota partitions list 
Partition table:
Label            Type Subtype         Address Size      
nvs              data nvs          0x00009000 16 KB
otadata          data ota          0x0000d000 8 KB
phy_init         data phy          0x0000f000 4 KB
ota_0            app  ota_0        0x00010000 7 MB
ota_1            app  ota_1        0x00710000 7 MB
store            data fat          0x00e10000 1984 KB
Digest:          520877a00f2b1f7506844246b950ab23 pass


I tried booting into main/3.3.005-3 after the full migration, which also still works flawlessly.

Btw, command `ota` is the same as `ota status nocheck`.

For others testing this: when mounting store2, `vfs df` will display nonsense, as it assumes mount 1 is the SD card, also file/path name completion doesn't know about `/store2`, you need to enter the path manually. Use `vfs rls /store2` to review the whole tree (be aware that will omit the `ovms_config` path on `/store`).

But I second removing the store2 mount option ASAP, the output of `ota partitions store migrate` should be sufficient to determine success/failure.


I'll review the code next.

And yes, we need to automate the upgrade, a normal user should only need to trigger the process, not perform the steps needed manually. We should even think about initiating the upgrade automatically once we gain some trust in it.

As part of the automation, I think we should disable the auto init on the intermediary reboots, so the system runs on a minimal setup when performing the critical steps.

Regarding remaining space, there are some code parts we can remove/disable. Not sure how much that gives us, I'll have a look.

Regards,
Michael

Am 17.04.26 um 10:30 schrieb Mark Webb-Johnson:

… and I forgot to mention you need to ensure that you are running from factory before doing this. 

On 17 Apr 2026, at 4:27 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:

This has been a nightmare. Then a digression migrating servers. Then more of a nightmare.

Firstly, please don’t do these commands on production modules at the moment. Things may change.

Secondly, only do these commands on boxes with standard partition tables and flash sizes.

The sequence that currently works is as follows:

1. ota partitions store upgrade
2. module reset (aka ‘reboot’)
3. sd unmount
4. ota partitions store migrate
5. ota partitions upgrade factory
6. module reset (aka ‘reboot’)

You can use ’ota status nocheck' and ‘ota partitions list’ to see status along the way.

This is my fourth re-write of this, and the best I can get it at the moment. It needs another set of eyes looking over it, because the process is fraught with danger along the way.

Some comments (all learned the hard way):

• After making the new store2 partition, it can’t be mounted before reboot (I guess that the esp idf caches the partition table at boot).
• Our esp idf components/fatfs/src/ffconf.h has #defined FF_VOLUMES 2 which means we can’t mount more than two FAT filesystems. /store and /store2 for migration, but what about SD CARD? We can increase that (our esp idf), but even then a lot of our code assumes 0: is /store and 1: is /sd, and things got messy when I tried.
• The recursive copy /store to /store2 without using a lot of stack was tricky, but seems to be ok and as reliable as I could make it. I keep a list of directories to copy, to avoid direct subroutine call recursion.
• There is a ’ota partitions store downgrade’ to go back before step 5.
• We’ll probably want to move ‘ota partitions store mount/unmount’ out before production, as it opens us up to the user passwords being accessible - or at least add those /store2 paths to the protected paths list.
• The good news is I learnt a lot about the internals of ESP IDF, fatfs, partitions, et al.

At the moment this is ugly and not idiot proof. We are also down to about 60KB of free flash space (4MB limit).

Please have a play with it. Happy to accept suggestions as to how to get this usable and production ready.

OVMS# vfs df

/store:  Size:   1936 kB  Avail:   1804 kB  Used:   7%

/sd   :  Size:   3802 MB  Avail:   3787 MB  Used:   1%

OVMS# ota partitions list 

Partition table:

Label            Type Subtype         Address Size      

nvs              data nvs          0x00009000 16 KB

otadata          data ota          0x0000d000 8 KB

phy_init         data phy          0x0000f000 4 KB

ota_0            app  ota_0        0x00010000 7 MB

ota_1            app  ota_1        0x00710000 7 MB

store            data fat          0x00e10000 1984 KB

Digest:          520877a00f2b1f7506844246b950ab23 pass

Regards, Mark.

On 26 Feb 2026, at 8:47 AM, Mark Webb-Johnson via OvmsDev <ovmsdev@lists.openvehicles.com> wrote:

Michael,

I think you are correct. The safest way of doing this is to split as you suggest:

For #1, I think it is done now and should be ok.

For #2, I will work on that. I can provide both an upgrade and a downgrade option. My only concern is renaming the existing store to store-old (or something like that) while it is mounted. But that will be necessary to ensure that the new store is accessible directly after reboot.

Then #3 will need just a small tweak on the partition sizes to complete it.

Trying to do it all in one command may be too complex.

I will firstly update the typedef enum ovms_flashpartition_t so that you/we can know what state we are in. Suggest to just change it to unknown, v30 (existing arrangement), v34 (store resized and moved), v35 (flash partitions resized and factory removed).

I think I can provide a rollback if store partition is not as expected following reboot. I suggest:

ota partition upgrade store

ota partition downgrade store

ota partition upgrade factory

I’m not sure how long #2 will take me. Perhaps over the weekend, depending on the success of the recursive filesystem copy.

In the meantime, I suggest people don’t play with this too much as I’m only going to handle v30<->v34->v35 formats. Anything else will have to be manually done with esptools.

Regards, Mark.

On 26 Feb 2026, at 2:07 AM, Michael Balzer via OvmsDev <ovmsdev@lists.openvehicles.com> wrote:

Signed PGP part

Thinking about the web UI:

I intend to add a fourth tab "Partitioning" to Config→Firmware that checks the current partitioning scheme and guides through the upgrade process, so this can be done with just a couple of clicks (+ reboots).

The UI will provide info & links on each step, what to do if something fails.

So the process will now become this?

1. Copy running ota image to "factory", reboot into "factory"
2. Upgrade store partition, reboot
3. Upgrade OTA partitions, reboot

If step 2 fails, can we provide a fallback, i.e. re-enabling the old store partition, so the user can retry after another reboot?

Regards,
Michael

Am 25.02.26 um 17:18 schrieb Michael Balzer via OvmsDev:

Mark,

I don't think the recursive copy operation is an issue. We already do a similar, but much more memory hungry operation with the config backup & restore operations (zip/unzip).

The config backup doesn't add all directories, but includes "usr" meant for anything not fitting our standard /store layout:

  backup_dir[] =
  {
    { "ovms_config", false },
    { "events", true },
    { "scripts", true },
    { "obd2ecu", true },
    { "dbc", true },
    { "plugin", true },
    { "tls", true },
    { "trustedca", true },
    { "usr", true },
    { NULL, false }
  };

We could say that's also what we copy over to the new store, anything else is left to the user, if you think that's necessary.

But I doubt anyone has been using multi level custom directory structures on /store, due to the limited space and the frequent corruption issues we had with /store that would have turned out impractical very soon. Could be an option now, as I didn't hear about any corruption since introducing the config transaction scheme, but that's still new and only on edge, but I'd still recommend using /sd for that kind of storage.

Regards,
Michael

Am 25.02.26 um 14:13 schrieb Mark Webb-Johnson:

Craig, Michael,

I have fixed the (a) partition type in ‘ota’ after upgrade, and (b) copy to factory. Both commits pushed.

I also experimented with store.

• I can modify the partition table, rename old store to xstore, and create a new store at 0xe10000 same size 1MB, copy over the store partition. Reboot and all comes up fine (with repositioned store accessible).

• If I do the same, but set new store size to 0x1f0000, then the store can’t be mounted (corrupt, presumably). I guess the wear-leveling thing, or maybe internal fat filesystem metadata is messing things up.

So it seems that either we leave store at 1MB (move it to the end of flash and make the code partitions as big as we can), or we resize it to the 0x1f0000 and handle the complexities of first modifying the partition table for the new store, then mounting two fat file systems, recursively copying over the files, and then adjusting the partition tables again. The recursive copy worries me (memory consumption and if deeply nested then all those directory handles open at the same time - one for each level).

Thoughts?

Regards, Mark.

On Feb 25, 2026, at 3:59 PM, Michael Balzer <dexter@expeedo.de> wrote:

@Craig: nice find!

Am 25.02.26 um 02:58 schrieb Mark Webb-Johnson:

Michael,

AFAICT "ota partitions upgrade" currently does not verify the system has booted from "factory" before allowing the operation -- I think that should be added as a safety measure.

I thought it did that check. Maybe not working?

No, you're right, should be OK that way. But as Craig's result shows, I still should test it ;-)

Here are the three partition table options:

Original (f12):

Label	Type	Subtype	Offset (Hex)	Size (Hex)	Next (Hex)	Offset (Dec)	Size (Dec)
bootloader	bootloader	bootloader	0x0000	0x8000	8000	0	32768
partition_table	partition_table	partition_table	0x8000	0x1000	9000	32768	4096
nvs	data	nvs	0x9000	0x4000	D000	36864	16384
otadata	data	ota	0xD000	0x2000	F000	53248	8192
phy_init	data	phy	0xF000	0x1000	10000	61440	4096
factory	app	factory	0x10000	0x400000	410000	65536	4194304
ota_0	app	ota_0	0x410000	0x400000	810000	4259840	4194304
ota_1	app	ota_1	0x810000	0x400000	C10000	8454144	4194304
store	data	fat	0xC10000	0x100000	D10000	12648448	1048576
*unused*			0xD10000	0x2F0000	1000000	13697024	3080192

Current (12):

Label	Type	Subtype	Offset (Hex)	Size (Hex)	Next (Hex)	Offset (Dec)	Size (Dec)
bootloader	bootloader	bootloader	0x0000	0x8000	8000	0	32768
partition_table	partition_table	partition_table	0x8000	0x1000	9000	32768	4096
nvs	data	nvs	0x9000	0x4000	D000	36864	16384
otadata	data	ota	0xD000	0x2000	F000	53248	8192
phy_init	data	phy	0xF000	0x1000	10000	61440	4096
ota_0	app	ota_0	0x10000	0x600000	610000	65536	6291456
ota_1	app	ota_1	0x610000	0x600000	C10000	6356992	6291456
store	data	fat	0xC10000	0x100000	D10000	12648448	1048576
*unused*			0xD10000	0x2F0000	1000000	13697024	3080192

Suggested (12, 7MB code, more store):

Label	Type	Subtype	Offset (Hex)	Size (Hex)	Next (Hex)	Offset (Dec)	Size (Dec)
bootloader	bootloader	bootloader	0x0000	0x8000	8000	0	32768
partition_table	partition_table	partition_table	0x8000	0x1000	9000	32768	4096
nvs	data	nvs	0x9000	0x4000	D000	36864	16384
otadata	data	ota	0xD000	0x2000	F000	53248	8192
phy_init	data	phy	0xF000	0x1000	10000	61440	4096
ota_0	app	ota_0	0x10000	0x700000	710000	65536	7340032
ota_1	app	ota_1	0x710000	0x700000	E10000	7405568	7340032
store	data	fat	0xE10000	0x1F0000	1000000	14745600	2031616

It would make sense to resize store *now* if we are going to do it, but I am wary of requiring a factory reset. Also need to double check that the unused space at the end is truly unused. I am assuming the boot loader is in the first 32KB, but haven’t actually checked.

Looking at the third (resize store) arrangement, it would be moving 0xC10000-0xD0FFFF to 0xE10000-0xFFFFFF, which doesn’t seem to overlap. I wonder what simply copy that partition over (without reformatting flash) would do? Either corrupt the filesystem, ignore the extra space, or magically have the new space available to FAT? I suspect corruption. So, there is another (more complex) alternative:

1. Create a 2nd fat store partition (0xE10000-0xFFFFFF)
2. Format and mount that as FAT
3. Copy files from old to new
4. Unmount and drop the old store
5. Finish the rest of the partition upgrade

I suspect #3 would be the most complex step.

Or the store move and resize could be done first as a separate step.

Thoughts? Much more complex and risky...

There's the fatresize tool, didn't check the code, but given we can simply create the new partition cleanly and copy the files there, we should go that way.

I second the store resize should be the first upgrade step:

Unmounting /store while running may be risky, that was never needed and up to now only occurrs during shutdown as a final step.

I suggest we just relable the old "store" partition and do a reboot instead. esp_vfs_fat_spiflash_mount() searches for the partition by name, so the system should automatically mount & use the new "store" on boot.

Regards,
Michael

-- 
Michael Balzer * Am Rahmen 5 * D-58313 Herdecke
Fon 02330 9104094 * Handy 0176 20698926

-- 
Michael Balzer * Am Rahmen 5 * D-58313 Herdecke
Fon 02330 9104094 * Handy 0176 20698926

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- 
Michael Balzer * Am Rahmen 5 * D-58313 Herdecke
Fon 02330 9104094 * Handy 0176 20698926

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

_______________________________________________
OvmsDev mailing list
OvmsDev@lists.openvehicles.com
http://lists.openvehicles.com/mailman/listinfo/ovmsdev

_______________________________________________
OvmsDev mailing list -- ovmsdev@lists.openvehicles.com
To unsubscribe send an email to ovmsdev-leave@lists.openvehicles.com

_______________________________________________
OvmsDev mailing list -- ovmsdev@lists.openvehicles.com
To unsubscribe send an email to ovmsdev-leave@lists.openvehicles.com

-- 
Michael Balzer * Am Rahmen 5 * D-58313 Herdecke
Fon 02330 9104094 * Handy 0176 20698926

_______________________________________________
OvmsDev mailing list -- ovmsdev@lists.openvehicles.com
To unsubscribe send an email to ovmsdev-leave@lists.openvehicles.com

-- 
Michael Balzer * Am Rahmen 5 * D-58313 Herdecke
Fon 02330 9104094 * Handy 0176 20698926