There seems to be a memory leak, after running the module over the past hours in my car, ssh couldn't allocate 4K for a new session and I had this status: OVMS > mo m Free 8-bit *14600*/281968, 32-bit 8052/24308, SPIRAM 4129588/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM no task* 5348 0 0 0 +5348 +0 +0 +0 main* 18084 0 0 0 +18084 +0 +0 +0 esp_timer 39268 0 644 23436 +39268 +0 +644 +23436 OVMS Events 53304 65064 0 10076 +53304 +65064 +0 +10076 ipc0 7776 0 0 0 +7776 +0 +0 +0 ipc1 12 0 0 0 +12 +0 +0 +0 tiT 232 908 0 6376 +232 +908 +0 +6376 OVMS SIMCOM 0 4512 0 0 +0 +4512 +0 +0 wifi 1464 64 0 3512 +1464 +64 +0 +3512 OVMS Console 0 20 0 0 +0 +20 +0 +0 OVMS NetMan 0 12688 0 68 +0 +12688 +0 +68 mo m 0 0 15488 12000 +0 +0 +15488 +12000 OVMS > mo t Number of Tasks = 20 Stack: Now Max Total Heap 32-bit SPIRAM 3FFAFB48 1 Blk esp_timer 388 644 4096 39268 644 23436 3FFBDE54 2 Blk eventTask 448 1328 4608 0 0 0 3FFBFC50 3 Blk OVMS Events 444 3932 6144 118368 0 10076 3FFC4974 4 Blk OVMS CanRx 432 544 1024 0 0 0 3FFC9090 5 Blk ipc0 396 444 1024 7776 0 0 3FFC9690 6 Blk ipc1 396 444 1024 12 0 0 3FFCB4B8 9 Rdy IDLE 356 484 1024 0 0 0 3FFCBA4C 10 Rdy IDLE 360 488 1024 0 0 0 3FFCC7E0 11 Blk Tmr Svc 396 1820 3072 0 0 0 3FFC98FC 16 Blk tiT 496 2272 3072 1136 0 6576 3FFD7540 17 Blk OVMS SIMCOM 460 3020 4096 4512 0 0 3FFD951C 18 Blk wifi 424 2168 4096 1528 0 3512 3FFDBCB4 19 Blk pmT 416 576 2560 0 0 0 3FFDDABC 20 Blk OVMS Vehicle 456 2008 3072 0 0 0 3FFDFE38 21 Blk OVMS COrx 452 580 3072 0 0 0 3FFE1C88 22 Blk OVMS COwrk 532 532 1536 0 0 0 3FFE3E78 23 Blk OVMS Console 548 1716 6144 20 0 0 3FFE9430 24 Blk OVMS NetMan 732 5980 7168 12692 0 68 3FFEA9C8 25 Blk mdns 404 1724 4096 0 0 0 3FFFCD14 36 Rdy mo t 704 1936 5120 72 0 0 Nothing special to see. +14k on the events task vs. a fresh boot, but nothing that can explain the loss of more than 40k. Odd, I'll watch this. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Michael, I don't have a quick explanation for the memory leak, but some aspects of the sample output you show are curious. The prompt "OVMS >" predates Mark's change on 3/19 to remove the space but includes my change on 3/20 that put the asterisk right after the task name like "main*". It also includes Mark's insertion of the "OVMS" prefix on task names on 4/8 but not my fix to change the prompt to '#' for ssh logins (assuming this was an ssh console). Also I had never seen tasks like "mo m". Perhaps this was built in your local prop-cmdtask branch or something? In which case perhaps an update would fix the memory leak? -- Steve On Sat, 21 Apr 2018, Michael Balzer wrote:
There seems to be a memory leak, after running the module over the past hours in my car, ssh couldn't allocate 4K for a new session and I had this status:
OVMS > mo m Free 8-bit 14600/281968, 32-bit 8052/24308, SPIRAM 4129588/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM no task* 5348 0 0 0 +5348 +0 +0 +0 main* 18084 0 0 0 +18084 +0 +0 +0 esp_timer 39268 0 644 23436 +39268 +0 +644 +23436 OVMS Events 53304 65064 0 10076 +53304 +65064 +0 +10076 ipc0 7776 0 0 0 +7776 +0 +0 +0 ipc1 12 0 0 0 +12 +0 +0 +0 tiT 232 908 0 6376 +232 +908 +0 +6376 OVMS SIMCOM 0 4512 0 0 +0 +4512 +0 +0 wifi 1464 64 0 3512 +1464 +64 +0 +3512 OVMS Console 0 20 0 0 +0 +20 +0 +0 OVMS NetMan 0 12688 0 68 +0 +12688 +0 +68 mo m 0 0 15488 12000 +0 +0 +15488 +12000
OVMS > mo t Number of Tasks = 20 Stack: Now Max Total Heap 32-bit SPIRAM 3FFAFB48 1 Blk esp_timer 388 644 4096 39268 644 23436 3FFBDE54 2 Blk eventTask 448 1328 4608 0 0 0 3FFBFC50 3 Blk OVMS Events 444 3932 6144 118368 0 10076 3FFC4974 4 Blk OVMS CanRx 432 544 1024 0 0 0 3FFC9090 5 Blk ipc0 396 444 1024 7776 0 0 3FFC9690 6 Blk ipc1 396 444 1024 12 0 0 3FFCB4B8 9 Rdy IDLE 356 484 1024 0 0 0 3FFCBA4C 10 Rdy IDLE 360 488 1024 0 0 0 3FFCC7E0 11 Blk Tmr Svc 396 1820 3072 0 0 0 3FFC98FC 16 Blk tiT 496 2272 3072 1136 0 6576 3FFD7540 17 Blk OVMS SIMCOM 460 3020 4096 4512 0 0 3FFD951C 18 Blk wifi 424 2168 4096 1528 0 3512 3FFDBCB4 19 Blk pmT 416 576 2560 0 0 0 3FFDDABC 20 Blk OVMS Vehicle 456 2008 3072 0 0 0 3FFDFE38 21 Blk OVMS COrx 452 580 3072 0 0 0 3FFE1C88 22 Blk OVMS COwrk 532 532 1536 0 0 0 3FFE3E78 23 Blk OVMS Console 548 1716 6144 20 0 0 3FFE9430 24 Blk OVMS NetMan 732 5980 7168 12692 0 68 3FFEA9C8 25 Blk mdns 404 1724 4096 0 0 0 3FFFCD14 36 Rdy mo t 704 1936 5120 72 0 0
Nothing special to see. +14k on the events task vs. a fresh boot, but nothing that can explain the loss of more than 40k.
Odd, I'll watch this.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Steve, the prompt is copied from the web shell. That's also the source of the "mo m" task, which is the execution task of the command. I merged prop-cmdtask into master some weeks ago. I kept the branch for documentation, guess it can be removed now. Regards, Michael Am 21.04.2018 um 22:05 schrieb Stephen Casner:
Michael,
I don't have a quick explanation for the memory leak, but some aspects of the sample output you show are curious. The prompt "OVMS >" predates Mark's change on 3/19 to remove the space but includes my change on 3/20 that put the asterisk right after the task name like "main*". It also includes Mark's insertion of the "OVMS" prefix on task names on 4/8 but not my fix to change the prompt to '#' for ssh logins (assuming this was an ssh console). Also I had never seen tasks like "mo m". Perhaps this was built in your local prop-cmdtask branch or something? In which case perhaps an update would fix the memory leak?
-- Steve
On Sat, 21 Apr 2018, Michael Balzer wrote:
There seems to be a memory leak, after running the module over the past hours in my car, ssh couldn't allocate 4K for a new session and I had this status:
OVMS > mo m Free 8-bit 14600/281968, 32-bit 8052/24308, SPIRAM 4129588/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM no task* 5348 0 0 0 +5348 +0 +0 +0 main* 18084 0 0 0 +18084 +0 +0 +0 esp_timer 39268 0 644 23436 +39268 +0 +644 +23436 OVMS Events 53304 65064 0 10076 +53304 +65064 +0 +10076 ipc0 7776 0 0 0 +7776 +0 +0 +0 ipc1 12 0 0 0 +12 +0 +0 +0 tiT 232 908 0 6376 +232 +908 +0 +6376 OVMS SIMCOM 0 4512 0 0 +0 +4512 +0 +0 wifi 1464 64 0 3512 +1464 +64 +0 +3512 OVMS Console 0 20 0 0 +0 +20 +0 +0 OVMS NetMan 0 12688 0 68 +0 +12688 +0 +68 mo m 0 0 15488 12000 +0 +0 +15488 +12000
OVMS > mo t Number of Tasks = 20 Stack: Now Max Total Heap 32-bit SPIRAM 3FFAFB48 1 Blk esp_timer 388 644 4096 39268 644 23436 3FFBDE54 2 Blk eventTask 448 1328 4608 0 0 0 3FFBFC50 3 Blk OVMS Events 444 3932 6144 118368 0 10076 3FFC4974 4 Blk OVMS CanRx 432 544 1024 0 0 0 3FFC9090 5 Blk ipc0 396 444 1024 7776 0 0 3FFC9690 6 Blk ipc1 396 444 1024 12 0 0 3FFCB4B8 9 Rdy IDLE 356 484 1024 0 0 0 3FFCBA4C 10 Rdy IDLE 360 488 1024 0 0 0 3FFCC7E0 11 Blk Tmr Svc 396 1820 3072 0 0 0 3FFC98FC 16 Blk tiT 496 2272 3072 1136 0 6576 3FFD7540 17 Blk OVMS SIMCOM 460 3020 4096 4512 0 0 3FFD951C 18 Blk wifi 424 2168 4096 1528 0 3512 3FFDBCB4 19 Blk pmT 416 576 2560 0 0 0 3FFDDABC 20 Blk OVMS Vehicle 456 2008 3072 0 0 0 3FFDFE38 21 Blk OVMS COrx 452 580 3072 0 0 0 3FFE1C88 22 Blk OVMS COwrk 532 532 1536 0 0 0 3FFE3E78 23 Blk OVMS Console 548 1716 6144 20 0 0 3FFE9430 24 Blk OVMS NetMan 732 5980 7168 12692 0 68 3FFEA9C8 25 Blk mdns 404 1724 4096 0 0 0 3FFFCD14 36 Rdy mo t 704 1936 5120 72 0 0
Nothing special to see. +14k on the events task vs. a fresh boot, but nothing that can explain the loss of more than 40k.
Odd, I'll watch this.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
On Sat, 21 Apr 2018, Michael Balzer wrote:
the prompt is copied from the web shell. That's also the source of the "mo m" task, which is the execution task of the command.
Aha.
I merged prop-cmdtask into master some weeks ago. I kept the branch for documentation, guess it can be removed now.
No worries.
Nothing special to see. +14k on the events task vs. a fresh boot, but nothing that can explain the loss of more than 40k.
Here is my tally based on your command captures: 281968 100.00% Total DRAM space 208816 74.06% Total DRAM allocated 118368 41.98% DRAM allocated by OVMS Events 14600 5.18% Free DRAM 58552 20.77% Overhead for 2927.6 blocks So OVMS Events is using 42% of the total DRAM, or more than half of what has been allocated. I wonder if some of that could move to SPIRAM? I think your more than 40K missing is not a leak, it is the heap overhead. With heap poisoning enabled to allow the diagnostics, the overhead on each block is 20 bytes. Without heap poisoning the overhead is one pointer, 4 bytes. The 58K of overhead corresponds to about 2900 allocated blocks. There could well be that many, because many of them are small. Try a "mo m *" command which will dump the details for individual blocks for all the tasks up to the buffer limit of 1000 blocks. Here are a coupl of sample sections of that output: + t=esp_timer s= 44 a=0x3ffca344 3F408AB4 3FFCA350 00000009 73626577 65767265 FEFE0072 FEFEFEFE 3FFCA384 + t=esp_timer s= 12 a=0x3ffca384 4011B808 00000000 3FFB4288 BAAD5678 3FFCA3B4 ABBA1234 3FFAFAB0 0000000C + t=esp_timer s= 12 a=0x3ffca3a4 3FFCA164 3FFCA164 3FFCA344 BAAD5678 3FFCA3DC ABBA1234 3FFAFAB0 00000014 + t=esp_timer s= 20 a=0x3ffca3c4 3F408F8C 3F405748 40127470 00000000 00000000 BAAD5678 3FFCA404 ABBA1234 + t=esp_timer s= 20 a=0x3ffca3ec 3F411C30 3F411C38 40125D30 00000000 00000000 BAAD5678 3FFCA42C ABBA1234 + t=esp_timer s= 20 a=0x3ffca414 3F411C44 3F411C4C 40125D30 00000000 00000000 BAAD5678 3FFCA448 ABBA1234 + t=esp_timer s= 8 a=0x3ffca43c 3FFCA480 3FFCA414 BAAD5678 3FFCA470 ABBA1234 3FFAFAB0 00000014 3F411C58 + t=esp_timer s= 20 a=0x3ffca458 3F411C58 3F411C60 40125D30 00000000 00000000 BAAD5678 3FFCA48C ABBA1234 + t=esp_timer s= 8 a=0x3ffca480 3FFCA4C4 3FFCA458 BAAD5678 3FFCA4B4 ABBA1234 3FFAFAB0 00000014 3F411C6C + t=esp_timer s= 20 a=0x3ffca49c 3F411C6C 3F402EC8 40125D30 00000000 00000000 BAAD5678 3FFCA4D0 ABBA1234 + t=OVMS Events s= 84 a=0x3ffcdd6c 3F4081F4 3F407000 3F441B9C 401276F8 3F465668 3FFCDD88 00000000 3FFD7E00 + t=OVMS Events s= 24 a=0x3ffcddd4 00000000 3FFCDE68 00000000 00000000 3F407000 3FFCDD6C BAAD5678 3FFCDE58 + t=OVMS Events s= 84 a=0x3ffcde00 3F4081F4 3F441B8C 3F441B9C 401276F8 3F465668 3FFCDE1C 00000000 3FFCDE00 + t=OVMS Events s= 24 a=0x3ffcde68 00000001 3FFCDD40 3FFCDDD4 3FFCDEFC 3F441B8C 3FFCDE00 BAAD5678 3FFCDEEC + t=OVMS Events s= 84 a=0x3ffcde94 3F4081F4 3F442398 3F441BAC 4012764C 3F465668 3FFCDEB0 00000000 400E7A00 + t=OVMS Events s= 24 a=0x3ffcdefc 00000000 3FFCDE68 00000000 00000000 3F442398 3FFCDE94 BAAD5678 3FFCDF50 + t=OVMS Events s= 36 a=0x3ffcdf28 3F445990 3F40AEEC 00000001 00000017 00000013 00000012 FFFFFFFF FFFFFFFF + t=OVMS Events s= 24 a=0x3ffcdf60 00000000 3FFD943C 3FFDC810 3FFDA8C0 3F40AEEC 3FFCDF28 BAAD5678 3FFCDFE4 + t=OVMS Events s= 84 a=0x3ffcdf8c 3F4081F4 3F40AEEC 3F441B9C 00000000 3F407BB8 3FFCDFA8 00000000 00000000 You see many of the blocks are 8, 12, 20, or 24 bytes, so the overhead doubles the space used. Some blocks are larger, so overall the overhead is about 21% according to my tally. The poisoning serves two purposes: it allows checks for buffer overruns (8 bytes) in addition to tracking of task ownership and size (8 bytes). It might be feasible to drop the former while still allowing the memory diagnostics to work. -- Steve
Am 21.04.2018 um 22:05 schrieb Stephen Casner:
Michael,
I don't have a quick explanation for the memory leak, but some aspects of the sample output you show are curious. The prompt "OVMS >" predates Mark's change on 3/19 to remove the space but includes my change on 3/20 that put the asterisk right after the task name like "main*". It also includes Mark's insertion of the "OVMS" prefix on task names on 4/8 but not my fix to change the prompt to '#' for ssh logins (assuming this was an ssh console). Also I had never seen tasks like "mo m". Perhaps this was built in your local prop-cmdtask branch or something? In which case perhaps an update would fix the memory leak?
-- Steve
On Sat, 21 Apr 2018, Michael Balzer wrote:
There seems to be a memory leak, after running the module over the past hours in my car, ssh couldn't allocate 4K for a new session and I had this status:
OVMS > mo m Free 8-bit 14600/281968, 32-bit 8052/24308, SPIRAM 4129588/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM no task* 5348 0 0 0 +5348 +0 +0 +0 main* 18084 0 0 0 +18084 +0 +0 +0 esp_timer 39268 0 644 23436 +39268 +0 +644 +23436 OVMS Events 53304 65064 0 10076 +53304 +65064 +0 +10076 ipc0 7776 0 0 0 +7776 +0 +0 +0 ipc1 12 0 0 0 +12 +0 +0 +0 tiT 232 908 0 6376 +232 +908 +0 +6376 OVMS SIMCOM 0 4512 0 0 +0 +4512 +0 +0 wifi 1464 64 0 3512 +1464 +64 +0 +3512 OVMS Console 0 20 0 0 +0 +20 +0 +0 OVMS NetMan 0 12688 0 68 +0 +12688 +0 +68 mo m 0 0 15488 12000 +0 +0 +15488 +12000
OVMS > mo t Number of Tasks = 20 Stack: Now Max Total Heap 32-bit SPIRAM 3FFAFB48 1 Blk esp_timer 388 644 4096 39268 644 23436 3FFBDE54 2 Blk eventTask 448 1328 4608 0 0 0 3FFBFC50 3 Blk OVMS Events 444 3932 6144 118368 0 10076 3FFC4974 4 Blk OVMS CanRx 432 544 1024 0 0 0 3FFC9090 5 Blk ipc0 396 444 1024 7776 0 0 3FFC9690 6 Blk ipc1 396 444 1024 12 0 0 3FFCB4B8 9 Rdy IDLE 356 484 1024 0 0 0 3FFCBA4C 10 Rdy IDLE 360 488 1024 0 0 0 3FFCC7E0 11 Blk Tmr Svc 396 1820 3072 0 0 0 3FFC98FC 16 Blk tiT 496 2272 3072 1136 0 6576 3FFD7540 17 Blk OVMS SIMCOM 460 3020 4096 4512 0 0 3FFD951C 18 Blk wifi 424 2168 4096 1528 0 3512 3FFDBCB4 19 Blk pmT 416 576 2560 0 0 0 3FFDDABC 20 Blk OVMS Vehicle 456 2008 3072 0 0 0 3FFDFE38 21 Blk OVMS COrx 452 580 3072 0 0 0 3FFE1C88 22 Blk OVMS COwrk 532 532 1536 0 0 0 3FFE3E78 23 Blk OVMS Console 548 1716 6144 20 0 0 3FFE9430 24 Blk OVMS NetMan 732 5980 7168 12692 0 68 3FFEA9C8 25 Blk mdns 404 1724 4096 0 0 0 3FFFCD14 36 Rdy mo t 704 1936 5120 72 0 0
Nothing special to see. +14k on the events task vs. a fresh boot, but nothing that can explain the loss of more than 40k.
Odd, I'll watch this.
Regards, Michael
I could reproduce the situation, it was caused by a combination of… (a) the modem being captured in an endless reconnect retry loop (b) the notifications, still being allocated from internal RAM, adding up while server.v2 waits for the connection I have pushed a solution for (b) and an attempt for (a) -- couldn't test that yet. The effect of (a) is the modem only answers "ERROR" to the +CGDATA="PPP" command. I have tried different commands to reset the internal state, but it was stuck, only a power cycle could solve this. I have added an automatic power cycle for this situation, but it didn't occur again yet. The solution for (b) is to move the notifications buffer to external RAM. As that went without any problems, I also moved the command and server communication buffers to external RAM, which also went without problems. In action: … (sending 15 battery data notifications) … I (645344) ovms-server-v2: Send MP-0 h70,1,RT-BAT-C,14,86400,2,1,4105,4105,4105,10,15,15,15,0 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4124384/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 12604 +0 +0 +0 +1768 … (receiving 15 acks) … I (648804) ovms-server-v2: Incoming Msg: MP-0 h70 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4125488/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 10836 +0 +0 +0 -1768 I have added a C++11 allocator template "ExtRamAllocator" for this, which can be used for all allocator-aware standard classes. "ovms_extram.h" provides a namespace "extram" for std type instances using the allocator. It now has "string" and "ostringstream", please add further types as needed. @Steve: I would like to keep the allocation/task ownership info for now, I think it's worth the overhead. Regards, Michael -- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
Michael, Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)? Regards, Mark
On 22 Apr 2018, at 10:28 PM, Michael Balzer <dexter@expeedo.de> wrote:
I could reproduce the situation, it was caused by a combination of…
(a) the modem being captured in an endless reconnect retry loop (b) the notifications, still being allocated from internal RAM, adding up while server.v2 waits for the connection
I have pushed a solution for (b) and an attempt for (a) -- couldn't test that yet.
The effect of (a) is the modem only answers "ERROR" to the +CGDATA="PPP" command. I have tried different commands to reset the internal state, but it was stuck, only a power cycle could solve this. I have added an automatic power cycle for this situation, but it didn't occur again yet.
The solution for (b) is to move the notifications buffer to external RAM. As that went without any problems, I also moved the command and server communication buffers to external RAM, which also went without problems.
In action:
… (sending 15 battery data notifications) … I (645344) ovms-server-v2: Send MP-0 h70,1,RT-BAT-C,14,86400,2,1,4105,4105,4105,10,15,15,15,0 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4124384/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 12604 +0 +0 +0 +1768 … (receiving 15 acks) … I (648804) ovms-server-v2: Incoming Msg: MP-0 h70 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4125488/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 10836 +0 +0 +0 -1768
I have added a C++11 allocator template "ExtRamAllocator" for this, which can be used for all allocator-aware standard classes. "ovms_extram.h" provides a namespace "extram" for std type instances using the allocator. It now has "string" and "ostringstream", please add further types as needed.
@Steve: I would like to keep the allocation/task ownership info for now, I think it's worth the overhead.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Mark, I separated it out of modularity concerns (ovms.h = tools, ovms_extram.h = apps) and thought it may also gain some compile speed when we add more std headers to _extram.h. Currently that speed gain is near null, because it's included now by _utils.h, but that include can possibly be replaced by a forward declaration of extram::string. But I don't object if you think it's better to have this in one place. Regards, Michael Am 23.04.2018 um 04:07 schrieb Mark Webb-Johnson:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
On 22 Apr 2018, at 10:28 PM, Michael Balzer <dexter@expeedo.de> wrote:
I could reproduce the situation, it was caused by a combination of…
(a) the modem being captured in an endless reconnect retry loop (b) the notifications, still being allocated from internal RAM, adding up while server.v2 waits for the connection
I have pushed a solution for (b) and an attempt for (a) -- couldn't test that yet.
The effect of (a) is the modem only answers "ERROR" to the +CGDATA="PPP" command. I have tried different commands to reset the internal state, but it was stuck, only a power cycle could solve this. I have added an automatic power cycle for this situation, but it didn't occur again yet.
The solution for (b) is to move the notifications buffer to external RAM. As that went without any problems, I also moved the command and server communication buffers to external RAM, which also went without problems.
In action:
… (sending 15 battery data notifications) … I (645344) ovms-server-v2: Send MP-0 h70,1,RT-BAT-C,14,86400,2,1,4105,4105,4105,10,15,15,15,0 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4124384/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 12604 +0 +0 +0 +1768 … (receiving 15 acks) … I (648804) ovms-server-v2: Incoming Msg: MP-0 h70 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4125488/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 10836 +0 +0 +0 -1768
I have added a C++11 allocator template "ExtRamAllocator" for this, which can be used for all allocator-aware standard classes. "ovms_extram.h" provides a namespace "extram" for std type instances using the allocator. It now has "string" and "ostringstream", please add further types as needed.
@Steve: I would like to keep the allocation/task ownership info for now, I think it's worth the overhead.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
I’d like to keep the core memory allocation stuff in ovms.h. It is used pretty much everywhere, and it is good to keep it easily available. I’ve committed the change, just now. Regards, Mark.
On 23 Apr 2018, at 2:19 PM, Michael Balzer <dexter@expeedo.de> wrote:
Mark,
I separated it out of modularity concerns (ovms.h = tools, ovms_extram.h = apps) and thought it may also gain some compile speed when we add more std headers to _extram.h.
Currently that speed gain is near null, because it's included now by _utils.h, but that include can possibly be replaced by a forward declaration of extram::string.
But I don't object if you think it's better to have this in one place.
Regards, Michael
Am 23.04.2018 um 04:07 schrieb Mark Webb-Johnson:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
On 22 Apr 2018, at 10:28 PM, Michael Balzer <dexter@expeedo.de> wrote:
I could reproduce the situation, it was caused by a combination of…
(a) the modem being captured in an endless reconnect retry loop (b) the notifications, still being allocated from internal RAM, adding up while server.v2 waits for the connection
I have pushed a solution for (b) and an attempt for (a) -- couldn't test that yet.
The effect of (a) is the modem only answers "ERROR" to the +CGDATA="PPP" command. I have tried different commands to reset the internal state, but it was stuck, only a power cycle could solve this. I have added an automatic power cycle for this situation, but it didn't occur again yet.
The solution for (b) is to move the notifications buffer to external RAM. As that went without any problems, I also moved the command and server communication buffers to external RAM, which also went without problems.
In action:
… (sending 15 battery data notifications) … I (645344) ovms-server-v2: Send MP-0 h70,1,RT-BAT-C,14,86400,2,1,4105,4105,4105,10,15,15,15,0 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4124384/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 12604 +0 +0 +0 +1768 … (receiving 15 acks) … I (648804) ovms-server-v2: Incoming Msg: MP-0 h70 OVMS# mo m Free 8-bit 57692/281968, 32-bit 8052/24308, SPIRAM 4125488/4194252 --Task-- Total DRAM D/IRAM IRAM SPIRAM +/- DRAM D/IRAM IRAM SPIRAM OVMS Events 53012 49124 0 10836 +0 +0 +0 -1768
I have added a C++11 allocator template "ExtRamAllocator" for this, which can be used for all allocator-aware standard classes. "ovms_extram.h" provides a namespace "extram" for std type instances using the allocator. It now has "string" and "ostringstream", please add further types as needed.
@Steve: I would like to keep the allocation/task ownership info for now, I think it's worth the overhead.
Regards, Michael
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up). Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does. If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM. I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else. For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not. Opinions? -- Steve On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
Steve, Because of things like this: https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492> I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup… Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc). Last time I tried it (earlier this year), the ESP-IDF just simply didn’t work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place. I know Espressif have been working on it, and perhaps they’ve fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use): https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4> IDF 3.0 has officially been released last night. Regards, Mark.
On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org> wrote:
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up).
Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does.
If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM.
I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else.
For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not.
Opinions?
-- Steve
On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Mark, OK, I'll change mongoose to explicitly allocate from SPIRAM when available. I won't be able to completely test it, though, since I only have v3.0. -- Steve On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Steve,
Because of things like this:
https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492>
I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup…
Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc).
Last time I tried it (earlier this year), the ESP-IDF just simply didn’t work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.
I know Espressif have been working on it, and perhaps they’ve fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):
https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4>
IDF 3.0 has officially been released last night.
Regards, Mark.
On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org> wrote:
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up).
Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does.
If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM.
I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else.
For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not.
Opinions?
-- Steve
On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Steve, I see: #ifndef MG_MALLOC #define MG_MALLOC malloc #endif #ifndef MG_CALLOC #define MG_CALLOC calloc #endif #ifndef MG_REALLOC #define MG_REALLOC realloc #endif #ifndef MG_FREE #define MG_FREE free #endif Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc. Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes? Regards, Mark.
On 25 Apr 2018, at 1:20 PM, Stephen Casner <casner@acm.org> wrote:
Mark,
OK, I'll change mongoose to explicitly allocate from SPIRAM when available. I won't be able to completely test it, though, since I only have v3.0.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Steve,
Because of things like this:
https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492>
I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup…
Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc).
Last time I tried it (earlier this year), the ESP-IDF just simply didn’t work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.
I know Espressif have been working on it, and perhaps they’ve fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):
https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4>
IDF 3.0 has officially been released last night.
Regards, Mark.
On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org> wrote:
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up).
Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does.
If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM.
I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else.
For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not.
Opinions?
-- Steve
On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Let me know if you want me to do this first. Should be relatively simple as ESP-IDF provides equivalent functions. Regards, Mark.
On 25 Apr 2018, at 1:30 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Steve,
I see:
#ifndef MG_MALLOC #define MG_MALLOC malloc #endif
#ifndef MG_CALLOC #define MG_CALLOC calloc #endif
#ifndef MG_REALLOC #define MG_REALLOC realloc #endif
#ifndef MG_FREE #define MG_FREE free #endif
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Regards, Mark.
On 25 Apr 2018, at 1:20 PM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
Mark,
OK, I'll change mongoose to explicitly allocate from SPIRAM when available. I won't be able to completely test it, though, since I only have v3.0.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Steve,
Because of things like this:
https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492> <https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492>>
I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup…
Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc).
Last time I tried it (earlier this year), the ESP-IDF just simply didn’t work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.
I know Espressif have been working on it, and perhaps they’ve fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):
https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4> <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4>>
IDF 3.0 has officially been released last night.
Regards, Mark.
On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up).
Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does.
If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM.
I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else.
For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not.
Opinions?
-- Steve
On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Mark,
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
I already did this, but ran into the problem that ovms.{h,cpp} is C++ code, whereas mongoose is C. I started down a path of making duplicate C code just for mongoose, but I was trying to make this change without modifying mongoose.{h,c}. I think the right approach would be to extract ExternalRamMalloc etc. into a separate pair of files ovms_malloc.{h,c} that are C and then include them from ovms.h. -- Steve On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Let me know if you want me to do this first. Should be relatively simple as ESP-IDF provides equivalent functions.
Regards, Mark.
On 25 Apr 2018, at 1:30 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Steve,
I see:
#ifndef MG_MALLOC #define MG_MALLOC malloc #endif
#ifndef MG_CALLOC #define MG_CALLOC calloc #endif
#ifndef MG_REALLOC #define MG_REALLOC realloc #endif
#ifndef MG_FREE #define MG_FREE free #endif
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Regards, Mark.
On 25 Apr 2018, at 1:20 PM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
Mark,
OK, I'll change mongoose to explicitly allocate from SPIRAM when available. I won't be able to completely test it, though, since I only have v3.0.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Steve,
Because of things like this:
https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492> <https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492>>
I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup…
Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc).
Last time I tried it (earlier this year), the ESP-IDF just simply didn’t work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.
I know Espressif have been working on it, and perhaps they’ve fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):
https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4> <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4>>
IDF 3.0 has officially been released last night.
Regards, Mark.
On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up).
Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does.
If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM.
I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else.
For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not.
Opinions?
-- Steve
On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
ok
On 25 Apr 2018, at 2:34 PM, Stephen Casner <casner@acm.org> wrote:
Mark,
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
I already did this, but ran into the problem that ovms.{h,cpp} is C++ code, whereas mongoose is C. I started down a path of making duplicate C code just for mongoose, but I was trying to make this change without modifying mongoose.{h,c}. I think the right approach would be to extract ExternalRamMalloc etc. into a separate pair of files ovms_malloc.{h,c} that are C and then include them from ovms.h.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Let me know if you want me to do this first. Should be relatively simple as ESP-IDF provides equivalent functions.
Regards, Mark.
On 25 Apr 2018, at 1:30 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Steve,
I see:
#ifndef MG_MALLOC #define MG_MALLOC malloc #endif
#ifndef MG_CALLOC #define MG_CALLOC calloc #endif
#ifndef MG_REALLOC #define MG_REALLOC realloc #endif
#ifndef MG_FREE #define MG_FREE free #endif
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Regards, Mark.
On 25 Apr 2018, at 1:20 PM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
Mark,
OK, I'll change mongoose to explicitly allocate from SPIRAM when available. I won't be able to completely test it, though, since I only have v3.0.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Steve,
Because of things like this:
https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492> <https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492>>
I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup…
Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc).
Last time I tried it (earlier this year), the ESP-IDF just simply didn’t work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.
I know Espressif have been working on it, and perhaps they’ve fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):
https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4> <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4>>
IDF 3.0 has officially been released last night.
Regards, Mark.
On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up).
Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does.
If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM.
I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else.
For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not.
Opinions?
-- Steve
On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
> Michael, > > Do we need ovms_extram.h? Can we just put the namespace directly > into ovms.h (where all the other external ram allocation stuff is)? > > Regards, Mark _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Done, but now I get a corrupt heap abort. That does not make sense because on my v3.0 hardware the new functions should boil down to vanilla malloc, calloc, realloc. More tomorrow when I'm more awake. -- Steve On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
ok
On 25 Apr 2018, at 2:34 PM, Stephen Casner <casner@acm.org> wrote:
Mark,
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
I already did this, but ran into the problem that ovms.{h,cpp} is C++ code, whereas mongoose is C. I started down a path of making duplicate C code just for mongoose, but I was trying to make this change without modifying mongoose.{h,c}. I think the right approach would be to extract ExternalRamMalloc etc. into a separate pair of files ovms_malloc.{h,c} that are C and then include them from ovms.h.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Let me know if you want me to do this first. Should be relatively simple as ESP-IDF provides equivalent functions.
Regards, Mark.
On 25 Apr 2018, at 1:30 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Steve,
I see:
#ifndef MG_MALLOC #define MG_MALLOC malloc #endif
#ifndef MG_CALLOC #define MG_CALLOC calloc #endif
#ifndef MG_REALLOC #define MG_REALLOC realloc #endif
#ifndef MG_FREE #define MG_FREE free #endif
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Regards, Mark.
On 25 Apr 2018, at 1:20 PM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
Mark,
OK, I'll change mongoose to explicitly allocate from SPIRAM when available. I won't be able to completely test it, though, since I only have v3.0.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Steve,
Because of things like this:
https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492> <https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492>>
I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup...
Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc).
Last time I tried it (earlier this year), the ESP-IDF just simply didn't work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.
I know Espressif have been working on it, and perhaps they've fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):
https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4> <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4>>
IDF 3.0 has officially been released last night.
Regards, Mark.
> On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote: > > We've been dancing around the external RAM allocation decision with > our own ExtenalRamMalloc() function and extensions to the C++ new > allocator to try SPIRAM first but then back off to internal memory if > that fails (presumably because of v3.0 hardware rather than because > all 4MB of SPIRAM is used up). > > Do we need all that? There is already a mechanism in the multi-heap > memory system for the defaut malloc to try allocating from SPIRAM for > any request with size larger than malloc_alwaysinternal_limit which > can be set dynamically with heap_caps_malloc_extmem_enable(). If the > allocation from SPIRAM fails (again, presumably only on v3.0 hardware) > then the allocation is retried without the MALLOC_CAP_SPIRAM > capability, which is just what our ExternalRamMalloc does. > > If there are only a few allocations such as stacks that must come from > internal memory, and if those allocations (in system code) already use > heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then > we could probably just set malloc_alwaysinternal_limit to a small > number (perhaps 0) so that everything else comes from SPIRAM. > > I don't know if any of our application memory uses are sufficiently > sensitive to memory performance that we would want to ensure that the > memory is internal. My guess is that those would be few and therefore > it makes sense to take special action for them rather than for > everything else. > > For continued support of v3.0 hardware we could even test whether > SPIRAM is present and leave malloc_alwaysinternal_limit at its default > value of -1 if not. > > Opinions? > > -- Steve > > On Mon, 23 Apr 2018, Mark Webb-Johnson wrote: > >> Michael, >> >> Do we need ovms_extram.h? Can we just put the namespace directly >> into ovms.h (where all the other external ram allocation stuff is)? >> >> Regards, Mark > _______________________________________________ > OvmsDev mailing list > OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> > http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
I’m seeing this also, when I stress-test by sending in a large canbus dump into retools: CORRUPT HEAP: Bad head at 0x3f80e3c0. Expected 0xabba1234 got 0x3f80eb00 assertion "head != NULL" failed: file "/Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c", line 229, function: multi_heap_realloc abort() was called at PC 0x401015e7 on core 1 0x401015e7: __assert_func at /home/jeroen/esp8266/esp32/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:63 (discriminator 8) Backtrace: 0x40092cec:0x3ffe50f0 0x40092ee7:0x3ffe5110 0x401015e7:0x3ffe5130 0x4009294e:0x3ffe5160 0x40084e59:0x3ffe5180 0x400e4a51:0x3ffe51a0 0x400fb78a:0x3ffe51c0 0x400fb7da:0x3ffe51e0 0x400fb7f0:0x3ffe5200 0x400fe51a:0x3ffe5220 0x4013abd9:0x3ffe5260 0x4013c29e:0x3ffe52b0 0x4013c2b9:0x3ffe52f0 0x40092cec: invoke_abort at /Users/mark/esp/esp-idf/components/esp32/./panic.c:669 0x40092ee7: abort at /Users/mark/esp/esp-idf/components/esp32/./panic.c:669 0x401015e7: __assert_func at /home/jeroen/esp8266/esp32/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:63 (discriminator 8) 0x4009294e: multi_heap_realloc at /Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c:326 0x40084e59: heap_caps_realloc at /Users/mark/esp/esp-idf/components/heap/./heap_caps.c:388 0x400e4a51: ExternalRamRealloc at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/main/./ovms_malloc.c:65 0x400fb78a: mbuf_insert at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403 0x400fb7da: mbuf_append at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403 0x400fb7f0: mg_socket_if_tcp_send at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403 0x400fe51a: mg_send at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11403 0x4013abd9: re::DoServe(CAN_frame_t*) at /Users/mark/Documents/ovms/Open-Vehicle-Monitoring-System-3.1/vehicle/OVMS.V3/components/retools/src/retools.cpp:312 … That code path does a lot of realloc’s. If I switch back to use normal malloc/calloc/realloc, it still crashes. If I disable the malloc #defined in mongoose mg_locals.h, it still crashes. The only way I can avoid this is to consume ALL the data on the MG_EV_RECV event, which means realloc is presumably not called (so often?) within mongoose. I assumed the conclusion is that fundamentally realloc is broken in the ESP IDF version we use. I tried updating our ESP-IDF to espressif/master, but get compile errors in build/heap: CC build/heap/multi_heap_poisoning.o In file included from /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:27:0: /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:66:32: error: unknown type name 'TaskHandle_t' #define MULTI_HEAP_BLOCK_OWNER TaskHandle_t task; ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:50:5: note: in expansion of macro 'MULTI_HEAP_BLOCK_OWNER' MULTI_HEAP_BLOCK_OWNER ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'poison_allocated_region': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:67:57: error: implicit declaration of function 'xTaskGetCurrentTaskHandle' [-Werror=implicit-function-declaration] #define MULTI_HEAP_SET_BLOCK_OWNER(HEAD) (HEAD)->task = xTaskGetCurrentTaskHandle() ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:71:5: note: in expansion of macro 'MULTI_HEAP_SET_BLOCK_OWNER' MULTI_HEAP_SET_BLOCK_OWNER(head); ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'multi_heap_get_block_owner': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:68:42: warning: return makes pointer from integer without a cast [-Wint-conversion] #define MULTI_HEAP_GET_BLOCK_OWNER(HEAD) ((HEAD)->task) ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:284:12: note: in expansion of macro 'MULTI_HEAP_GET_BLOCK_OWNER' return MULTI_HEAP_GET_BLOCK_OWNER((poison_head_t*)multi_heap_get_block_address_impl(block)); ^ cc1: some warnings being treated as errors make[1]: *** [multi_heap_poisoning.o] Error 1 I then tried: +CONFIG_HEAP_POISONING_DISABLED=y +CONFIG_HEAP_POISONING_LIGHT= -CONFIG_HEAP_TASK_TRACKING=y But no difference. Still crashes. Also tried: -CONFIG_FREERTOS_USE_TRACE_FACILITY=y -CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS= +CONFIG_FREERTOS_USE_TRACE_FACILITY= But no difference. Still crashes. Although not in a simply way: OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4141392/4194252 OVMS# test realloc First check heap integrity... Now allocate 4KB RAM... Check heap integrity... Now re-allocate bigger, 1,000 times... Check heap integrity... Now re-allocate smaller, 1,000 times... Check heap integrity... And free the buffer... Final check of heap integrity… OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4140892/4194252 Maybe something inside mongoose itself? As that is the only place we are seeing this. Regards, Mark.
On 25 Apr 2018, at 3:18 PM, Stephen Casner <casner@acm.org> wrote:
Done, but now I get a corrupt heap abort. That does not make sense because on my v3.0 hardware the new functions should boil down to vanilla malloc, calloc, realloc. More tomorrow when I'm more awake.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
ok
On 25 Apr 2018, at 2:34 PM, Stephen Casner <casner@acm.org> wrote:
Mark,
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
I already did this, but ran into the problem that ovms.{h,cpp} is C++ code, whereas mongoose is C. I started down a path of making duplicate C code just for mongoose, but I was trying to make this change without modifying mongoose.{h,c}. I think the right approach would be to extract ExternalRamMalloc etc. into a separate pair of files ovms_malloc.{h,c} that are C and then include them from ovms.h.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Let me know if you want me to do this first. Should be relatively simple as ESP-IDF provides equivalent functions.
Regards, Mark.
On 25 Apr 2018, at 1:30 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
Steve,
I see:
#ifndef MG_MALLOC #define MG_MALLOC malloc #endif
#ifndef MG_CALLOC #define MG_CALLOC calloc #endif
#ifndef MG_REALLOC #define MG_REALLOC realloc #endif
#ifndef MG_FREE #define MG_FREE free #endif
Presumably defining those in mg_locals.h? Then I guess ovms.{h,cpp} needs our ExternalRamMalloc extended with ExternalRamCalloc and ExternalRamRealloc.
Or shall we just make a ov_malloc, ov_calloc, and ov_realloc, to be easier on the eyes?
Regards, Mark.
On 25 Apr 2018, at 1:20 PM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
Mark,
OK, I'll change mongoose to explicitly allocate from SPIRAM when available. I won't be able to completely test it, though, since I only have v3.0.
-- Steve
On Wed, 25 Apr 2018, Mark Webb-Johnson wrote:
> Steve, > > Because of things like this: > > https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492> <https://github.com/espressif/esp-idf/issues/1492 <https://github.com/espressif/esp-idf/issues/1492>> > > I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup... > > Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc). > > Last time I tried it (earlier this year), the ESP-IDF just simply didn't work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place. > > I know Espressif have been working on it, and perhaps they've fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use): > > https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4> <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825... <https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a08252be720d4>> > > IDF 3.0 has officially been released last night. > > Regards, Mark. > >> On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote: >> >> We've been dancing around the external RAM allocation decision with >> our own ExtenalRamMalloc() function and extensions to the C++ new >> allocator to try SPIRAM first but then back off to internal memory if >> that fails (presumably because of v3.0 hardware rather than because >> all 4MB of SPIRAM is used up). >> >> Do we need all that? There is already a mechanism in the multi-heap >> memory system for the defaut malloc to try allocating from SPIRAM for >> any request with size larger than malloc_alwaysinternal_limit which >> can be set dynamically with heap_caps_malloc_extmem_enable(). If the >> allocation from SPIRAM fails (again, presumably only on v3.0 hardware) >> then the allocation is retried without the MALLOC_CAP_SPIRAM >> capability, which is just what our ExternalRamMalloc does. >> >> If there are only a few allocations such as stacks that must come from >> internal memory, and if those allocations (in system code) already use >> heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then >> we could probably just set malloc_alwaysinternal_limit to a small >> number (perhaps 0) so that everything else comes from SPIRAM. >> >> I don't know if any of our application memory uses are sufficiently >> sensitive to memory performance that we would want to ensure that the >> memory is internal. My guess is that those would be few and therefore >> it makes sense to take special action for them rather than for >> everything else. >> >> For continued support of v3.0 hardware we could even test whether >> SPIRAM is present and leave malloc_alwaysinternal_limit at its default >> value of -1 if not. >> >> Opinions? >> >> -- Steve >> >> On Mon, 23 Apr 2018, Mark Webb-Johnson wrote: >> >>> Michael, >>> >>> Do we need ovms_extram.h? Can we just put the namespace directly >>> into ovms.h (where all the other external ram allocation stuff is)? >>> >>> Regards, Mark >> _______________________________________________ >> OvmsDev mailing list >> OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> >> http://lists.openvehicles.com/mailman/listinfo/ovmsdev > _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
On Thu, 26 Apr 2018, Mark Webb-Johnson wrote:
I’m seeing this also, when I stress-test by sending in a large canbus dump into retools:
CORRUPT HEAP: Bad head at 0x3f80e3c0. Expected 0xabba1234 got 0x3f80eb00 assertion "head != NULL" failed: file "/Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c", line 229, function: multi_heap_realloc
Indeed, this is the same error signature as what I was seeing, but the reason I saw it was because of a mistake in my code. I had not included the "if (size == 0)" test in ExternalRamRealloc(). Since mongoose calls MG_REALLOC with size == 0, that caused heap_caps_realloc() to do heap_caps_free() and return NULL, so my code called realloc() on that same ptr that had just been freed. But before my commit this morning I fixed that bug by adding my own test for size==0 so I could do my own heap_caps_free() and not call calling heap_caps_realloc(). So, as I stare at the code now, I don't see anything yet. Also, your crash occurred when MG_REALLOC was called with nonzero size.
That code path does a lot of realloc’s.
If I switch back to use normal malloc/calloc/realloc, it still crashes.
If I disable the malloc #defined in mongoose mg_locals.h, it still crashes.
The only way I can avoid this is to consume ALL the data on the MG_EV_RECV event, which means realloc is presumably not called (so often?) within mongoose.
I assumed the conclusion is that fundamentally realloc is broken in the ESP IDF version we use.
So, by switching back to the old code you've also verified that it was not my new code. But why are you seeing this crash now when we have not seen it before? Is there any way that my moving ExternalRamMalloc() to a C namespace could cause trouble in some allocation outside of mongoose leaving a bad heap for the mongoose call to trip over?
I tried updating our ESP-IDF to espressif/master, but get compile errors in build/heap:
CC build/heap/multi_heap_poisoning.o In file included from /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:27:0: /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:66:32: error: unknown type name 'TaskHandle_t' #define MULTI_HEAP_BLOCK_OWNER TaskHandle_t task; ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:50:5: note: in expansion of macro 'MULTI_HEAP_BLOCK_OWNER' MULTI_HEAP_BLOCK_OWNER ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'poison_allocated_region': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:67:57: error: implicit declaration of function 'xTaskGetCurrentTaskHandle' [-Werror=implicit-function-declaration] #define MULTI_HEAP_SET_BLOCK_OWNER(HEAD) (HEAD)->task = xTaskGetCurrentTaskHandle() ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:71:5: note: in expansion of macro 'MULTI_HEAP_SET_BLOCK_OWNER' MULTI_HEAP_SET_BLOCK_OWNER(head); ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'multi_heap_get_block_owner': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:68:42: warning: return makes pointer from integer without a cast [-Wint-conversion] #define MULTI_HEAP_GET_BLOCK_OWNER(HEAD) ((HEAD)->task) ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:284:12: note: in expansion of macro 'MULTI_HEAP_GET_BLOCK_OWNER' return MULTI_HEAP_GET_BLOCK_OWNER((poison_head_t*)multi_heap_get_block_address_impl(block)); ^ cc1: some warnings being treated as errors make[1]: *** [multi_heap_poisoning.o] Error 1
Those three macros are part of the code I provided in the pull request, but in 4c532a59b2503dd8bc60d437bb65961676db9e55 we already updated to a version of expressif/esp-idf master with the pull request included. It seems that the references in the macros are to other code in ESP-IDF that used to be valid but now may not be. Perhaps they made a change but didn't test compilation with CONFIG_HEAP_TASK_TRACKING=y?
I then tried:
+CONFIG_HEAP_POISONING_DISABLED=y +CONFIG_HEAP_POISONING_LIGHT= -CONFIG_HEAP_TASK_TRACKING=y
But no difference. Still crashes.
Do you mean that it then compiles OK, but when you run the stress test it crashes? If it compiles OK, then my hypothesis that they made a change and didn't test with CONFIG_HEAP_TASK_TRACKING=y might be right.
Also tried:
-CONFIG_FREERTOS_USE_TRACE_FACILITY=y -CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS= +CONFIG_FREERTOS_USE_TRACE_FACILITY=
But no difference. Still crashes.
Although not in a simply way:
OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4141392/4194252
OVMS# test realloc First check heap integrity... Now allocate 4KB RAM... Check heap integrity... Now re-allocate bigger, 1,000 times... Check heap integrity... Now re-allocate smaller, 1,000 times... Check heap integrity... And free the buffer... Final check of heap integrity…
OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4140892/4194252
Maybe something inside mongoose itself? As that is the only place we are seeing this.
You say it still crashes, but you don't show a crash here? -- Steve
But why are you seeing this crash now when we have not seen it before?
I haven’t really done much with that code path in the past. It is a high volume of data being uploaded to the module, and due to the way my code was handling it (badly), it resulted in a lot of realloc’s. For CRTD format, I read line-by-line. The issue is that the mongoose buffer presented may not have a complete line in it. Mongoose was sending me a MG_EV_RECV with about 1KB in the buffer. I removed about 30 bytes (one line), and returned. That meant mongoose realloc’d down to 1KB - 30 bytes, then called me again. Rinse and repeat. I first noticed this because SPI RAM memory consumption was going through the roof during retools canbus dump simulations. I was handling the incoming messages so slowly, that they were coming in faster than they were handled. Lesson learned: when mongoose tells you it has data, consume it otherwise mongoose will keep receiving (a sensible framework would back off the receiving if the consumer was not fast enough). My change now is to consume all I can on the MG_EV_RECV (in a while loop), but if there is half a line left over that gets left in the mongoose buffer and that causes a realloc. To get around this, I’ll have to do my own line buffering (so I can completely drain the mongoose buffer on MG_EV_RECV). A hassle, but I’ll try to get it done tonight. That will resolve my problem, but I don’t think it addresses the core mongoose heap corruption issue. My code is in retools.cpp re::MongooseHandler MG_EV_RECV part.
You say it still crashes, but you don't show a crash here?
I can get it to crash in retools. Repeatedly. Easy and every time. Start a wifi Start a vehicle mode, on can1 (maybe not necessary, but I do it anyway) OVMS# re start OVMS# re serve mode simulate $ cat my-canbus-dump.crtd | nc TESTBOX.local 3000 Sit back and watch the crash If you don’t have a CRTD file, just use this snippet: 593.138 1R11 7E8 00 00 00 00 00 00 00 00 593.138 1R11 202 4D 5F 07 9D C4 09 3F 23 593.138 1R11 20A 00 01 01 83 D1 3E 00 00 593.138 1R11 228 40 C0 30 F8 593.138 1R11 6F2 1F 18 08 F9 A1 80 08 1F 593.138 1R11 212 D8 09 12 71 80 Make 1,000 copies of it, appending them all to one .crtd file. That can be your test data. So, I then wrote a ‘test realloc’ in test_framework.cpp to see if I could repeat it. I couldn’t. Which leads me to suspect the issue is in mongoose (or with the way I am only consuming part of the mongoose buffer). Regards, Mark.
On 26 Apr 2018, at 2:16 PM, Stephen Casner <casner@acm.org> wrote:
On Thu, 26 Apr 2018, Mark Webb-Johnson wrote:
I’m seeing this also, when I stress-test by sending in a large canbus dump into retools:
CORRUPT HEAP: Bad head at 0x3f80e3c0. Expected 0xabba1234 got 0x3f80eb00 assertion "head != NULL" failed: file "/Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c", line 229, function: multi_heap_realloc
Indeed, this is the same error signature as what I was seeing, but the reason I saw it was because of a mistake in my code. I had not included the "if (size == 0)" test in ExternalRamRealloc(). Since mongoose calls MG_REALLOC with size == 0, that caused heap_caps_realloc() to do heap_caps_free() and return NULL, so my code called realloc() on that same ptr that had just been freed.
But before my commit this morning I fixed that bug by adding my own test for size==0 so I could do my own heap_caps_free() and not call calling heap_caps_realloc().
So, as I stare at the code now, I don't see anything yet. Also, your crash occurred when MG_REALLOC was called with nonzero size.
That code path does a lot of realloc’s.
If I switch back to use normal malloc/calloc/realloc, it still crashes.
If I disable the malloc #defined in mongoose mg_locals.h, it still crashes.
The only way I can avoid this is to consume ALL the data on the MG_EV_RECV event, which means realloc is presumably not called (so often?) within mongoose.
I assumed the conclusion is that fundamentally realloc is broken in the ESP IDF version we use.
So, by switching back to the old code you've also verified that it was not my new code. But why are you seeing this crash now when we have not seen it before? Is there any way that my moving ExternalRamMalloc() to a C namespace could cause trouble in some allocation outside of mongoose leaving a bad heap for the mongoose call to trip over?
I tried updating our ESP-IDF to espressif/master, but get compile errors in build/heap:
CC build/heap/multi_heap_poisoning.o In file included from /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:27:0: /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:66:32: error: unknown type name 'TaskHandle_t' #define MULTI_HEAP_BLOCK_OWNER TaskHandle_t task; ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:50:5: note: in expansion of macro 'MULTI_HEAP_BLOCK_OWNER' MULTI_HEAP_BLOCK_OWNER ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'poison_allocated_region': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:67:57: error: implicit declaration of function 'xTaskGetCurrentTaskHandle' [-Werror=implicit-function-declaration] #define MULTI_HEAP_SET_BLOCK_OWNER(HEAD) (HEAD)->task = xTaskGetCurrentTaskHandle() ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:71:5: note: in expansion of macro 'MULTI_HEAP_SET_BLOCK_OWNER' MULTI_HEAP_SET_BLOCK_OWNER(head); ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'multi_heap_get_block_owner': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:68:42: warning: return makes pointer from integer without a cast [-Wint-conversion] #define MULTI_HEAP_GET_BLOCK_OWNER(HEAD) ((HEAD)->task) ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:284:12: note: in expansion of macro 'MULTI_HEAP_GET_BLOCK_OWNER' return MULTI_HEAP_GET_BLOCK_OWNER((poison_head_t*)multi_heap_get_block_address_impl(block)); ^ cc1: some warnings being treated as errors make[1]: *** [multi_heap_poisoning.o] Error 1
Those three macros are part of the code I provided in the pull request, but in 4c532a59b2503dd8bc60d437bb65961676db9e55 we already updated to a version of expressif/esp-idf master with the pull request included. It seems that the references in the macros are to other code in ESP-IDF that used to be valid but now may not be. Perhaps they made a change but didn't test compilation with CONFIG_HEAP_TASK_TRACKING=y?
I then tried:
+CONFIG_HEAP_POISONING_DISABLED=y +CONFIG_HEAP_POISONING_LIGHT= -CONFIG_HEAP_TASK_TRACKING=y
But no difference. Still crashes.
Do you mean that it then compiles OK, but when you run the stress test it crashes? If it compiles OK, then my hypothesis that they made a change and didn't test with CONFIG_HEAP_TASK_TRACKING=y might be right.
Also tried:
-CONFIG_FREERTOS_USE_TRACE_FACILITY=y -CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS= +CONFIG_FREERTOS_USE_TRACE_FACILITY=
But no difference. Still crashes.
Although not in a simply way:
OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4141392/4194252
OVMS# test realloc First check heap integrity... Now allocate 4KB RAM... Check heap integrity... Now re-allocate bigger, 1,000 times... Check heap integrity... Now re-allocate smaller, 1,000 times... Check heap integrity... And free the buffer... Final check of heap integrity…
OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4140892/4194252
Maybe something inside mongoose itself? As that is the only place we are seeing this.
You say it still crashes, but you don't show a crash here?
-- Steve_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
The code I’ve just committed re-factors the can dump system, and retools, to always consume all the data from MG_EV_RECV. That reduces memory consumption, and works around the realloc issue. It seems to work in my limited testing here, and performance is acceptable. I can transmit a 1MB CRTD file in about 10 seconds. About 25,000 CAN bus messages. The CRTB format (binary) will require less encoding/decoding and should be faster still. Regards, Mark.
On 26 Apr 2018, at 5:03 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
But why are you seeing this crash now when we have not seen it before?
I haven’t really done much with that code path in the past. It is a high volume of data being uploaded to the module, and due to the way my code was handling it (badly), it resulted in a lot of realloc’s.
For CRTD format, I read line-by-line. The issue is that the mongoose buffer presented may not have a complete line in it.
Mongoose was sending me a MG_EV_RECV with about 1KB in the buffer. I removed about 30 bytes (one line), and returned. That meant mongoose realloc’d down to 1KB - 30 bytes, then called me again. Rinse and repeat.
I first noticed this because SPI RAM memory consumption was going through the roof during retools canbus dump simulations. I was handling the incoming messages so slowly, that they were coming in faster than they were handled. Lesson learned: when mongoose tells you it has data, consume it otherwise mongoose will keep receiving (a sensible framework would back off the receiving if the consumer was not fast enough).
My change now is to consume all I can on the MG_EV_RECV (in a while loop), but if there is half a line left over that gets left in the mongoose buffer and that causes a realloc.
To get around this, I’ll have to do my own line buffering (so I can completely drain the mongoose buffer on MG_EV_RECV). A hassle, but I’ll try to get it done tonight. That will resolve my problem, but I don’t think it addresses the core mongoose heap corruption issue.
My code is in retools.cpp re::MongooseHandler MG_EV_RECV part.
You say it still crashes, but you don't show a crash here?
I can get it to crash in retools. Repeatedly. Easy and every time.
Start a wifi Start a vehicle mode, on can1 (maybe not necessary, but I do it anyway) OVMS# re start OVMS# re serve mode simulate $ cat my-canbus-dump.crtd | nc TESTBOX.local 3000 Sit back and watch the crash
If you don’t have a CRTD file, just use this snippet:
593.138 1R11 7E8 00 00 00 00 00 00 00 00 593.138 1R11 202 4D 5F 07 9D C4 09 3F 23 593.138 1R11 20A 00 01 01 83 D1 3E 00 00 593.138 1R11 228 40 C0 30 F8 593.138 1R11 6F2 1F 18 08 F9 A1 80 08 1F 593.138 1R11 212 D8 09 12 71 80
Make 1,000 copies of it, appending them all to one .crtd file. That can be your test data.
So, I then wrote a ‘test realloc’ in test_framework.cpp to see if I could repeat it. I couldn’t. Which leads me to suspect the issue is in mongoose (or with the way I am only consuming part of the mongoose buffer).
Regards, Mark.
On 26 Apr 2018, at 2:16 PM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
On Thu, 26 Apr 2018, Mark Webb-Johnson wrote:
I’m seeing this also, when I stress-test by sending in a large canbus dump into retools:
CORRUPT HEAP: Bad head at 0x3f80e3c0. Expected 0xabba1234 got 0x3f80eb00 assertion "head != NULL" failed: file "/Users/mark/esp/esp-idf/components/heap/./multi_heap_poisoning.c", line 229, function: multi_heap_realloc
Indeed, this is the same error signature as what I was seeing, but the reason I saw it was because of a mistake in my code. I had not included the "if (size == 0)" test in ExternalRamRealloc(). Since mongoose calls MG_REALLOC with size == 0, that caused heap_caps_realloc() to do heap_caps_free() and return NULL, so my code called realloc() on that same ptr that had just been freed.
But before my commit this morning I fixed that bug by adding my own test for size==0 so I could do my own heap_caps_free() and not call calling heap_caps_realloc().
So, as I stare at the code now, I don't see anything yet. Also, your crash occurred when MG_REALLOC was called with nonzero size.
That code path does a lot of realloc’s.
If I switch back to use normal malloc/calloc/realloc, it still crashes.
If I disable the malloc #defined in mongoose mg_locals.h, it still crashes.
The only way I can avoid this is to consume ALL the data on the MG_EV_RECV event, which means realloc is presumably not called (so often?) within mongoose.
I assumed the conclusion is that fundamentally realloc is broken in the ESP IDF version we use.
So, by switching back to the old code you've also verified that it was not my new code. But why are you seeing this crash now when we have not seen it before? Is there any way that my moving ExternalRamMalloc() to a C namespace could cause trouble in some allocation outside of mongoose leaving a bad heap for the mongoose call to trip over?
I tried updating our ESP-IDF to espressif/master, but get compile errors in build/heap:
CC build/heap/multi_heap_poisoning.o In file included from /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:27:0: /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:66:32: error: unknown type name 'TaskHandle_t' #define MULTI_HEAP_BLOCK_OWNER TaskHandle_t task; ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:50:5: note: in expansion of macro 'MULTI_HEAP_BLOCK_OWNER' MULTI_HEAP_BLOCK_OWNER ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'poison_allocated_region': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:67:57: error: implicit declaration of function 'xTaskGetCurrentTaskHandle' [-Werror=implicit-function-declaration] #define MULTI_HEAP_SET_BLOCK_OWNER(HEAD) (HEAD)->task = xTaskGetCurrentTaskHandle() ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:71:5: note: in expansion of macro 'MULTI_HEAP_SET_BLOCK_OWNER' MULTI_HEAP_SET_BLOCK_OWNER(head); ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c: In function 'multi_heap_get_block_owner': /Users/mark/esp/esp-idf/components/heap/multi_heap_platform.h:68:42: warning: return makes pointer from integer without a cast [-Wint-conversion] #define MULTI_HEAP_GET_BLOCK_OWNER(HEAD) ((HEAD)->task) ^ /Users/mark/esp/esp-idf/components/heap/multi_heap_poisoning.c:284:12: note: in expansion of macro 'MULTI_HEAP_GET_BLOCK_OWNER' return MULTI_HEAP_GET_BLOCK_OWNER((poison_head_t*)multi_heap_get_block_address_impl(block)); ^ cc1: some warnings being treated as errors make[1]: *** [multi_heap_poisoning.o] Error 1
Those three macros are part of the code I provided in the pull request, but in 4c532a59b2503dd8bc60d437bb65961676db9e55 we already updated to a version of expressif/esp-idf master with the pull request included. It seems that the references in the macros are to other code in ESP-IDF that used to be valid but now may not be. Perhaps they made a change but didn't test compilation with CONFIG_HEAP_TASK_TRACKING=y?
I then tried:
+CONFIG_HEAP_POISONING_DISABLED=y +CONFIG_HEAP_POISONING_LIGHT= -CONFIG_HEAP_TASK_TRACKING=y
But no difference. Still crashes.
Do you mean that it then compiles OK, but when you run the stress test it crashes? If it compiles OK, then my hypothesis that they made a change and didn't test with CONFIG_HEAP_TASK_TRACKING=y might be right.
Also tried:
-CONFIG_FREERTOS_USE_TRACE_FACILITY=y -CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS= +CONFIG_FREERTOS_USE_TRACE_FACILITY=
But no difference. Still crashes.
Although not in a simply way:
OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4141392/4194252
OVMS# test realloc First check heap integrity... Now allocate 4KB RAM... Check heap integrity... Now re-allocate bigger, 1,000 times... Check heap integrity... Now re-allocate smaller, 1,000 times... Check heap integrity... And free the buffer... Final check of heap integrity…
OVMS# module memory Free 8-bit 119432/281952, 32-bit 8020/24276, SPIRAM 4140892/4194252
Maybe something inside mongoose itself? As that is the only place we are seeing this.
You say it still crashes, but you don't show a crash here?
-- Steve_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
On Thu, 26 Apr 2018, Mark Webb-Johnson wrote:
I tried updating our ESP-IDF to espressif/master, but get compile errors in build/heap:
I figured out what is wrong and just now filed this issue: https://github.com/espressif/esp-idf/issues/1886 -- Steve
We should update our ESP-IDF anyway, got behind a bit. Maybe there are also some filesystem fixes like a working fflush()? :) I second that internal RAM should be special, not external. But we need to be careful about which components and objects must use internal RAM. I've got a user report of a CAN affecting performance issue. I need to check if that can be resolved by another task priority and/or CPU core allocation scheme, but I think all low level processes should generally remain in internal RAM. If time critical CAN TX get delayed because buffers need to get paged in, that may lead to nasty things. Regards, Michael Am 25.04.2018 um 01:57 schrieb Mark Webb-Johnson:
Steve,
Because of things like this:
https://github.com/espressif/esp-idf/issues/1492
/I tried setting the "SPI RAM access method" to "Make RAM allocatable using malloc() as well", and reducing "Maximum malloc() size, in bytes, to always put in internal memory" to 32 bytes, but get this crash on startup…/ / / /Presumably that is esp_event_loop_init trying to create a FreeRTOS queue, and then objecting because it is not in internal RAM? It seems that the ESP IDF framework is not fully working with SPI RAM yet (due to inherent limitations). It would be better if the memory to be allocated must be internal, it is specifically allocated as internal (rather than just rely on a generic malloc)./
Last time I tried it (earlier this year), the ESP-IDF just simply didn’t work when that menuconfig option was set. When I fixed the above specific bug, it happened in another place.
I know Espressif have been working on it, and perhaps they’ve fixed it now? This commit seems to try to address it in a generic way (at least for the freertos port stuff, if not for all the other libraries that we use):
https://github.com/espressif/esp-idf/commit/16de6bff245dec5e63eee994f53a0825...
IDF 3.0 has officially been released last night.
Regards, Mark.
On 25 Apr 2018, at 4:40 AM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
We've been dancing around the external RAM allocation decision with our own ExtenalRamMalloc() function and extensions to the C++ new allocator to try SPIRAM first but then back off to internal memory if that fails (presumably because of v3.0 hardware rather than because all 4MB of SPIRAM is used up).
Do we need all that? There is already a mechanism in the multi-heap memory system for the defaut malloc to try allocating from SPIRAM for any request with size larger than malloc_alwaysinternal_limit which can be set dynamically with heap_caps_malloc_extmem_enable(). If the allocation from SPIRAM fails (again, presumably only on v3.0 hardware) then the allocation is retried without the MALLOC_CAP_SPIRAM capability, which is just what our ExternalRamMalloc does.
If there are only a few allocations such as stacks that must come from internal memory, and if those allocations (in system code) already use heap_caps_malloc(MALLOC_CAP_INTERNAL) to meet that requirement, then we could probably just set malloc_alwaysinternal_limit to a small number (perhaps 0) so that everything else comes from SPIRAM.
I don't know if any of our application memory uses are sufficiently sensitive to memory performance that we would want to ensure that the memory is internal. My guess is that those would be few and therefore it makes sense to take special action for them rather than for everything else.
For continued support of v3.0 hardware we could even test whether SPIRAM is present and leave malloc_alwaysinternal_limit at its default value of -1 if not.
Opinions?
-- Steve
On Mon, 23 Apr 2018, Mark Webb-Johnson wrote:
Michael,
Do we need ovms_extram.h? Can we just put the namespace directly into ovms.h (where all the other external ram allocation stuff is)?
Regards, Mark
OvmsDev mailing list OvmsDev@lists.openvehicles.com <mailto:OvmsDev@lists.openvehicles.com> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
-- Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
On Sun, 22 Apr 2018, Michael Balzer wrote:
The effect of (a) is the modem only answers "ERROR" to the +CGDATA="PPP" command. I have tried different commands to reset the internal state, but it was stuck, only a power cycle could solve this. I have added an automatic power cycle for this situation, but it didn't occur again yet.
My modem has been in the powered-off state for quite some time because I have only been testing on the bench where I have wifi connectivity. Today I installed the module in my Roadster to test some metrics as requested by Greg, and when I plugged it in I saw the blue modem light come on. After I did my test and removed the module to bring it back in and connect to my laptop, the blue light is still on. About every 20 seconds I see this cycle repeating: I (43342) simcom: State timeout, transition to 1 I (43342) simcom: State: Enter CheckPowerOff state I (54352) simcom: State: Enter PoweringOff state I (54352) gsm-ppp: Shutdown (direct) I (54352) gsm-nmea: Shutdown (direct) I (54352) simcom: Power Cycle I (54352) netmanager: Interface priority is st1 (192.168.1.84/255.255.255.0 gateway 192.168.1.254) A command "simcom setstate PoweredOff" is not effective. -- Steve
Steven, What does ‘simcom status’ show you? Also ‘config list auto’? I’m looking for the boot-time auto, and current desired power on status for the simcom. Can you try ‘power simcom off’? Regards, Mark.
On 24 Apr 2018, at 8:56 AM, Stephen Casner <casner@acm.org> wrote:
On Sun, 22 Apr 2018, Michael Balzer wrote:
The effect of (a) is the modem only answers "ERROR" to the +CGDATA="PPP" command. I have tried different commands to reset the internal state, but it was stuck, only a power cycle could solve this. I have added an automatic power cycle for this situation, but it didn't occur again yet.
My modem has been in the powered-off state for quite some time because I have only been testing on the bench where I have wifi connectivity. Today I installed the module in my Roadster to test some metrics as requested by Greg, and when I plugged it in I saw the blue modem light come on. After I did my test and removed the module to bring it back in and connect to my laptop, the blue light is still on. About every 20 seconds I see this cycle repeating:
I (43342) simcom: State timeout, transition to 1 I (43342) simcom: State: Enter CheckPowerOff state I (54352) simcom: State: Enter PoweringOff state I (54352) gsm-ppp: Shutdown (direct) I (54352) gsm-nmea: Shutdown (direct) I (54352) simcom: Power Cycle I (54352) netmanager: Interface priority is st1 (192.168.1.84/255.255.255.0 gateway 192.168.1.254)
A command "simcom setstate PoweredOff" is not effective.
-- Steve _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
Mark, My previous message was sent before I noticed that additional relevant commits were made today. After updating, I find that the looping still happens but "simcom setstate PoweredOff" stops the looping. At that point: OVMS# simcom status SIMCOM Network Registration: NotRegistered State: PoweredOff Ticker: 576 User Data: 0 Mux Status: down Open Channels: 0 Framing Errors: 0 Last RX frame: 0 sec(s) ago RX frames: 0 TX frames: 0 PPP Not Connected Last Error: Undefined GPS Status: disabled Time: disabled NMEA: GPS/GLONASS Not Connected OVMS# conf list auto auto (readable writeable) ext12v: no init: yes modem: no obd2ecu: server.v2: no server.v3: no vehicle.type: TR wifi.mode: client wifi.ssid.ap: OVMS wifi.ssid.client: Revelstoke OVMS# But "power simcom off" causes the looping to start again. -- Steve On Tue, 24 Apr 2018, Mark Webb-Johnson wrote:
Steven,
What does 'simcom status' show you? Also 'config list auto'? I'm looking for the boot-time auto, and current desired power on status for the simcom.
Can you try 'power simcom off'?
Regards, Mark.
On 24 Apr 2018, at 8:56 AM, Stephen Casner <casner@acm.org> wrote:
On Sun, 22 Apr 2018, Michael Balzer wrote:
The effect of (a) is the modem only answers "ERROR" to the +CGDATA="PPP" command. I have tried different commands to reset the internal state, but it was stuck, only a power cycle could solve this. I have added an automatic power cycle for this situation, but it didn't occur again yet.
My modem has been in the powered-off state for quite some time because I have only been testing on the bench where I have wifi connectivity. Today I installed the module in my Roadster to test some metrics as requested by Greg, and when I plugged it in I saw the blue modem light come on. After I did my test and removed the module to bring it back in and connect to my laptop, the blue light is still on. About every 20 seconds I see this cycle repeating:
I (43342) simcom: State timeout, transition to 1 I (43342) simcom: State: Enter CheckPowerOff state I (54352) simcom: State: Enter PoweringOff state I (54352) gsm-ppp: Shutdown (direct) I (54352) gsm-nmea: Shutdown (direct) I (54352) simcom: Power Cycle I (54352) netmanager: Interface priority is st1 (192.168.1.84/255.255.255.0 gateway 192.168.1.254)
A command "simcom setstate PoweredOff" is not effective.
-- Steve _______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
_______________________________________________ OvmsDev mailing list OvmsDev@lists.openvehicles.com http://lists.openvehicles.com/mailman/listinfo/ovmsdev
participants (3)
-
Mark Webb-Johnson -
Michael Balzer -
Stephen Casner