Now that we can now see this, it is quite scary: Number of Tasks = 17 Stack: Now Max Total Heap 32-bit SPIRAM 3FFF2658 23 Rdy NetManTask 3116 5340 7168 12152 27488 0 $ ~/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-addr2line -pfiaC -e build/ovms3.elf 0x40000000 0x40000000 0x400ffe7c 0x400f2466 0x400f45f6 0x400f4726 0x400d8335 0x40152cdd 0x40151b95 0x400d81ee 0x400d830e 0x400e9d20 0x400e2e7d 0x400e2f94 0x400e2f86 0x400e2f86 0x400e2fbc 0x400edbab 0x400f11e4 0x400f124b 0x400edbda 0x40152976 0x401529e9 0x402050ee 0x400d8ded 0x400e5b4c 0x400d8474 0x400f6d25 0x400f6f1d 0x4008c604 0x40081374 0x400da5bb 0x400da622 0x400f647e 0x401e58d8 0x401d12f6 0x400ffe7c 0x401d1d8c 0x400f2466 0x400f45f6 0x400f777a 0x400f778d 0x400f778d 0x400f7801 0x400f7a59 0x400f7ce8 0x400f4494 0x400f4494 0x400eaf4d 0x400eaf8c 0x40000000: ?? ??:0 0x40000000: ?? ??:0 0x400ffe7c: gettimeofday at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/syscalls/../../../.././newlib/libc/syscalls/sysgettod.c:13 0x400f2466: cs_time at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f45f6: mg_time at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f4726: mg_send at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400d8335: SendCallback(WOLFSSH*, void*, unsigned int, void*) at vehicle/OVMS.V3/components/console_ssh/src/console_ssh.cpp:976 0x40152cdd: HighwaterCheck at vehicle/OVMS.V3/components/wolfssh/src/internal.c:4194 (inlined by) SendBuffered at vehicle/OVMS.V3/components/wolfssh/src/internal.c:849 0x40151b95: wolfSSH_stream_send at vehicle/OVMS.V3/components/wolfssh/src/ssh.c:635 0x400d81ee: ConsoleSSH::write(void const*, unsigned int) at vehicle/OVMS.V3/components/console_ssh/src/console_ssh.cpp:926 0x400d830e: ConsoleSSH::printf(char const*, ...) at vehicle/OVMS.V3/components/console_ssh/src/console_ssh.cpp:897 0x400e9d20: module_tasks(int, OvmsWriter*, OvmsCommand*, int, char const* const*) at vehicle/OVMS.V3/main/./ovms_module.cpp:823 0x400e2e7d: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at vehicle/OVMS.V3/main/./ovms_command.cpp:94 0x400e2f94: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at vehicle/OVMS.V3/main/./ovms_command.cpp:94 0x400e2f86: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at vehicle/OVMS.V3/main/./ovms_command.cpp:94 0x400e2f86: OvmsCommand::Execute(int, OvmsWriter*, int, char const* const*) at vehicle/OVMS.V3/main/./ovms_command.cpp:94 0x400e2fbc: OvmsCommandApp::Execute(int, OvmsWriter*, int, char const* const*) at vehicle/OVMS.V3/main/./ovms_command.cpp:94 0x400edbab: Execute(microrl*, int, char const* const*) at vehicle/OVMS.V3/main/./ovms_shell.cpp:49 0x400f11e4: print_prompt at vehicle/OVMS.V3/components/microrl/./microrl.c:281 (inlined by) new_line_handler at vehicle/OVMS.V3/components/microrl/./microrl.c:621 0x400f124b: microrl_insert_char at vehicle/OVMS.V3/components/microrl/./microrl.c:669 0x400edbda: OvmsShell::ProcessChar(char) at vehicle/OVMS.V3/main/./ovms_shell.cpp:70 0x40152976: GrowBuffer at vehicle/OVMS.V3/components/wolfssh/src/internal.c:4194 0x401529e9: GetInputData at vehicle/OVMS.V3/components/wolfssh/src/internal.c:4194 0x402050ee: OvmsShell::ProcessChars(char const*, int) at vehicle/OVMS.V3/main/./ovms_shell.cpp:75 (discriminator 2) 0x400d8ded: ConsoleSSH::HandleDeviceEvent(void*) at vehicle/OVMS.V3/components/console_ssh/src/console_ssh.cpp:479 0x400e5b4c: OvmsConsole::Poll(unsigned int, void*) at vehicle/OVMS.V3/main/./ovms_console.cpp:153 0x400d8474: ConsoleSSH::Receive() at vehicle/OVMS.V3/components/console_ssh/src/console_ssh.cpp:308 0x400f6d25: mg_http_handler2 at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f6f1d: mg_http_handler at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x4008c604: vTaskExitCritical at /Users/mark/esp/esp-idf/components/freertos/./tasks.c:4837 0x40081374: esp_crosscore_int_send at /Users/mark/esp/esp-idf/components/esp32/./crosscore_int.c:103 0x400da5bb: OvmsSSH::EventHandler(mg_connection*, int, void*) at vehicle/OVMS.V3/components/console_ssh/src/console_ssh.cpp:110 0x400da622: MongooseHandler(mg_connection*, int, void*) at vehicle/OVMS.V3/components/console_ssh/src/console_ssh.cpp:61 0x400f647e: mg_call at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x401e58d8: netconn_recved at /Users/mark/esp/esp-idf/components/lwip/api/api_lib.c:830 (discriminator 4) 0x401d12f6: lwip_recvfrom at /Users/mark/esp/esp-idf/components/lwip/api/sockets.c:3229 0x400ffe7c: gettimeofday at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/syscalls/../../../.././newlib/libc/syscalls/sysgettod.c:13 0x401d1d8c: lwip_recvfrom_r at /Users/mark/esp/esp-idf/components/lwip/api/sockets.c:3229 0x400f2466: cs_time at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f45f6: mg_time at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f777a: mg_recv_common at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f778d: mg_if_recv_tcp_cb at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f778d: mg_if_recv_tcp_cb at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f7801: mg_handle_tcp_read at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f7a59: mg_mgr_handle_conn at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f7ce8: mg_socket_if_poll at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f4494: mg_mgr_poll at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400f4494: mg_mgr_poll at vehicle/OVMS.V3/components/mongoose/mongoose/mongoose.c:11373 0x400eaf4d: OvmsNetManager::MongooseTask() at vehicle/OVMS.V3/main/./ovms_netmanager.cpp:380 0x400eaf8c: MongooseRawTask(void*) at vehicle/OVMS.V3/main/./ovms_netmanager.cpp:370 That is a huge stack for a little box. Regards, Mark.
On 26 Mar 2018, at 3:31 PM, Mark Webb-Johnson <mark@webb-johnson.net> wrote:
That is pretty cool. The ‘make monitor’ auto-magic address discovery makes the output useful:
3FFC8BAC 16 Blk tiT 484 692 6144 0 0 128 0x401fe551 0x4008ebfd 0x401f23f8 0x401ebd0c 0x401e978e 0x401e978e 0x401fe551: sys_arch_mbox_fetch at /Users/mark/esp/esp-idf/components/lwip/port/freertos/sys_arch.c:548 0x4008ebfd: xQueueGenericReceive at /Users/mark/esp/esp-idf/components/freertos/./queue.c:2037 0x401f23f8: sys_timeouts_mbox_fetch at /Users/mark/esp/esp-idf/components/lwip/core/timers.c:551 0x401ebd0c: netif_poll at /Users/mark/esp/esp-idf/components/lwip/core/netif.c:440 0x401e978e: tcpip_thread at /Users/mark/esp/esp-idf/components/lwip/api/tcpip.c:474 0x401e978e: tcpip_thread at /Users/mark/esp/esp-idf/components/lwip/api/tcpip.c:474
Now, the only issue is recreating the problem in a manner where I can get in with ‘make monitor’. Or, maybe not…
$ ~/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-addr2line -pfiaC -e build/ovms3.elf 0x401fe551 0x4008ebfd 0x401f23f8 0x401ebd0c 0x401e978e 0x401e978e 0x401fe551: sys_arch_mbox_fetch at /Users/mark/esp/esp-idf/components/lwip/port/freertos/sys_arch.c:548 0x4008ebfd: xQueueGenericReceive at /Users/mark/esp/esp-idf/components/freertos/./queue.c:2037 0x401f23f8: sys_timeouts_mbox_fetch at /Users/mark/esp/esp-idf/components/lwip/core/timers.c:551 0x401ebd0c: netif_poll at /Users/mark/esp/esp-idf/components/lwip/core/netif.c:440 0x401e978e: tcpip_thread at /Users/mark/esp/esp-idf/components/lwip/api/tcpip.c:474 0x401e978e: tcpip_thread at /Users/mark/esp/esp-idf/components/lwip/api/tcpip.c:474
It seems that I only need the ‘.elf’ file that was used to build the .bin that went on the module.
This might be workable. I will try to keep the .elf and .bin files, and see if I can get something the next time it locks up (although I think my watchdog reboot may now avoid that).
I’m 99.9% certain the issue is in the TiT tcpip task. Both wifi and pppos go down, while other parts of the system still seem to be running ok.
Regards, Mark.
On 26 Mar 2018, at 2:33 PM, Stephen Casner <casner@acm.org <mailto:casner@acm.org>> wrote:
On Sun, 25 Mar 2018, Stephen Casner wrote:
I might be able to add an option on the command to print each task's PC, which the python code would look up to translate to a source line number.
I have now done this. "module tasks" works as before, but "module tasks stack" shows a crude stack trace for each task.
-- Steve _______________________________________________ OvmsDev mailing list OvmsDev@lists.teslaclub.hk <mailto:OvmsDev@lists.teslaclub.hk> http://lists.teslaclub.hk/mailman/listinfo/ovmsdev