<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><blockquote type="cite" class="">But this will only catch the problem if there is some task that is<br class="">running away. It could be some kind of synchronization block instead.<br class=""></blockquote><div class=""><br class=""></div>Yes, that is my guess (a sync block).<div class=""><br class=""></div><div class=""><blockquote type="cite" class="">Another<br class="">possibility that could be used in scenarios where wifi is still<br class="">working and doesn't need to be brought up/down would be to connect<br class="">with telnet or ssh and then issue the the simcom command in that<br class="">console.</blockquote><div class=""><br class=""></div>From my experience, the async console is fine, up until the time we touch the network (at which point the async console locks up). I think the issue is in the TiT tcp/ip task (which is the only viable explanation for those tcp/ip timeouts I see on wifi, shortly after ppp goes down:</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (136619030) gsm-ppp: Shutting down (hard)...^[[0m</span><br class="" style="font-family: "Andale Mono"; font-size: 14px;"><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (136619040) events: Signal(system.modem.down)^[[0m</span><br class="" style="font-family: "Andale Mono"; font-size: 14px;"><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (136619040) netmanager: Interface priority is st1 (x.y.z.212/255.255.248.0 gateway x.y.z.64)</span></div><div class=""><span style="font-family: "Andale Mono"; font-size: 14px;" class=""><br class=""></span></div><div class=""><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (179290100) wifi: bcn_timout,ap_probe_send_start</span><br class="" style="font-family: "Andale Mono"; font-size: 14px;"><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (179292610) wifi: ap_probe_send over, resett wifi status to disassoc</span><br class="" style="font-family: "Andale Mono"; font-size: 14px;"><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (179292610) wifi: state: run -> init (1)</span><br class="" style="font-family: "Andale Mono"; font-size: 14px;"><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (179292620) wifi: pm stop, total sleep time: 0/138543477</span><br class="" style="font-family: "Andale Mono"; font-size: 14px;"><span style="font-family: "Andale Mono"; font-size: 14px;" class="">I (179292620) wifi: n:13 0, o:13 0, ap:255 255, sta:13 0, prof:1</span></div></blockquote><div class=""><div><br class=""></div><div>I think that is 179292 - 136619 = 11 hours, though!</div><div><br class=""></div><div>With Steve’s latest extension to ‘module tasks stack’, we should be able to narrow this down. I think I will firstly see if the watchdog works around it.</div><div><br class=""></div><div>Regards, Mark.</div><div><br class=""><blockquote type="cite" class=""><div class="">On 26 Mar 2018, at 12:28 AM, Stephen Casner <<a href="mailto:casner@acm.org" class="">casner@acm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Enabling the watchdog timer is a good idea. I've had it enabled in my<br class="">config for some time. When I was recently working on enhancements to<br class="">the "module memory" command and had an infinite loop in my code, I got<br class="">a timer trap:<br class=""><br class="">Task watchdog got triggered. The following tasks did not reset the watchdog in time:<br class=""> - IDLE (CPU 1)<br class="">Tasks currently running:<br class="">CPU 0: IDLE<br class="">CPU 1: AsyncConsole<br class=""><br class="">But this will only catch the problem if there is some task that is<br class="">running away. It could be some kind of synchronization block instead.<br class=""><br class="">If there is a particular command that seems to get stuck, we could<br class="">consider (temporarily) invoking that command as a separate task so<br class="">that the async console would still be usable to investigate. Another<br class="">possibility that could be used in scenarios where wifi is still<br class="">working and doesn't need to be brought up/down would be to connect<br class="">with telnet or ssh and then issue the the simcom command in that<br class="">console.<br class=""><br class="">I have added printing of each task's state in the "module tasks"<br class="">output, but I have yet to observe any task in the "Run" state. (I<br class="">would expect AsyncConsole to be in Run state while executing that<br class="">command, but it is not.)<br class=""><br class="">I might be able to add an option on the command to print each task's<br class="">PC, which the python code would look up to translate to a source<br class="">line number.<br class=""><br class=""> -- Steve<br class=""><br class="">On Sun, 25 Mar 2018, Mark Webb-Johnson wrote:<br class=""><br class=""><blockquote type="cite" class="">I think that the housekeeping task is locked up. With the latest code I have, the per-10-minute housekeeping message has stopped.<br class=""><br class="">My guess is still the ppp code, during session teardown.<br class=""><br class="">I sent a detailed message on this an hour or so, with my analysis of this.<br class=""><br class="">Regards, Mark.<br class=""><br class=""><blockquote type="cite" class="">On 25 Mar 2018, at 5:52 PM, Tom Parker <<a href="mailto:tom@carrott.org" class="">tom@carrott.org</a>> wrote:<br class=""><br class="">On 25/03/18 02:41, Mark Webb-Johnson wrote:<br class=""><blockquote type="cite" class="">The issue I was having was some task (I suspect lwip) gets messed up. Stopping the modem, trying to connect wifi, etc, all just locked up the async console. I think you had the same?<br class=""></blockquote><br class="">Re-winding this thread to the 'power simcom off' freezes the console aspect as distinct from the disabling the simcard issue. I had the module disconnect again this evening, though without a datalogger attached. Some more information about this state:<br class=""><br class="">The mux is up<br class="">AT communication with the simcom works on channel 3 and logs the expected tx and rx<br class="">The simcom is connected to the cellular network<br class="">No log messages recording periodic communications with the simcom are being recorded<br class="">The monotonic and park time counters are not advancing.<br class=""><br class="">Could the cause of this problem be that the timers have stopped? This could explain why the periodic interrogation of the simcom has stopped, and perhaps simcom power off has a spin wait or other loop waiting forever for the timer to advance?<br class=""><br class="">What is the best way to investigate the state of the timers and periodic execution?<br class=""><br class="">See attached for a transcript of this debugging session.<br class=""><ovms_2018-03-25T09_29_07+0000.log.bz2>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.teslaclub.hk" class="">OvmsDev@lists.teslaclub.hk</a><br class="">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev<br class=""></blockquote><br class="">_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.teslaclub.hk" class="">OvmsDev@lists.teslaclub.hk</a><br class="">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev<br class=""><br class=""><br class=""></blockquote>_______________________________________________<br class="">OvmsDev mailing list<br class=""><a href="mailto:OvmsDev@lists.teslaclub.hk" class="">OvmsDev@lists.teslaclub.hk</a><br class="">http://lists.teslaclub.hk/mailman/listinfo/ovmsdev<br class=""></div></div></blockquote></div><br class=""></div></body></html>