<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I'd say we should release this version now.<br>
<br>
No new issues so far, the #241 workaround works as designed, and
crash ratio is low.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 09.09.19 um 17:18 schrieb Michael
Balzer:<br>
</div>
<blockquote type="cite"
cite="mid:c107cc46-32e3-30c8-bb2e-777a751e01c2@expeedo.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Mark,<br>
<br>
I did the same checks as you and came to the same conclusion.<br>
<br>
For the webserver timer, that looks non-trivial but really only
pushes requests into the job queue and takes care to never block.
A bit of a potential issue there is it does some logging in case
of trouble (job queue full), and that may be bad if logging to an
SD file is enabled. We still need to rework the file logging to
use a separate task as well (issue #107).<br>
<br>
From my task data today, here are some first impressions on our
normal CPU load:<br>
<br>
a) Idle tasks:<br>
<br>
<img src="cid:part1.6F0619A2.5B8C50B2@expeedo.de" alt="" class=""><br>
<br>
b) Active tasks:<br>
<br>
<img src="cid:part2.15B6112D.B51716A0@expeedo.de" alt="" class=""><br>
<br>
<br>
10:00 - 10:30 was a drive from home (wifi) to office (no wifi),
then another drive 13:00 - 13:30 and back home 14:00 - 14:15.<br>
<br>
Driving load on the NetMan is caused by the web dashboard. The
half hour peaks are from the 12V topping from the main battery the
Twizy does when parked.<br>
<br>
So that's nowhere near "too much" in terms of CPU load --
normally. I'm waiting to catch the issue situation with this
instrumentation.<br>
<br>
Regards,<br>
Michael<br>
<br>
<br>
<div class="moz-cite-prefix">Am 09.09.19 um 05:43 schrieb Mark
Webb-Johnson:<br>
</div>
<blockquote type="cite"
cite="mid:B87B8287-6913-42F8-A27E-ED7E0C76C520@webb-johnson.net">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<div class="">tldr; Perhaps we should be using esp_timer and
esp_timer_get_time() to update monotonic time then dispatch
the ticker.* events?</div>
<div class=""><br class="">
</div>
<div class="">Long story: Trying to see how monotonic time could
not be updating...</div>
<div class=""><br class="">
</div>
I never really worked out how this operates in ESP32 IDF
freertos. We have three tasks involved in timers:
<div class=""><br class="">
</div>
<div class="">
<ul class="MailOutline">
<li class="">esp_timer</li>
<li class="">Tmr Svc</li>
<li class="">eventTask</li>
</ul>
<div><br class="">
</div>
<div>From my understanding, Tmr Svc provides high level timer
support for freertos (xTimerCreate, etc). And esp_timer
provides esp32 specific timer support (<a
href="https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/esp_timer.html"
class="" moz-do-not-send="true">https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/esp_timer.html</a>).
We use Tmr Svc, and note the warnings provided in the
espressif documentation:</div>
<div><br class="">
</div>
<div>
<div class="">
<ul class="MailOutline">
<li class="">Maximum resolution is equal to RTOS tick
period</li>
<li class="">Timer callbacks are dispatched from a
low-priority task</li>
</ul>
</div>
</div>
<div><br class="">
</div>
<div>Currently, our housekeeping uses xTimerCreate to create a
1 second timer (calling HousekeepingTicker1). So, in which
task is HousekeepingTicker1() called (esp_timer or Tmr Svc)
- I always suspected the latter (but never checked).
Then HousekeepingTicker1() raises a signal (using ovms
events) that pushes it onto a queue (if the queue is full,
it discards). The eventTask will then read that queue, and
dispatch it to our ticker.1 listeners.</div>
<div><br class="">
</div>
<div>HousekeepingTicker1 is responsible for
updating monotonictime, and given it’s simple calling of
MyEvents.SignalEvent (which just queues it and discards on
overflow), I don’t think that can block for any substantial
time.</div>
<div><br class="">
</div>
<div>I did a search for xTimerCreate in our code base, and
find these used:</div>
<div><br class="">
</div>
<div>
<ul class="MailOutline">
<li class="">components/ovms_webserver/src/ovms_webserver.cpp<br
class="">
m_update_ticker = xTimerCreate("Web client update
ticker", 250 / portTICK_PERIOD_MS, pdTRUE, NULL,
UpdateTicker);<br class="">
<br class="">
This seems to do quite a bit. In particular queue
handling and semaphores. All seem to be non-blocking,
but the flow is non-trivial.<br class="">
<br class="">
</li>
<li class="">components/vehicle_nissanleaf/src/vehicle_nissanleaf.cpp<br
class="">
m_remoteCommandTimer = xTimerCreate("Nissan Leaf Remote
Command", 100 / portTICK_PERIOD_MS, pdTRUE, this,
remoteCommandTimer);<br class="">
m_ccDisableTimer = xTimerCreate("Nissan Leaf CC
Disable", 1000 / portTICK_PERIOD_MS, pdFALSE, this,
ccDisableTimer);<br class="">
<br class="">
Seem ok, and non-blocking.<br class="">
<br class="">
</li>
<li class="">components/vehicle_renaulttwizy/src/rt_sevcon.cpp<br
class="">
m_kickdown_timer = xTimerCreate("RT kickdown",
pdMS_TO_TICKS(100), pdTRUE, NULL, KickdownTimer);<br
class="">
<br class="">
Seems ok.<br class="">
<br class="">
</li>
<li class="">components/vehicle_smarted/src/vehicle_smarted.cpp<br
class="">
m_locking_timer = xTimerCreate("Smart ED Locking Timer",
500 / portTICK_PERIOD_MS, pdTRUE, this,
SmartEDLockingTimer);<br class="">
<br class="">
This code looks a bit dodgy because CommandLock and
CommandUnlock both create this timer, and start it - but
neither check if it is already created. That said, after
it fires the timer is deleted by the handler.<br
class="">
<br class="">
</li>
<li class="">components/vehicle_teslaroadster/src/vehicle_teslaroadster.cpp<br
class="">
m_speedo_timer = xTimerCreate("TR ticker",<br class="">
m_homelink_timer = xTimerCreate("Tesla Roadster Homelink
Timer", durationms / portTICK_PERIOD_MS, pdTRUE, this,
TeslaRoadsterHomelinkTimer);<br class="">
<br class="">
Similar lack of checking for duplicate timers. But I
don’t see any blocking.</li>
</ul>
</div>
<div><br class="">
</div>
<div>So, I don’t really think _we_ are starving the TmrSvc.
Most likely something in the core framework.</div>
<div><br class="">
</div>
<div>Regards, Mark.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 7 Sep 2019, at 4:55 PM, Michael Balzer
<<a href="mailto:dexter@expeedo.de" class=""
moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""> I think
the RTOS timer service task starves. It's running on
core 0 with priority 1.<br class="">
<br class="">
Taks on core 0 sorted by priority:<br class="">
<br class="">
<tt class="">Number of Tasks = 20 Stack: Now
Max Total Heap 32-bit SPIRAM C# PRI</tt><tt
class=""><br class="">
</tt><tt class=""><tt class="">3FFC84A8 6 Blk
ipc0 388 500 1024 7788
0 0 0 24</tt><tt class=""><br class="">
</tt></tt><tt class=""><tt class="">3FFC77F0 5 Blk
OVMS CanRx 428 428 2048 3052 0
31844 0 23</tt><tt class=""><br class="">
</tt>3FFAFBF4 1 Blk esp_timer 400 656
4096 35928 644 25804 0 22</tt><tt class=""><br
class="">
</tt><tt class=""><tt class="">3FFD3240 19 Blk
wifi 460 2716 3584 43720
0 20 0 22</tt><tt class=""><br class="">
</tt>3FFC03C4 2 Blk eventTask 448 1984
4608 104 0 0 0 20</tt><tt class=""><br
class="">
</tt><tt class=""><tt class="">3FFC8F14 17 Blk
tiT 500 2308 3072 6552
0 0 * 18</tt><tt class=""><br class="">
</tt></tt><tt class=""><tt class="">3FFE14F0 26 Blk
OVMS COrx 456 456 4096 0
0 0 0 7</tt><tt class=""><br class="">
</tt><tt class="">3FFE19D4 27 Blk OVMS COwrk
476 476 3072 0 0 0 0 7</tt><tt
class=""><br class="">
</tt>3FFCBC34 12 Blk Tmr Svc 352 928
3072 88 0 0 0 1</tt><tt class=""><br
class="">
</tt><tt class="">3FFE7708 23 Blk mdns
468 1396 4096 108 0 0 0 1</tt><tt
class=""><br class="">
</tt><br class="">
I don't think it's our CanRx, as that only fetches and
queues CAN frames, the actual work is done by the
listeners. The CO tasks only run for CANopen jobs,
which are few for normal operation.<br class="">
<br class="">
That leaves the system tasks, with main suspect -once
again- the wifi blob.<br class="">
<br class="">
We need to know how much CPU time the tasks actually
use now. I think I saw some option for this in the
FreeRTOS config.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 06.09.19 um 23:15
schrieb Michael Balzer:<br class="">
</div>
<blockquote type="cite"
cite="mid:01014878-5a73-b611-4334-989b8aea76c5@expeedo.de"
class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
The workaround is based on the monotonictime being
updated per second, as do the history record
offsets.<br class="">
<br class="">
Apparently, that mechanism doesn't work reliably.
That may be an indicator for some bigger underlying
issue.<br class="">
<br class="">
Example log excerpt:<br class="">
<br class="">
<tt class="">2019-09-06 22:07:48.126919 +0200 info
main: #173 C MITPROHB rx msg h
964,0,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:03.089031 +0200
info main: #173 C MITPROHB rx msg h
964,-10,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:05.041574 +0200
info main: #173 C MITPROHB rx msg h
964,-20,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:05.052644 +0200
info main: #173 C MITPROHB rx msg h
964,-30,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:05.063617 +0200
info main: #173 C MITPROHB rx msg h
964,-49,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:05.077527 +0200
info main: #173 C MITPROHB rx msg h
964,-59,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:05.193775 +0200
info main: #173 C MITPROHB rx msg h
964,-70,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:13.190645 +0200
info main: #173 C MITPROHB rx msg h
964,-80,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:22.077994 +0200
info main: #173 C MITPROHB rx msg h
964,-90,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:09:54.590300 +0200
info main: #173 C MITPROHB rx msg h
964,-109,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:11:10.127054 +0200
info main: #173 C MITPROHB rx msg h
964,-119,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:11:16.794200 +0200
info main: #173 C MITPROHB rx msg h
964,-130,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:11:22.455652 +0200
info main: #173 C MITPROHB rx msg h
964,-140,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:12:49.423412 +0200
info main: #173 C MITPROHB rx msg h
964,-150,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:12:49.442096 +0200
info main: #173 C MITPROHB rx msg h
964,-169,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:12:49.461941 +0200
info main: #173 C MITPROHB rx msg h
964,-179,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:14:39.828133 +0200
info main: #173 C MITPROHB rx msg h
964,-190,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:14:39.858144 +0200
info main: #173 C MITPROHB rx msg h
964,-200,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:14:52.020319 +0200
info main: #173 C MITPROHB rx msg h
964,-210,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:14:54.452637 +0200
info main: #173 C MITPROHB rx msg h
964,-229,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:15:12.613935 +0200
info main: #173 C MITPROHB rx msg h
964,-239,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:15:35.223845 +0200
info main: #173 C MITPROHB rx msg h
964,-250,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:16:09.255059 +0200
info main: #173 C MITPROHB rx msg h
964,-260,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:17:31.919754 +0200
info main: #173 C MITPROHB rx msg h
964,-270,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:19:23.366267 +0200
info main: #173 C MITPROHB rx msg h
964,-289,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:21:57.344609 +0200
info main: #173 C MITPROHB rx msg h
964,-299,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:23:40.082406 +0200
info main: #31 C MITPROHB rx msg h
964,-1027,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><tt class="">2019-09-06 22:25:58.061883 +0200
info main: #31 C MITPROHB rx msg h
964,-1040,RT-BAT-C,5,86400,2,1,3830,3795,3830,-10,25,25,25,0</tt><tt
class=""><br class="">
</tt><br class="">
<br class="">
This shows the ticker was only run 299 times from
22:07:48 to 22:21:57.<br class="">
<br class="">
After 22:21:57 the workaround was triggered and did
a reconnect. Apparently during that network
reinitialization of 103 seconds, the per second
ticker was run 628 times.<br class="">
<br class="">
That can't be catching up on the event queue, as
that queue has only 20 slots. So something strange
is going on here.<br class="">
<br class="">
Any ideas?<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 06.09.19 um 08:04
schrieb Michael Balzer:<br class="">
</div>
<blockquote type="cite"
cite="mid:9ee4a7b5-59a2-a05d-046d-e932b65e3e34@expeedo.de"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
Mark & anyone else running a V2 server,<br
class="">
<br class="">
as most cars don't send history records, this also
needs the change to the server I just pushed, i.e.
server version 2.4.2.<br class="">
<br class="">
<a
href="https://github.com/openvehicles/Open-Vehicle-Monitoring-System/commits/master"
moz-do-not-send="true" class="">https://github.com/openvehicles/Open-Vehicle-Monitoring-System/commits/master</a><br
class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 05.09.19 um 19:55
schrieb Michael Balzer:<br class="">
</div>
<blockquote type="cite"
cite="mid:91892018-3ecb-2dc9-b430-e79fe598cc4c@expeedo.de"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
I've pushed the nasty workaround: the v2 server
checks for no RX over 15 minutes, then restarts
the network (wifi & modem) as configured for
autostart.<br class="">
<br class="">
Rolled out on my server in edge as
3.2.002-237-ge075f655.<br class="">
<br class="">
Please test.<br class="">
<br class="">
Regards,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am 05.09.19 um
01:58 schrieb Mark Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:D4C2658B-49FB-48CB-A687-9A7325117BC4@webb-johnson.net"
class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8" class="">
<div class="">
<blockquote type="cite" class="">
<div text="#000000" bgcolor="#FFFFFF"
class="">Mark, you can check your server
logs for history messages with
ridiculous time offsets:<br class="">
<blockquote class=""><tt class="">[sddexter@ns27
server]$ cat log-20190903 | egrep
"rx msg h [0-9]+,-[0-9]{4}" | wc -l<br
class="">
455283</tt></blockquote>
</div>
</blockquote>
<br class="">
</div>
I checked my logs and see 12 vehicles showing
this. But, 2 only show this for a debugcrash
log (which is expected, I guess, if the time
is not synced at report time). I’ve got 4 cars
with the offset > 10,000.
<div class=""><br class="">
</div>
<div class="">Regards, Mark.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 4 Sep 2019, at 4:45 AM,
Michael Balzer <<a
href="mailto:dexter@expeedo.de"
class="" moz-do-not-send="true">dexter@expeedo.de</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8"
class="">
<div text="#000000" bgcolor="#FFFFFF"
class=""> Everyone,<br class="">
<br class="">
I've pushed a change that needs some
testing.<br class="">
<br class="">
I had the issue myself now parking
at a certain distance from my garage
wifi AP, i.e. on the edge of "in",
after wifi had been disconnected for
some hours, and with the module
still connected via modem. The wifi
blob had been trying to connect to
the AP for about two hours.<br
class="">
<br class="">
As seen before, the module saw no
error, just the server responses and
commands stopped coming in. I
noticed the default interface was
still "st1" despite wifi having been
disconnected and modem connected.
The DNS was also still configured
for my wifi network, and the
interface seemed to have an IP
address -- but wasn't pingable from
the wifi network.<br class="">
<br class="">
A power cycle of the modem solved
the issue without reboot. So the
cause may be in the modem/ppp
subsystem, or it may be related (in
some weird way) to the default
interface / DNS setup.<br class="">
<br class="">
More tests showed the default
interface again/still got set by the
wifi blob itself at some point,
overriding our modem prioritization.
The events we didn't handle up to
now were "sta.connected" and
"sta.lostip", so I added these, and
the bug didn't show up again since
then. That doesn't mean anything, so
we need to test this.<br class="">
<br class="">
The default interface really
shouldn't affect inbound packet
routing of an established
connection, but there always may be
strange bugs lurking in those libs.<br
class="">
<br class="">
The change also reimplements the
wifi signal strength reading, as the
tests also showed that still wasn't
working well using the CSI callback.
It now seems to be much more
reliable.<br class="">
<br class="">
Please test & report. The single
module will be hard to test, as the
bug isn't reproducable easily, but
you can still try if wifi / modem
transitions work well.<br class="">
<br class="">
Mark, you can check your server logs
for history messages with ridiculous
time offsets:<br class="">
<blockquote class=""><tt class="">[sddexter@ns27
server]$ cat log-20190903 |
egrep "rx msg h
[0-9]+,-[0-9]{4}" | wc -l<br
class="">
455283</tt><br class="">
</blockquote>
The bug now severely affects the V2
server performance, as the server is
single threaded and doesn't scale
very well to this kind of bulk data
bursts, especially when coming from
multiple modules in parallel. So we
really need to solve this now. Slow
reactions or connection drops from
my server lately have been due to
this bug. If this change doesn't
solve it, we'll need to add some
reboot trigger on "too many server
v2 notification retransmissions" --
or maybe a modem power cycle will
do, that wouldn't discard the data.<br
class="">
<br class="">
Thanks,<br class="">
Michael<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">Am
03.09.19 um 07:46 schrieb Mark
Webb-Johnson:<br class="">
</div>
<blockquote type="cite"
cite="mid:7A81965B-AD97-4DFC-9768-9A03A74107BA@webb-johnson.net"
class="">
<pre class="moz-quote-pre" wrap="">No problem. We can hold. I won’t commit anything for the next few days (and agree to hold-off on Markos’s pull). Let me know when you are ready.
Regards, Mark.
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">On 3 Sep 2019, at 1:58 AM, Michael Balzer <a class="moz-txt-link-rfc2396E" href="mailto:dexter@expeedo.de" moz-do-not-send="true"><dexter@expeedo.de></a> wrote:
Mark, please wait.
I may just have found the cause for issue #241, or at least something I need to investigate before releasing.
I need to dig into my logs first, and try something.
Regards,
Michael
Am 02.09.19 um 12:23 schrieb Michael Balzer:
</pre>
<blockquote type="cite" class="">
<pre class="moz-quote-pre" wrap="">Nothing open from my side at the moment.
I haven't had the time to look in to Markos pull request, but from a first check also think that's going too deep to be included in this release.
Regards,
Michael
Am 02.09.19 um 04:15 schrieb Mark Webb-Johnson:
</pre>
<blockquote type="cite"
class="">
<pre class="moz-quote-pre" wrap="">I think it is well past time for a 3.2.003 release. Things seems table in edge (although some things only partially implemented).
Anything people want to include at the last minute, or can we go ahead and build?
Regards, Mark.
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br class="">
OvmsDev mailing list<br class="">
<a
href="mailto:OvmsDev@lists.openvehicles.com"
class="" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</div>
_______________________________________________<br
class="">
OvmsDev mailing list<br class="">
<a href="mailto:OvmsDev@lists.openvehicles.com" class=""
moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev"
moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com" moz-do-not-send="true">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev" moz-do-not-send="true">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
OvmsDev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OvmsDev@lists.openvehicles.com">OvmsDev@lists.openvehicles.com</a>
<a class="moz-txt-link-freetext" href="http://lists.openvehicles.com/mailman/listinfo/ovmsdev">http://lists.openvehicles.com/mailman/listinfo/ovmsdev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="160">--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
</pre>
</body>
</html>