[Ovmsdev] SIMCOM UART interrupt issue & optimization

[Michael Balzer]

From my logs, the UART overflows mostly happened with NMEA sentences, so maybe you need to enable GPS to see them. But they are rare now.

We're subscribing to sentences 2 + 64 (RMC + GNS), which happen to add up to more than 128 bytes and are sent together once per second.

CAN2 + 3 handling still is done on core 1. I don't know if the optimization of that ISR already has full effect, as the underlying GPIO ISR isn't allocated with
ESP_INTR_FLAG_IRAM. The esp-idf main GPIO ISR is in IRAM, but I think all registered sub handlers need to be as well before we can do that.

Regards,
Michael


Am 01.11.19 um 09:43 schrieb Mark Webb-Johnson:
>
> Initial feedback from me is that this looks good (both for this one and for the subsequent CAN one). Great work!
>
> My remote access via modem seems more ’snappy’ now. I get solid green LIVE on the App quicker, and response to messages is better. Perhaps there was some
> TCP/IP packet retries I had not been aware of before, or just the async is less delayed. On the async simcom port, I have never seen fifo overflows.
>
> For the past week or two, CAN2 on my car has been locking up on a daily basis, and required a ‘can can2 stop’, ‘can can2 start active 500000’ to recover. I’ve
> been running the new firmware all day, and it seems stable.
>
> I will try to stress it over the weekend, on my bench, to see how it copes. The can logging to savvycan is fairly high stress (with all 3 can buses active, a
> lot of wifi, and a bunch of translation code running on the cpus).
>
> Regards, Mark.
>
>> On 1 Nov 2019, at 5:19 AM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>
>> Everyone,
>>
>> I've pushed another optimization for the SIMCOM component that reduces the UART hardware
>> FIFO overflows (https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/issues/274)
>> to nearly zero by changing core associations.
>>
>> TL;DR:
>>
>>   * you also should apply the sdkconfig changes from the commit
>>   * we need to optimize interrupt latency
>>
>>
>> Analyzing our current core usage I found we've been running nearly all interrupt handlers on
>> core #1. That's because interrupts are bound to the core that allocates them, and after system
>> init, the housekeeper created all peripheral components on core #1.
>>
>> This has been our interrupt allocation up to now:
>>
>> Task           Run#  Lvl  ISR#  Usage                    Source
>> ----------------------------------------------------------------------------------------
>> -              0     1    0     RTC Core                 ETS_RTC_CORE_INTR_SOURCE
>> esp_timer      0     1    0     RTOS esp_timer           ETS_TIMER2_INTR_SOURCE
>> ipc0           0     1    0     RTOS Scheduler Core0     ETS_FROM_CPU_INTR0_SOURCE
>> ipc1           1     1    1     RTOS Scheduler Core1     ETS_FROM_CPU_INTR1_SOURCE
>> main           0     1    0     RTOS Watchdog            ETS_TG0_WDT_LEVEL_INTR_SOURCE
>> OVMS Events    1     1    1     Peripherals (GPIO ISR)   ETS_GPIO_INTR_SOURCE
>> OVMS Events    1     1    1     MAX7317 (spi_nodma)      ETS_SPI3_INTR_SOURCE
>> OVMS Events    1     1    1     ESP32 CAN                ETS_CAN_INTR_SOURCE
>> OVMS Events    1     1    1     SIMCOM UART              ETS_UART1_INTR_SOURCE
>> OVMS Events    1     1    1     USB console (UART)       ETS_UART0_INTR_SOURCE
>> OVMS Events    1     1    1     SD card                  ETS_SDIO_HOST_INTR_SOURCE
>> ----------------------------------------------------------------------------------------
>>
>> I had some advances on the FIFO overflows by allocating a level 3 interrupt for the SIMCOM UART,
>> but not much. I then tried moving the SIMCOM interrupt handler to core #0, and that immediately
>> reduced the overflows drastically -- most of the time.
>>
>> I read through the Espressif forums and issues on interrupt latency optimization and found a
>> precious hint here: https://www.esp32.com/viewtopic.php?t=11462#p46437
>>
>> /"Finally, by default the LWIP task is allowed to run on either core, and will sometimes take a//
>> //spinlock which requires disabling interrupts. Try pinning this task to CPU 0, only"/
>>
>> I did so, and the FIFO overflows disappeared completely on my bench. In my car there still
>> are some, but very few. I also tested pinning LWIP to core #1, that also worked and may be
>> an alternative.
>>
>> I then searched for more config candidates affecting interrupt performance and also disabled
>> the interrupt backtrace feature. That saves just 4 instructions on interrupt entry, so has
>> little effect and can be re-enabled in case we need it.
>>
>> I've reduced the UART IRQ level to 2 again as a support for my second finding regarding
>> the CAN controller, will describe that in the next post.
>>
>> Regards,
>> Michael
>>
>> -- 
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev

-- 
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20191101/43eb7486/attachment.html>