[Ovmsdev] Repeated WDT crashes after upgrade to 3.3 with 4G
Michael Balzer
dexter at expeedo.de
Fri Mar 4 14:50:41 HKT 2022
Mark,
there really seems to be some issue with -285.
To help in tracking down release correlating issues I've just added
another hourly developer report on my server:
https://dexters-web.de/f/firmware/developer/crashreport-overview.csv
That's basically the same source as…
https://dexters-web.de/f/firmware/developer/crashreport.csv
…just with reduced detail and more strict grouping.
Regards,
Michael
crashreport.csv:
SELECT
LEFT(h_data, INSTR(h_data,'/')-1) AS firmware_version,
SUBSTRING_INDEX(SUBSTRING_INDEX(m_msg, ',', 5), ',', -1) AS car_type,
COUNT(DISTINCT c.vehicleid) AS vehicle_count,
COUNT(*) AS crashtype_count,
CONVERT_TZ(MAX(h_timestamp),'+00:00','SYSTEM') AS last_time,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 12,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',13),',',-1), '') AS
resetreason_name,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',8),',',-1) AS crashtype,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',9),',',-1) AS crashcore,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',14),',',-1), '') AS
curr_event_name,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',15),',',-1), '') AS
curr_event_handler,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',16),',',-1), '') AS
curr_event_runtime,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 13,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',17),',',-1), '') AS
wdt_tasknames,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',11),',',-1) AS backtrace,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',10),',',-1) AS registers,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 17,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',18),',',-1), '') AS
hardware_info
FROM ovms_historicalmessages h
JOIN ovms_carmessages c ON ( c.vehicleid = h.vehicleid AND m_code = 'F' )
WHERE
h_recordtype = '*-OVM-DebugCrash' AND h_data LIKE '%build idf%'
AND h_timestamp >= NOW() - INTERVAL 10 DAY
GROUP BY firmware_version, car_type, resetreason_name, backtrace,
curr_event_name, wdt_tasknames
ORDER BY firmware_version, car_type, last_time
crashreport-overview.csv:
SELECT
SUBSTRING_INDEX(SUBSTRING_INDEX(m_msg, ',', 5), ',', -1) AS car_type,
LEFT(h_data, INSTR(h_data,'/')-1) AS firmware_version,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 17,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',18),',',-1), '') AS
hardware_info,
IF(LENGTH(h_data) - LENGTH(REPLACE(h_data, ',', '')) >= 12,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',13),',',-1), '') AS
resetreason_name,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',8),',',-1) AS crashtype,
SUBSTRING_INDEX(SUBSTRING_INDEX(h_data,',',9),',',-1) AS crashcore,
COUNT(DISTINCT c.vehicleid) AS vehicle_count,
COUNT(*) AS crashtype_count,
CONVERT_TZ(MAX(h_timestamp),'+00:00','SYSTEM') AS last_time
FROM ovms_historicalmessages h
JOIN ovms_carmessages c ON ( c.vehicleid = h.vehicleid AND m_code = 'F' )
WHERE
h_recordtype = '*-OVM-DebugCrash' AND h_data LIKE '%build idf%'
AND h_timestamp >= NOW() - INTERVAL 10 DAY
GROUP BY car_type, firmware_version, hardware_info, resetreason_name,
crashtype, crashcore
ORDER BY car_type, firmware_version, hardware_info, resetreason_name,
crashtype, crashcore
Am 04.03.22 um 06:55 schrieb Mark Webb-Johnson:
> Steve, Jeff,
>
> I have noticed you running '3.3.001-285’. Can you try with the
> standard ‘3.3.001’ that was released a while ago? You can get it from:
>
> http://api.openvehicles.com/firmware/ota/v3.3/main/ovms3.bin
>
> or
>
> http://api.openvehicles.com/firmware/ota/v3.2/edge/3.3.001.ovms3.bin
>
>
> I wonder if something is broken in the latest edge?
>
> P.S. I think that in general we may be confusing people here. The
> firmware/ota/<X> directory is for those version <X> mainboards. So
> /v3.1/, /v3.2/ or /v3.3/. The idea is that we can then have different
> build settings (sdkconfig) for different base hardware (in particular
> esp32 revision) and make improvements. At the moment v3.1 and v3.2 are
> symlinked together (on api.openvehicles.com
> <http://api.openvehicles.com>) as ESP32 is the same.
>
> Regards, Mark
>
>> On 4 Mar 2022, at 9:17 AM, Stephen Casner <casner at acm.org> wrote:
>>
>> On Thu, 3 Mar 2022, Mark Webb-Johnson wrote:
>>
>>> Can you try to get a debug log for the cellular system?
>>
>> Inline below. This is with cellular, gsm-mux and SIM7600 set to
>> debug. Any other useful tags? Looks like it dies not long after
>> entering NetStart state.
>>
>> -- Steve
>>
>> 2022-03-03 17:08:45.249 PST I (547059) cellular: Set modem driver to
>> 'auto'
>> 2022-03-03 17:08:45.249 PST I (547059) cellular: State: Enter
>> PowerOffOn state
>> 2022-03-03 17:08:45.249 PST I (547059) cellular-modem-auto: Power
>> Cycle 2000ms
>> 2022-03-03 17:08:47.539 PST D (549349) cellular: State timeout
>> PowerOffOn => PoweringOn
>> 2022-03-03 17:08:47.539 PST I (549349) cellular: State: Enter
>> PoweringOn state
>> 2022-03-03 17:08:47.539 PST I (549349) cellular-modem-auto: Power
>> Cycle 4000ms
>> 2022-03-03 17:08:51.539 PST D (553349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:51.539 PST D (553349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:51.539 PST D (553349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:51.539 PST D (553349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:52.539 PST D (554349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:53.539 PST D (555349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:54.539 PST D (556349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:55.539 PST D (557349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:56.539 PST D (558349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:57.539 PST D (559349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:58.539 PST D (560349) cellular: tx-cmd: AT
>> 2022-03-03 17:08:59.539 PST D (561349) cellular: tx-cmd: AT
>> 2022-03-03 17:09:00.539 PST D (562349) cellular: tx-cmd: AT
>> 2022-03-03 17:09:01.539 PST D (563349) cellular: tx-cmd: AT
>> 2022-03-03 17:09:02.539 PST D (564349) cellular: tx-cmd: AT
>> 2022-03-03 17:09:02.979 PST I (564789) cellular: State: Enter
>> Identify state
>> 2022-03-03 17:09:03.539 PST D (565349) cellular: tx-cmd: AT+CGMM
>> 2022-03-03 17:09:03.549 PST I (565359) cellular: Identified cellular
>> modem: SIM7600/Experimental support for SIMCOM SIM7600
>> 2022-03-03 17:09:03.549 PST D (565359) cellular: Remove old 'auto'
>> modem driver
>> 2022-03-03 17:09:03.549 PST I (565359) cellular: Set modem driver to
>> 'SIM7600'
>> 2022-03-03 17:09:03.549 PST I (565359) cellular: State: Enter
>> PoweredOn state
>> 2022-03-03 17:09:13.539 PST D (575349) cellular: tx-cmd:
>> AT+CPIN?;+CREG=1;+CTZU=1;+CTZR=1;+CLIP=1;+CMGF=1;+CNMI=1,2,0,0,0;+CSDH=1;+CMEE=2;+CSQ;+AUTOCSQ=1,1;E0;S0=0
>> 2022-03-03 17:09:13.569 PST D (575379) cellular: mux-rx-line #0:
>> AT+CPIN?;+CREG=1;+CTZU=1;+CTZR=1;+CLIP=1;+CMGF=1;+CNMI=1,2,0,0,0;+CSDH=1;+CMEE=2;+CSQ;+AUTOCSQ=1,1;E0;S0=0
>> 2022-03-03 17:09:13.569 PST D (575379) cellular: mux-rx-line #0:
>> +CPIN: READY
>> 2022-03-03 17:09:13.569 PST D (575379) cellular: mux-rx-line #0:
>> +CSQ: 13,99
>> 2022-03-03 17:09:13.579 PST I (575389) cellular: Signal Quality is:
>> 13 (-87 dBm)
>> 2022-03-03 17:09:15.539 PST D (577349) cellular: tx-cmd: AT+CGMR;+ICCID
>> 2022-03-03 17:09:15.559 PST D (577369) cellular: mux-rx-line #0:
>> +CGMR: LE20B03SIM7600M21-A
>> 2022-03-03 17:09:15.559 PST D (577369) cellular: mux-rx-line #0:
>> +ICCID: 8944501805175416178
>> 2022-03-03 17:09:23.539 PST D (585349) cellular: tx-cmd: AT+CMUX=0
>> 2022-03-03 17:09:23.549 PST I (585359) cellular: State: Enter
>> MuxStart state
>> 2022-03-03 17:09:23.549 PST I (585359) gsm-mux: Start MUX
>> 2022-03-03 17:09:23.559 PST I (585369) gsm-mux: Channel #0 is open
>> 2022-03-03 17:09:23.569 PST I (585379) gsm-mux: Channel #1 is open
>> 2022-03-03 17:09:23.579 PST I (585389) gsm-mux: Channel #2 is open
>> 2022-03-03 17:09:23.589 PST I (585399) gsm-mux: Channel #3 is open
>> 2022-03-03 17:09:23.599 PST I (585409) gsm-mux: Channel #4 is open
>> 2022-03-03 17:09:24.539 PST D (586349) cellular: State transition
>> MuxStart => NetWait
>> 2022-03-03 17:09:24.539 PST I (586349) cellular: State: Enter NetWait
>> state
>> 2022-03-03 17:09:34.539 PST D (596349) cellular: mux-tx #3:
>> AT+CREG?;+CCLK?;+CSQ;+COPS?
>> 2022-03-03 17:09:34.569 PST D (596379) cellular: mux-rx-line #3:
>> +CREG: 1,5
>> 2022-03-03 17:09:34.569 PST I (596379) cellular: Network Registration
>> status: RegisteredRoaming
>> 2022-03-03 17:09:34.569 PST D (596379) cellular: mux-rx-line #3:
>> +CCLK: "22/03/03,17:09:32-32"
>> 2022-03-03 17:09:34.569 PST D (596379) cellular: mux-rx-line #3:
>> +CSQ: 13,99
>> 2022-03-03 17:09:34.569 PST D (596379) cellular: mux-rx-line #3:
>> +COPS: 0,0,"AT&T Hologram",2
>> 2022-03-03 17:09:34.569 PST I (596379) cellular: Network Provider is:
>> AT&T Hologram
>> 2022-03-03 17:09:35.539 PST D (597349) cellular: State transition
>> NetWait => NetStart
>> 2022-03-03 17:09:35.539 PST I (597349) cellular: State: Enter
>> NetStart state
>> 2022-03-03 17:09:40.549 PST I (602359) housekeeping: 2022-03-03
>> 17:09:39 PST (RAM: 8b=78500-79076 32b=12344 SPI=3439196-3442228)
>> 2022-03-03 17:09:48.539 PST I (610349) ovms-server-v2: Send MP-0
>> S78.0,M,65535,127.00,done,standard,164,138,17,0,0,140,7,4,0,0,0,0,0.00,0,0,0,0,-1,0,0,0,0,0,0,0,0.00,0.00,0,0.00,0.00,1.00,0.0
>> 2022-03-03 17:09:48.539 PST I (610349) ovms-server-v2: Send MP-0
>> D64,0,5,19,17,17,512,625508,0,95954,19,0,0,0,11.54,0,12.6,0,19,0,0
>> 2022-03-03 17:09:48.549 PST I (610359) ovms-server-v2: Send MP-0
>> L37.346333,-122.037285,80,54,1,1,0.0,512,0,0.393,0,0,0,0,,0,0,0
>> 2022-03-03 17:09:48.549 PST I (610359) ovms-server-v2: Send MP-0
>> F3.3.001-285-g601f2a70/ota_1/edge (build idf v3.3.4-848-g1ff5e24b1
>> Feb 20 2022
>> 00:04:30),5YJRE11B181000033,30,0,TR1N,Revelstoke,-1,-1,OVMS WIFI BLE
>> BT cores=2 rev=ESP32/1; MODEM SIM7600
>> 2022-03-03 17:10:47.479 PST I (9639) command: OpenLogfile: now
>> logging to file '/sd/crash.log'
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20220304/571d1085/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20220304/571d1085/attachment-0001.sig>
More information about the OvmsDev
mailing list