[Ovmsdev] Apple push notifications no longer working
Michael Balzer
dexter at expeedo.de
Wed Dec 16 05:42:40 HKT 2020
Found & fixed it.
Grepping for apns in the log wasn't sufficient, the relevant hint was on
the output line following the log entry:
2020-12-15 20:01:10.969115 +0100 info main: - - - msg apns connected to
gateway.sandbox.push.apple.com, now establishing SSL security
EV: error in callback (ignoring): Your vendor has not defined SSLeay
macro ST_OK at ovms_server.pl line 1683.
Looking for this on the web, I found this 2018 bug report on
AnyEvent::Handle vs. OpenSSL:
https://rt.cpan.org/Public/Bug/Display.html?id=124723
The perl test command showed the exact error, so I applied the linked
patch to the AnyEvent::Handle module, and voila, it's working again.
Strange this only affected the APNS connection.
The cause was probably the perl-OpenSSL update on my server on
2020-10-01, which came after the latest AnyEvent update (2020-09-18). I
have to admit, over the years I grew a bit of hatred for perl, from a
server management point of view. No other subsystems on my servers (with
the exception of python of course) have had so many issues with
incompatible package updates.
Regards,
Michael
Am 15.12.20 um 07:55 schrieb Mark Webb-Johnson:
> I think perhaps the best is to add an overall timer, launched just
> before the tcp_connect, and cancelled in all the known exit paths.
> Then that timer (perhaps 60 seconds) could cleanup the push. At least
> that would avoid the whole system jamming up.
>
> But that doesn’t solve the core problem of why you can’t connect to
> apple (but I can). Your tcp connection state is “UNCONN”, which means
> the disconnected, I assume.
>
> I think there is an AnyEvent debug setting, but that is likely to
> produce a lot of verbose output on a production server.
>
> Regards, Mark,
>
>> On 15 Dec 2020, at 4:20 AM, Michael Balzer <dexter at expeedo.de
>> <mailto:dexter at expeedo.de>> wrote:
>>
>> Signed PGP part
>> The certificates are those you sent me. Just checked, they're both valid.
>>
>> I've added your suggestion and also added the "connected" log message
>> from PushAPNS.pm, no luck.
>>
>> 2020-12-14 13:52:00.840238 +0100 info main: - - XXXXXXXX msg queued
>> apns notification for sandbox:XXXXXXXXXXXXXX
>> 2020-12-14 13:52:01.579649 +0100 info main: - - - msg apns
>> processing queue for gateway.sandbox.push.apple.com
>> <http://gateway.sandbox.push.apple.com>
>> 2020-12-14 13:52:01.740147 +0100 info main: - - - msg apns connected
>> to gateway.sandbox.push.apple.com
>> <http://gateway.sandbox.push.apple.com>, now establishing SSL security
>>
>> I tried reducing the APNS channels to "production", still no luck.
>>
>> I can see the socket getting created when following socket events:
>>
>> [root at ns34 ~]# ss -E | grep "17\.188"
>> tcp UNCONN 0 0 146.0.237.226:59102 17.188.136.189:2195
>>
>> So the TLS init somehow fails in a way AnyEvent::Handle doesn't
>> recognize as a failure / error.
>>
>> I've had a look at the AnyEvent::Handle documentation but cannot see
>> anything we're doing wrong or missed regarding error handling.
>>
>> Maybe Apple is blocking me, dropping all packets? But wouldn't that
>> trigger a timeout or TLS init error?
>>
>> Very strange. Any other ideas?
>>
>> Regards,
>> Michael
>>
>>
>> Am 14.12.20 um 09:19 schrieb Mark Webb-Johnson:
>>> Do you have conf/ovms_apns_sandbox.pem file in place? Valid and not
>>> expired?
>>>
>>> I had a quick review, and it seems the main flow handles errors.
>>> Perhaps some other callback on the AnyEvent::Handle for an error
>>> condition is being missed? Or perhaps the AnyEvent::Handle could not
>>> be created at all. Can you try to add:
>>>
>>> If (!defined $apns_handle)
>>> {
>>> AE::log error => "- - - msg apns handle could not be created”;
>>> $apns_running = 0;
>>> }
>>>
>>>
>>> After the block of ‘$apns_handle = new AnyEvent::Handle(…’?
>>>
>>> Regards, Mark.
>>>
>>>> On 14 Dec 2020, at 3:21 PM, Michael Balzer <dexter at expeedo.de
>>>> <mailto:dexter at expeedo.de>> wrote:
>>>>
>>>> Signed PGP part
>>>> That's the strange part: I don't get any error, and I don't get a
>>>> timeout either. That's what I meant by "fails in a way we don't
>>>> handle".
>>>>
>>>> This is the only log entry on "apns processing", I get this once
>>>> after restarting the server as soon as the first APN is due for
>>>> delivery:
>>>>
>>>> 2020-12-10 16:55:30.351530 +0100 info main: - - - msg apns
>>>> processing queue for gateway.sandbox.push.apple.com
>>>> <http://gateway.sandbox.push.apple.com/>
>>>>
>>>> (still running server v2 due to lack of time)
>>>>
>>>> After that, no more apns processing – I guess because $apns_running
>>>> never gets reset.
>>>>
>>>> Any idea?
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>>
>>>> Am 14.12.20 um 07:17 schrieb Mark Webb-Johnson:
>>>>>
>>>>> Push notifications are still working ok for me. What is the error
>>>>> you get back from the gateway?
>>>>>
>>>>> We are using this protocol (see apns_send function):
>>>>>
>>>>> https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/BinaryProviderAPI.html
>>>>> <https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/BinaryProviderAPI.html>
>>>>>
>>>>>
>>>>> We need to convert to this one:
>>>>>
>>>>> https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/
>>>>> <https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/>
>>>>>
>>>>>
>>>>> But have until the end of March 2021 to do it. The switch is not
>>>>> complex, and is probably slightly easier for us. It becomes just a
>>>>> simple http request, with pretty much the same payload we
>>>>> currently use (in json format).
>>>>>
>>>>> Regards, Mark.
>>>>>
>>>>>> On 11 Dec 2020, at 12:27 AM, Michael Balzer <dexter at expeedo.de
>>>>>> <mailto:dexter at expeedo.de>> wrote:
>>>>>>
>>>>>> Signed PGP part
>>>>>> Mark,
>>>>>>
>>>>>> a user informed me he no longer gets any push notifications to iOS.
>>>>>>
>>>>>> Looking into the logs, it seems the initial connect to the
>>>>>> gateway fails in a way the perl code does not handle.
>>>>>>
>>>>>> I've found this in the Apple forums:
>>>>>> https://developer.apple.com/forums/thread/667248
>>>>>> <https://developer.apple.com/forums/thread/667248>
>>>>>>
>>>>>> It seems the protocol we use has been deprecated, but it should
>>>>>> continue to work until March. Do you see a similar effect on your
>>>>>> server?
>>>>>>
>>>>>> Regards,
>>>>>> Michael
>>>>>>
>>>>>> --
>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OvmsDev mailing list
>>>>> OvmsDev at lists.openvehicles.com
>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>>>
>>>> --
>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>
>>>
>>>
>>> _______________________________________________
>>> OvmsDev mailing list
>>> OvmsDev at lists.openvehicles.com
>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
>>
>> --
>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>
>
>
> _______________________________________________
> OvmsDev mailing list
> OvmsDev at lists.openvehicles.com
> http://lists.openvehicles.com/mailman/listinfo/ovmsdev
--
Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20201215/538ad78e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20201215/538ad78e/attachment-0001.sig>
More information about the OvmsDev
mailing list