[Ovmsdev] Apple push notifications no longer working
Mark Webb-Johnson
mark at webb-johnson.net
Thu Dec 17 16:39:10 HKT 2020
Michael,
Glad you found this.
The openssl library is a nightmare. A lot of distributions rigidly stick to one version for this kind of reason. Like libc, glibc, etc. Updating it can be scary.
Perl does have some powerful dependency checking and versioning (even including scripting capability on the imported module version number), but it is rare to see developers use it.
Regards, Mark.
> On 16 Dec 2020, at 5:42 AM, Michael Balzer <dexter at expeedo.de> wrote:
>
> Signed PGP part
> Found & fixed it.
>
> Grepping for apns in the log wasn't sufficient, the relevant hint was on the output line following the log entry:
>
> 2020-12-15 20:01:10.969115 +0100 info main: - - - msg apns connected to gateway.sandbox.push.apple.com, now establishing SSL security
> EV: error in callback (ignoring): Your vendor has not defined SSLeay macro ST_OK at ovms_server.pl line 1683.
>
> Looking for this on the web, I found this 2018 bug report on AnyEvent::Handle vs. OpenSSL:
>
> https://rt.cpan.org/Public/Bug/Display.html?id=124723 <https://rt.cpan.org/Public/Bug/Display.html?id=124723>
>
> The perl test command showed the exact error, so I applied the linked patch to the AnyEvent::Handle module, and voila, it's working again.
>
> Strange this only affected the APNS connection.
>
> The cause was probably the perl-OpenSSL update on my server on 2020-10-01, which came after the latest AnyEvent update (2020-09-18). I have to admit, over the years I grew a bit of hatred for perl, from a server management point of view. No other subsystems on my servers (with the exception of python of course) have had so many issues with incompatible package updates.
>
> Regards,
> Michael
>
>
> Am 15.12.20 um 07:55 schrieb Mark Webb-Johnson:
>> I think perhaps the best is to add an overall timer, launched just before the tcp_connect, and cancelled in all the known exit paths. Then that timer (perhaps 60 seconds) could cleanup the push. At least that would avoid the whole system jamming up.
>>
>> But that doesn’t solve the core problem of why you can’t connect to apple (but I can). Your tcp connection state is “UNCONN”, which means the disconnected, I assume.
>>
>> I think there is an AnyEvent debug setting, but that is likely to produce a lot of verbose output on a production server.
>>
>> Regards, Mark,
>>
>>> On 15 Dec 2020, at 4:20 AM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>
>>> Signed PGP part
>>> The certificates are those you sent me. Just checked, they're both valid.
>>>
>>> I've added your suggestion and also added the "connected" log message from PushAPNS.pm, no luck.
>>>
>>> 2020-12-14 13:52:00.840238 +0100 info main: - - XXXXXXXX msg queued apns notification for sandbox:XXXXXXXXXXXXXX
>>> 2020-12-14 13:52:01.579649 +0100 info main: - - - msg apns processing queue for gateway.sandbox.push.apple.com <http://gateway.sandbox.push.apple.com/>
>>> 2020-12-14 13:52:01.740147 +0100 info main: - - - msg apns connected to gateway.sandbox.push.apple.com <http://gateway.sandbox.push.apple.com/>, now establishing SSL security
>>>
>>> I tried reducing the APNS channels to "production", still no luck.
>>>
>>> I can see the socket getting created when following socket events:
>>>
>>> [root at ns34 ~]# ss -E | grep "17\.188"
>>> tcp UNCONN 0 0 146.0.237.226:59102 17.188.136.189:2195
>>>
>>> So the TLS init somehow fails in a way AnyEvent::Handle doesn't recognize as a failure / error.
>>>
>>> I've had a look at the AnyEvent::Handle documentation but cannot see anything we're doing wrong or missed regarding error handling.
>>>
>>> Maybe Apple is blocking me, dropping all packets? But wouldn't that trigger a timeout or TLS init error?
>>>
>>> Very strange. Any other ideas?
>>>
>>> Regards,
>>> Michael
>>>
>>>
>>> Am 14.12.20 um 09:19 schrieb Mark Webb-Johnson:
>>>> Do you have conf/ovms_apns_sandbox.pem file in place? Valid and not expired?
>>>>
>>>> I had a quick review, and it seems the main flow handles errors. Perhaps some other callback on the AnyEvent::Handle for an error condition is being missed? Or perhaps the AnyEvent::Handle could not be created at all. Can you try to add:
>>>>
>>>> If (!defined $apns_handle)
>>>> {
>>>> AE::log error => "- - - msg apns handle could not be created”;
>>>> $apns_running = 0;
>>>> }
>>>>
>>>> After the block of ‘$apns_handle = new AnyEvent::Handle(…’?
>>>>
>>>> Regards, Mark.
>>>>
>>>>> On 14 Dec 2020, at 3:21 PM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>>
>>>>> Signed PGP part
>>>>> That's the strange part: I don't get any error, and I don't get a timeout either. That's what I meant by "fails in a way we don't handle".
>>>>>
>>>>> This is the only log entry on "apns processing", I get this once after restarting the server as soon as the first APN is due for delivery:
>>>>>
>>>>> 2020-12-10 16:55:30.351530 +0100 info main: - - - msg apns processing queue for gateway.sandbox.push.apple.com <http://gateway.sandbox.push.apple.com/>
>>>>>
>>>>> (still running server v2 due to lack of time)
>>>>>
>>>>> After that, no more apns processing – I guess because $apns_running never gets reset.
>>>>>
>>>>> Any idea?
>>>>>
>>>>> Regards,
>>>>> Michael
>>>>>
>>>>>
>>>>> Am 14.12.20 um 07:17 schrieb Mark Webb-Johnson:
>>>>>>
>>>>>> Push notifications are still working ok for me. What is the error you get back from the gateway?
>>>>>>
>>>>>> We are using this protocol (see apns_send function):
>>>>>>
>>>>>> https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/BinaryProviderAPI.html <https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/BinaryProviderAPI.html>
>>>>>>
>>>>>> We need to convert to this one:
>>>>>>
>>>>>> https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/ <https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/>
>>>>>>
>>>>>> But have until the end of March 2021 to do it. The switch is not complex, and is probably slightly easier for us. It becomes just a simple http request, with pretty much the same payload we currently use (in json format).
>>>>>>
>>>>>> Regards, Mark.
>>>>>>
>>>>>>> On 11 Dec 2020, at 12:27 AM, Michael Balzer <dexter at expeedo.de <mailto:dexter at expeedo.de>> wrote:
>>>>>>>
>>>>>>> Signed PGP part
>>>>>>> Mark,
>>>>>>>
>>>>>>> a user informed me he no longer gets any push notifications to iOS.
>>>>>>>
>>>>>>> Looking into the logs, it seems the initial connect to the gateway fails in a way the perl code does not handle.
>>>>>>>
>>>>>>> I've found this in the Apple forums: https://developer.apple.com/forums/thread/667248 <https://developer.apple.com/forums/thread/667248>
>>>>>>>
>>>>>>> It seems the protocol we use has been deprecated, but it should continue to work until March. Do you see a similar effect on your server?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Michael
>>>>>>>
>>>>>>> --
>>>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OvmsDev mailing list
>>>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>>>
>>>>> --
>>>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OvmsDev mailing list
>>>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>>>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>>>
>>> --
>>> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
>>> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>>>
>>>
>>
>>
>>
>> _______________________________________________
>> OvmsDev mailing list
>> OvmsDev at lists.openvehicles.com <mailto:OvmsDev at lists.openvehicles.com>
>> http://lists.openvehicles.com/mailman/listinfo/ovmsdev <http://lists.openvehicles.com/mailman/listinfo/ovmsdev>
>
> --
> Michael Balzer * Helkenberger Weg 9 * D-58256 Ennepetal
> Fon 02333 / 833 5735 * Handy 0176 / 206 989 26
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20201217/fe5ebe47/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.openvehicles.com/pipermail/ovmsdev/attachments/20201217/fe5ebe47/attachment-0001.sig>
More information about the OvmsDev
mailing list