Site drops connection at regular intervals (Socket Disconnect, Push Connection shutting down)

Audience:

Technical Teams, Account Managers, Engineers, & Product Teams

Description: 

We have seen some cases where a location will show an outage every 5, 10, or even 31 minutes. The interval between outages will be consistent for that location but could be a different set interval at a different location for the same brand. The outage never lasts more than a minute.  If an order is submitted during this outage, it will fail since there is no connection. But, due to the short length of each outage, failed orders are few and far between.  

Below are some examples of what you would see in the Olo.log during the outage periods.

Example A:

2019-06-03 01:26:39.8697|2019-06-03 08:26:39.8697|INFO|PushConnection is shutting down (exceptional)
2019-06-03 01:26:39.8697|2019-06-03 08:26:39.8697|INFO|Socket 4040 closing
2019-06-03 01:26:39.8697|2019-06-03 08:26:39.8697|INFO|Closing vendor session '141057029'
2019-06-03 01:26:40.2759|2019-06-03 08:26:40.2759|INFO|Trying to connect. Supported capabilities: HealthCheck, ArrivalNotification, UpdateTtl, AdjustTip
2019-06-03 01:26:40.4478|2019-06-03 08:26:40.4478|INFO|Created vendor session '141059179'
2019-06-03 01:26:40.4478|2019-06-03 08:26:40.4478|INFO|Socket 2708 created
2019-06-03 01:26:40.4478|2019-06-03 08:26:40.4478|INFO|Socket 2708 connecting
2019-06-03 01:26:40.5259|2019-06-03 08:26:40.5259|INFO|Connected

Example B with trace logging enabled: 

2019-06-03 10:18:21.6747|2019-06-03 16:18:21.6747|TRACE|Socket has been inactive for 81.9495275 seconds. Last activity time 06/03/2019 10:16:59
2019-06-03 10:18:21.6747|2019-06-03 16:18:21.6747|INFO|PushConnection is shutting down (exceptional)
2019-06-03 10:18:21.6747|2019-06-03 16:18:21.6747|INFO|Socket 1880 closing
2019-06-03 10:18:21.6747|2019-06-03 16:18:21.6747|TRACE|Connection failed, will try again soon
Mobo.POS.PosCoreException: No socket activity for 60s - reconnecting 

2019-06-03 10:18:21.6747|2019-06-03 16:18:21.6747|INFO|Closing vendor session '141078975'
2019-06-03 10:18:21.6747|2019-06-03 16:18:21.6747|TRACE|https://api.olo.com/pos/v4/sessions/141078975|DELETE|********
2019-06-03 10:18:21.9715|2019-06-03 16:18:21.9715|TRACE|200|DELETE|********
2019-06-03 10:18:21.9715|2019-06-03 16:18:21.9715|INFO|Trying to connect. Supported capabilities: HealthCheck, ArrivalNotification, UpdateTtl, AdjustTip
2019-06-03 10:18:21.9715|2019-06-03 16:18:21.9715|TRACE|https://api.olo.com/pos/v4/sessions|POST|********
2019-06-03 10:18:22.0964|2019-06-03 16:18:22.0964|TRACE|200|POST|********
2019-06-03 10:18:22.0964|2019-06-03 16:18:22.0964|INFO|Created vendor session '141079244'
2019-06-03 10:18:22.0964|2019-06-03 16:18:22.0964|INFO|Socket 2728 created
2019-06-03 10:18:22.0964|2019-06-03 16:18:22.0964|INFO|Socket 2728 connecting
2019-06-03 10:18:22.1589|2019-06-03 16:18:22.1589|INFO|Connected

Solution: 

Some of our clients have reported that having their ISP replace their modem has resolved this issue for them. We don't have information on the model type or firmware version they are using, but this is a potential solution to this symptom. Please consult with your ISP for next steps.

If you are experiencing this issue, you can help our team gather information by submitting some general information about your local environment. Please send the following information to [e-mail]

• Brand name and location
• POS type
• Firewall make/model and who manages your policies
• Modem make/model and ISP
• POS Support company

We will add this information to our database, but cannot guarantee that we will be able to determine the root of the issue as this is specific to environments out of our control and is not directly related to the Olo Agent that is installed. Our hope is that in gathering this information from clients like you, we will eventually be able to publish a common denominator for this issue so that you can resolve it. 

Caveat: 

This does not apply to irregular socket disconnects. For those, please contact your network admin or ISP