Sense Status - Intermittent Connectivity Issues

Intermittent Connectivity Issues - Resolved

Incident Report for Sense

Postmortem

Between Monday, July 25, 2022 to Thursday, July 28, 2022, we experienced a series of intermittent issues with our websocket service. The websocket service is responsible for pushing live updates to open web pages and Chrome extension tabs of Sense Messaging users when time-sensitive information comes in, such as receiving new SMS messages from candidates. The issues were intermittent and did not impact all users equally.

The websocket service issue was caused by a scalability tipping point. The websocket service was unable to safely scale with the addition of new servers and increased scale.

The team has deployed a newly designed service as of Thursday, July 28 and are continuously monitoring its performance as well as customer support tickets. We are seeing stable connectivity since. We are fully auditing all older services to meet our current design and scalability requirements, and we have put in additional rigorous standards and procedures to mitigate issues like this in the future.

If you or your team are experiencing any issues, please submit a ticket through our portal and our Technical Support Team will review: https://support.sensehq.co/hc/en-us.

Posted Aug 04, 2022 - 13:52 PDT

Resolved

Our team has confirmed that we are no longer seeing any evidence of connectivity issues for our users, and our product is functioning as expected once more. Please have any affected user log out and back in to Sense if still experiencing this error.

If you or your users are still experiencing this issue after logging out and back in, please submit a ticket through our support portal:
https://support.sensehq.co/hc/en-us/requests/new

Posted Jul 28, 2022 - 16:05 PDT

Monitoring

A fix has been implemented to eliminate the connectivity issues some users have been facing throughout the day. We will be monitoring closely to ensure this does not recur. Please have any affected user log out and back in to Sense if still experiencing this error.

If you or your users are still experiencing this issue after logging out and back in, please submit a ticket through our support portal:
https://support.sensehq.co/hc/en-us/requests/new

Posted Jul 28, 2022 - 15:27 PDT

Update

We appreciate your patience as we continue to work on correcting the current connectivity issue that some users are experiencing. We are actively working to deploy this fix as soon as possible.

We will continue to provide updates as we receive more information

Posted Jul 28, 2022 - 14:37 PDT

Update

We are continuing to work on the connectivity issue that some users are experiencing and actively progressing to have a resolution as soon as possible. We will continue to provide updates as we receive more information

Posted Jul 28, 2022 - 12:34 PDT

Update

We are continuing to work on this connectivity issue that some users are experiencing. We will continue to provide updates as we receive more information

Posted Jul 28, 2022 - 10:26 PDT

Identified

We have seen the connectivity issue is still persisting and users are experiencing issues not being able hit the send option for SMS due to no connectivity, as well as new messages not loading. We have identified the root cause of the bug and are working to have this corrected on priority

We will continue to update as we receive more information

Posted Jul 28, 2022 - 08:47 PDT

This incident affected: Sense Messaging (Messaging Chrome Extension, Messaging Web Portal, Messaging Mobile App).