tag:status.sensehq.com,2005:/historySense Status - Incident History2024-03-28T22:23:00-07:00Sensetag:status.sensehq.com,2005:Incident/198200862024-01-28T08:00:36-08:002024-01-28T08:00:36-08:00Sense Database Maintenance<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>08:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>20:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:28</var> PST</small><br><strong>Scheduled</strong> - Sense will be performing scheduled maintenance from Saturday January 27, 8pm PST to Sunday January 28th, 8am PST.<br /><br />During this time, users may not be able to access the Sense platform through the web browser, web extension, or mobile application. This may also produce server errors on chatbot or scheduler links. Any scheduled messages to go out during this time will need to be rescheduled outside this window and any data triggered events will still be sent and caught up when maintenance is completed. All inbound SMS messages will be received once maintenance is completed as well.<br /><br />If you have any questions or concerns, please submit a ticket at https://support.sensehq.co/hc/en-us/requests/new for our Support team to look into</p>tag:status.sensehq.com,2005:Incident/188581442023-10-18T18:00:00-07:002023-10-19T13:31:22-07:00Bulk Cleanup Downtime<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>18:00</var> PDT</small><br><strong>Resolved</strong> - Our team found there was an issue with the bulk cleanup function within our Engage product that caused cleanups scheduled between October 12th through October 18th to not run during that time frame. We have since identified the root cause of this and any active cleanup that did not run has since been backfilled and completed<br /><br />If you believe you have any issues due to this downtime, please reach out to Sense support at https://support.sensehq.co/hc/en-us/requests/new</p>tag:status.sensehq.com,2005:Incident/187463042023-10-10T16:30:56-07:002023-10-10T16:30:56-07:00AI Generated Messages Not Loading<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>16:30</var> PDT</small><br><strong>Resolved</strong> - After monitoring the affected components, we are seeing everything operational, and all products functioning as expected. <br /><br />If you believe you are still experiencing issues, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new or support@sensehq.com</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>16:11</var> PDT</small><br><strong>Monitoring</strong> - Our team has reported Sense AI and Chatbot issues have subsided. We expect systems should be fully functional at this point. This outage is now being moved to the “monitoring” stage to confirm the recent changes before officially declaring as resolved.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>16:09</var> PDT</small><br><strong>Update</strong> - Our team has reported Sense AI and Chatbot issues have subsided. We expect systems should be fully functional at this point. This outage is now being moved to the “monitoring” stage to confirm the recent changes before officially declaring as resolved.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>14:30</var> PDT</small><br><strong>Update</strong> - Our team is still actively investigating this error when attempting to generate AI messages through Sense products<br /><br />We'll provide an update as soon as we receive more information</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>11:33</var> PDT</small><br><strong>Investigating</strong> - We are currently seeing increased reports of the Sense AI app is giving an error when attempting to generate a message for intermittent users.<br /><br />Our team is actively investigating this and we'll provide updates as we receive them</p>tag:status.sensehq.com,2005:Incident/186356362023-09-28T15:45:08-07:002023-09-28T15:45:08-07:00Possible Carrier Filtering Message<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>15:45</var> PDT</small><br><strong>Resolved</strong> - After monitoring deliverability traffic and confirming with our downstream partners, we are seeing successful deliveries once more to all carriers. Any message that failed during this time will not be resent automatically and will need to be resent manually as needed<br /><br />If you believe you are still experiencing issues, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new or support@sensehq.com</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>13:45</var> PDT</small><br><strong>Monitoring</strong> - We received confirmation from our downstream vendors that they were able to identify and mitigate the issue with the affected carriers, and we’re seeing successful traffic once more. Any message that failed during this time will not be resent automatically and will need to be resent manually as needed<br /><br />Our team is continuing monitor the traffic for all numbers and continuing to work with our vendors incase of a recurrence. If you believe you are still experiencing issues, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new or support@sensehq.com</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>13:02</var> PDT</small><br><strong>Investigating</strong> - We are receiving reports of some users receiving an error message when attempting to send messages stating "Possible Carrier Filtering". We’ve received confirmation from our downstream vendors that there is a current blocker that is causing messages to not be delivered intermittently, specifically to AT&T and Verizon carriers. <br /><br />We are working with our downstream vendors to resolve this blocker, and we will continue to provide updates as we receive them</p>tag:status.sensehq.com,2005:Incident/185377162023-09-18T15:31:38-07:002023-09-18T15:31:38-07:00AWS Services Down<p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>15:31</var> PDT</small><br><strong>Resolved</strong> - Our team has continued to monitor and confirmed we’re seeing all products function as expected once more. We will continually monitor the servers on the backend and review to see the full scope of the effects and will communicate this once confirmed.<br /><br />If you believe you are still experiencing issues, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new or support@sensehq.com</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>14:10</var> PDT</small><br><strong>Monitoring</strong> - As our team continues to monitor the AWS outage, we are seeing a decrease in load failures within Sense pages and components. We are still seeing some failures, mainly on our Chatbot product, that is still causing users to not be able to load or view pages, and will continue to monitor these statuses.<br /><br />We’ll provide more updates as we receive them</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>11:45</var> PDT</small><br><strong>Identified</strong> - We are currently seeing issues due to an AWS outage. You may be experiencing intermittent load failures across various pages and components. Furthermore, some inbound messages and delivery analytics may be delayed or missing as a result. Our team is conducting a thorough impact analysis to better understand the situation and determine the best course of action. <br /><br />We apologize for any inconvenience caused and appreciate your patience as we work to resolve this issue</p>tag:status.sensehq.com,2005:Incident/180333982023-08-02T11:30:21-07:002023-08-02T11:30:21-07:00Sense Discover Outage<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>11:30</var> PDT</small><br><strong>Resolved</strong> - After monitoring the affected components, we are seeing everything operational, and all products functioning as expected. <br /><br />If you believe you are still experiencing issues, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new or support@sensehq.com</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>10:40</var> PDT</small><br><strong>Monitoring</strong> - We have identified the root cause and rolled out a fix to correct this. We are seeing the Discover Product and Field Writebacks function as expected once more. Any missed writebacks during this time are currently reprocessing and we should see this all catch up shortly.<br /><br />Our team will continue to monitor to ensure we don’t see this revert, and we’ll provide more information once we have it. If you believe you are still experiencing issues, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new or support@sensehq.com</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>10:15</var> PDT</small><br><strong>Investigating</strong> - Our team has found an issue with our Discover Product in Sense, and is actively investigating the root cause for a fix. In addition we are seeing the use for Field Writebacks are not showing available or recording back to the candidate record. We did confirm any missed data will be caught up once we implement a fix.<br /><br />We'll continue to provide updates as we receive them</p>tag:status.sensehq.com,2005:Incident/176993412023-06-27T08:00:38-07:002023-06-27T08:00:39-07:00Sense API Error<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>Resolved</strong> - Our team has confirmed we're seeing all systems operate as expected once more, and all users are able to navigate the portal as expected. <br /><br />If you believe you are still experiencing issues, please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>07:15</var> PDT</small><br><strong>Monitoring</strong> - Our team has identified the root cause of the API error, and has implemented a fix. We are seeing servers stabilize for all users once more, and are continuing to monitor to ensure all fixes have taken correctly.<br /><br />If you believe you are still experiencing issues, please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>06:45</var> PDT</small><br><strong>Investigating</strong> - We are seeing increased reports of users receiving an API error when attempting to send messages through Sense or navigating through the portal. Our team is actively investigating the root cause and working to get all systems operational once more. <br /><br />We’ll be sure to provide updates as we receive them”</p>tag:status.sensehq.com,2005:Incident/174909672023-06-06T16:31:42-07:002023-06-06T16:31:42-07:00Sense Meetings Errors<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>16:31</var> PDT</small><br><strong>Resolved</strong> - We have received confirmation that this issue has been resolved. If you continue to see any issues, please reach out to Support@sensehq.com.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>15:46</var> PDT</small><br><strong>Monitoring</strong> - Our downstream vendor has confirmed they are once again seeing latency reduced to normal levels, and all functions stabilized. We will continue to monitor this with them and provide updates as we receive them</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>14:51</var> PDT</small><br><strong>Identified</strong> - We have received an update from our downstream vendor that they have made an update to their server and are seeing latency decrease. They are continuing to make this update and monitoring the status as this is implemented. We’ll continue to provide updates as we receive them</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>13:28</var> PDT</small><br><strong>Update</strong> - Our downstream vendor is still actively investigating the cause for the increased latency on the server. We are working with them and will be sure to continue provide updates as we receive them</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>10:40</var> PDT</small><br><strong>Investigating</strong> - There has been reports that the fix implemented did not take as expected and our downstream vendor is seeing latency pick back up causing errors to come through once more. <br /><br />We’ll continue to work with them for updates and provide what we have here as we receive them</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>10:10</var> PDT</small><br><strong>Monitoring</strong> - We have received reports that our downstream vendor has put in a fix and are seeing their servers working as expected once more. Sense is seeing the product function as expected once more for users as well, but will continue to monitor the progress.<br /><br />If you believe you are still experiencing issues, please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>08:36</var> PDT</small><br><strong>Investigating</strong> - We have received notification from our downstream scheduler service that they are seeing increased latency on their servers, causing appointments to not be able to be booked or throwing errors for intermittent recipients. They have confirmed they are actively investigating this and we will provide updates as soon as we receive them</p>tag:status.sensehq.com,2005:Incident/173334282023-05-23T12:25:15-07:002023-05-23T12:25:15-07:00Delay in Note Writebacks for Messaging<p><small>May <var data-var='date'>23</var>, <var data-var='time'>12:25</var> PDT</small><br><strong>Resolved</strong> - We have confirmed we are continuing to see all note writebacks process as expected throughout today after monitoring the fix. We will still continue to backfill the writebacks that were delayed during this time and provide updates on a case by case basis.<br /><br />If you believe you are not seeing new note writebacks process correctly, please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Update</strong> - After monitoring, we can confirm all note writebacks are processing as expected for today’s notes. We are still in the process of backfilling all other missed writebacks and will continue to monitor on our end they continue processing. <br /><br />We will continue to provide updates on the backfill as we receive them. If you believe you are not seeing new note writebacks process correctly, please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>06:00</var> PDT</small><br><strong>Update</strong> - Our team has confirmed that we are seeing all Note Writebacks process in realtime once more. We are continuing to reprocess any that may have been missed during this time as well. We will continue to monitor the status of these and provide updates as we receive them<br /><br />If you believe you are not seeing new note writebacks process correctly, please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>13:49</var> PDT</small><br><strong>Monitoring</strong> - Our team has discovered a delay in Note Writebacks being posted from the Messaging product. The cause has already been identified and a solution has been implemented. We will continue to monitor as the delayed Writebacks are being processed and provide updates once that is complete.</p>tag:status.sensehq.com,2005:Incident/172121762023-05-11T09:10:49-07:002023-05-11T09:10:49-07:00Deliverability Failures to T-Mobile Recipients<p><small>May <var data-var='date'>11</var>, <var data-var='time'>09:10</var> PDT</small><br><strong>Resolved</strong> - We are seeing normal deliverability rates for all networks once more. <br /><br />We will continue to monitor delivery traffic to ensure we do not see any spikes recur. If you believe you are still experiencing failures please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>08:01</var> PDT</small><br><strong>Monitoring</strong> - We have received confirmation that our vendor was able to work with the downstream carrier and reduce the amount of SMS filtering with the T-Mobile network. We are now seeing normal delivery rates to all carriers once more. <br /><br />We are continuing to monitor all outbound traffic to ensure we don’t see another spike in filtering. If you believe you are still experiencing failures please reach out to our support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation. We’ll provide more information as we continue to monitor.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>06:09</var> PDT</small><br><strong>Update</strong> - We are still seeing higher than normal SMS filtering by the T-Mobile Network for intermittent numbers. Our team is actively engaged with our partners on the cause of these errors and we’ll continue to share updates as we receive them.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>16:03</var> PDT</small><br><strong>Investigating</strong> - We are getting reports of higher than normal SMS filtering by the T-Mobile network for intermittent numbers. We are working with our partners on this for those affected and will provide updates as we find more</p>tag:status.sensehq.com,2005:Incident/171953582023-05-09T13:17:56-07:002023-05-09T13:17:56-07:00Chatbot Conversation Errors<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>13:17</var> PDT</small><br><strong>Resolved</strong> - After monitoring the results of the fix deployed, we are seeing all services function correctly once more. Any end user that may have faced this issue can re-engage the bot to finish any conversation initiated. <br /><br />If you believe end users are still experiencing issues when interacting with conversation bots, please reach out to support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>11:34</var> PDT</small><br><strong>Monitoring</strong> - We have deployed the fix and are seeing all services restore once more. We will continue to review chatbot conversation status and will provide updates as we monitor.<br /><br />If you believe end users are still experiencing issues when interacting with conversation bots, please reach out to support team at https://support.sensehq.co/hc/en-us/requests/new to submit a ticket for further investigation.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>10:31</var> PDT</small><br><strong>Identified</strong> - Our team has found Sense Chatbot conversations are experiencing errors when attempting to be interacted with end users. We have identified the error on the API and are actively working on a fix to be deployed. <br /><br />We will provide updates as we have new information.</p>tag:status.sensehq.com,2005:Incident/171363822023-05-03T22:30:11-07:002023-05-03T22:30:11-07:00Messaging Maintenance<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>22:30</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>14:08</var> PDT</small><br><strong>Scheduled</strong> - We will be performing a scheduled maintenance on our messaging product this evening, May 3rd, at 10pm PST for up to 30 minutes. During this time our messaging product will be unavailable for users. Any messages received during this window will populate once completed.<br /><br />If you have any questions or concerns, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new</p>tag:status.sensehq.com,2005:Incident/168910092023-04-17T08:48:35-07:002023-04-17T08:48:35-07:00Calendars Disconnected from Sense<p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>08:48</var> PDT</small><br><strong>Resolved</strong> - We have confirmed we’re seeing all users able to connect their calendars correctly once more. Any user that was disconnected during this time will need to re-connect directly in Sense once more. For information on how to do so, please see the below support article for assistance. <br /><br />https://support.sensehq.co/hc/en-us/articles/13981726944013-Sense-Scheduling-Setup-Guide#h_01GVG68ND94WTZBNQP2H0YTZR4 <br /><br />If you believe you are still having connection errors when attempting, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new for further review</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>08:01</var> PDT</small><br><strong>Monitoring</strong> - Our team has identified the root cause to the disconnection of MS 0365 calendars, and have updated the configurations on the backend to correct the error when reconnecting. We are seeing calendars connect correctly once more.<br /><br />Any users that were experiencing this will need to reconnect their calendars in the Sense instance directly. For more information on connecting your calendar, please review the below support article on how to do so:<br /><br />https://support.sensehq.co/hc/en-us/articles/13981726944013-Sense-Scheduling-Setup-Guide#h_01GVG68ND94WTZBNQP2H0YTZR4<br /><br />We will continue to monitor this on our end to confirm no other users are seeing issues when reconnecting. If you believe you are still having issues please submit a support ticket to our team at https://support.sensehq.co/hc/en-us/requests/new</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>06:50</var> PDT</small><br><strong>Investigating</strong> - We are receiving reports of intermittent users being disconnected from their calendars within Sense Meetings, and unable to reconnect due to a server error. It appears to be affecting users using Microsoft 365 calendars at this time <br /><br />Our team is actively investigating the root cause of the disconnections and we will provide updates as soon as we learn more information</p>tag:status.sensehq.com,2005:Incident/168108152023-04-10T09:50:08-07:002023-04-10T09:50:08-07:00Messaging Connection Issues<p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>09:50</var> PDT</small><br><strong>Resolved</strong> - After monitoring, we have confirmed all systems are connecting correctly and we’re seeing the product function as expected once more. <br /><br />If you believe you are still experiencing any connection issues, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>09:20</var> PDT</small><br><strong>Monitoring</strong> - Our team has found the root cause and was able to implement a fix to ensure all systems are connecting correctly once more. We did confirm there will be no missed messages within the product, however live updates did not occur during this time. If users are not seeing any new messages, please ensure the user refreshes their browser or extension and all messages will appear. <br /><br />We will continue to monitor for any recurrences and if you believe you are still seeing an issue please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>08:50</var> PDT</small><br><strong>Investigating</strong> - We have received reports about a connection issue popup that is displaying on the Messaging tab within the Sense UI and the Sense Messaging Extension. Our team is actively investigating this and working to find the root cause and overall impact. <br /><br />We will be sure to provide updates as we discover new information on this</p>tag:status.sensehq.com,2005:Incident/167017022023-04-01T22:00:27-07:002023-04-01T22:00:27-07:00Database Maintenance<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>21:30</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>14:17</var> PDT</small><br><strong>Scheduled</strong> - Sense will be performing a quick database maintenance update this weekend from Saturday evening, April 1st at 9:30pm PST. This maintenance should not last any longer than 30 minutes before we see all affected products fully functional once more. This will affect all Sense products during this time, and these will not be available for use. <br /><br />Any scheduled messages during this time will be captured after the downtime is completed. If you have any questions or concerns please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new</p>tag:status.sensehq.com,2005:Incident/167130812023-03-31T15:30:25-07:002023-03-31T15:30:25-07:00Intermittent Runtime Errors in Sense<p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>15:30</var> PDT</small><br><strong>Resolved</strong> - All Sense pages are loading correctly once more and we can confirm we are no longer seeing the error recur. <br /><br />If you believe you are still experiencing this issue please submit a ticket to https://support.sensehq.co/hc/en-us/requests/new for our support team to review, and we’ll provide further updates after review</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>14:45</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>14:44</var> PDT</small><br><strong>Monitoring</strong> - We have updated the configurations affecting this and we are seeing pages loading successful once more. We will continue to monitor after this update to make sure this does not recur.<br /><br />If you believe you are still experiencing this issue please submit a ticket to https://support.sensehq.co/hc/en-us/requests/new for our support team to review, and we’ll provide further updates after review</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>13:30</var> PDT</small><br><strong>Identified</strong> - Our team has identified a configuration update that is causing runtime errors intermittently for some users while navigating through the Sense Platform. <br /><br />We are currently working on updating this and having the product load pages correctly. We'll follow up as we receive more information here.</p>tag:status.sensehq.com,2005:Incident/167016502023-03-30T14:13:08-07:002023-03-30T14:13:08-07:00Status Update Page Error<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>14:13</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p>tag:status.sensehq.com,2005:Incident/166882772023-03-30T09:00:55-07:002023-03-30T09:00:55-07:00Analytics Dashboard Loading Issues<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Resolved</strong> - We have confirmed the bug fix that was deployed did take successfully and we're seeing the affected pages load correctly. We are still spot checking for the remaining data points missed to be updated, but will continue to handle these on a case by case basis. <br /><br />If you believe you are still experiencing this issue please submit a ticket to https://support.sensehq.co/hc/en-us/requests/new for our support team to review.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>08:03</var> PDT</small><br><strong>Monitoring</strong> - Bug fix has been deployed and we are seeing the analytics page load correctly across the platform. There are still minor data points that maybe missing, but our team has identified these and are handling them on a case by case basis. <br /><br />We will continue to monitor this and provide further updates as we receive them. If you believe you are still experiencing this issue please submit a ticket to https://support.sensehq.co/hc/en-us/requests/new for our support team to review</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>05:21</var> PDT</small><br><strong>Update</strong> - Fix is applied to load data points. There are few data points missing which the team has encountered, will continue working. Recommendation is to downgrade the outage to MINOR</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>22:48</var> PDT</small><br><strong>Update</strong> - We have tested the fix for the failing scenario, doing final smoke testing. Going to start the production deployment in some time.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>17:01</var> PDT</small><br><strong>Update</strong> - Our team is actively working to have the bug fix deployed as soon as possible, and continuing to test. We will continue to work overnight and provide more updates in the morning or if they become available before then.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>14:11</var> PDT</small><br><strong>Update</strong> - We are still processing the bug fix to deploy and correct the current issue we’re seeing when attempting to load analytic data on some pages. We will continue to test and confirm when ready to deploy<br /><br />We will provide updates as we receive more information on a resolution</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>11:23</var> PDT</small><br><strong>Update</strong> - Our team is continuing to work on the fix for this issue. Once we have this ready, we'll deploy it and monitor for results<br /><br />We'll continue to provide updates as we receive them</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>08:19</var> PDT</small><br><strong>Identified</strong> - Our team has identified an ongoing issue with loading journey and touch point summary data on the analytics overview dashboard. We have identified the root cause and are working on deploying a fix. <br /><br />We will continue to provide updates as we find new information</p>tag:status.sensehq.com,2005:Incident/166078872023-03-22T11:04:24-07:002023-03-22T11:04:24-07:00Missing Writebacks<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>11:04</var> PDT</small><br><strong>Resolved</strong> - We have confirmed that all writebacks are processing correctly for all ATS partners. Any missed writebacks during this time will be backfilled before the end of the day.<br /><br />If you believe you are still seeing missed writebacks before end of today, please submit a ticket at https://support.sensehq.co/hc/en-us/requests/new and our support team will review</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>10:01</var> PDT</small><br><strong>Monitoring</strong> - Our team confirmed that the fix deployed for Sense API customers worked as expected and we’re seeing all missed writebacks backfill correctly for those that may have experienced this delay. We are seeing most already processed and the remaining in the queue to be completed today. We will continue to monitor the remaining progress to ensure there are no other misses and we’ll provide and findings we have.<br /><br />If you believe you are still seeing missed writebacks before end of today, please submit a ticket at https://support.sensehq.co/hc/en-us/requests/new and our support team will review</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Identified</strong> - Our team discovered Sense API customer’s writebacks stopped sending to their ATSs as early as Tuesday morning at 12pm PST. We have identified the issue and have deployed a fix. Missing writebacks have begun processing and will catch up within the next few hours. We have confirmed there will be no data loss.<br /><br />We will monitor the backfill process and provide updates as systems are caught up.</p>tag:status.sensehq.com,2005:Incident/165126112023-03-15T12:06:00-07:002023-03-15T12:06:00-07:00Intermittent SMS Outbound Delivery Failures - T-Mobile<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:06</var> PDT</small><br><strong>Resolved</strong> - We have confirmed that we are seeing expected traffic to all carriers once more.<br /><br />If you are still experiencing any deliverability issues please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new to investigate further</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>10:09</var> PDT</small><br><strong>Monitoring</strong> - We have received confirmation from our vendor that they are seeing successful traffic to T-Mobile once more for affected numbers.<br />Our team is continuing to monitor to ensure we are seeing the same delivery rates.<br /><br />If you believe you are still experiencing issues please submit a ticket with our support team at https://support.sensehq.co/hc/en-us/requests/new</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>Identified</strong> - We have received an update from one of our providers for SMS services that we are seeing an increased failure rate to T-Mobile recipients for some users. They have confirmed this error did not originate on their servers and are in contact with the downstream carriers to resolve this as soon as possible. <br />We will continue to provide updates as we receive them.</p>tag:status.sensehq.com,2005:Incident/163073362023-03-01T02:37:59-08:002023-03-01T02:37:59-08:00Sense Analytics Not Loading<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>02:37</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>02:35</var> PST</small><br><strong>Update</strong> - Our team has identified the issue and deployed a fix for it. The issue has been resolve now. All the components are working fine now.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>16:05</var> PST</small><br><strong>Update</strong> - Our team is still working to determine the root cause for the analytics page to fail loading. We will be working continuously overnight to narrow down this issue, and provide any information we find</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>14:06</var> PST</small><br><strong>Update</strong> - We are still troubleshooting what maybe causing the error when loading analytics for some users in Sense. Our team is actively working to find the root cause as soon as possible and we’ll continue to provide updates when received.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:11</var> PST</small><br><strong>Update</strong> - Our team is still investigating the root cause for the analytics not loading correctly for some users. We will continue to provide any updates as the information becomes available on this issue.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:10</var> PST</small><br><strong>Investigating</strong> - Our team has confirmed there is an ongoing issue causing our analytics to not load results intermittently for some users. Our team is actively investigating the root cause and we’ll provide updates as we receive them</p>tag:status.sensehq.com,2005:Incident/162762082023-02-25T05:05:08-08:002023-02-25T05:05:08-08:00Database Maintenance<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>05:05</var> PST</small><br><strong>Resolved</strong> - The database maintenance has been completed, The entire system is now operational.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>05:01</var> PST</small><br><strong>Investigating</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p>tag:status.sensehq.com,2005:Incident/161823512023-02-24T22:30:07-08:002023-02-24T23:02:30-08:00Database Optimization<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>22:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>19:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>14:14</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing a database optimization starting on Friday 7:30pm PST and ending by 10:30pm PST.<br /><br />This will result in all products being down during this time, and will resume automatically once completed. We have confirmed that no data or events will be missed during this time.</p>tag:status.sensehq.com,2005:Incident/162092432023-02-20T16:01:10-08:002023-02-20T16:01:10-08:00Sense Platform Increased Loading Times<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>16:01</var> PST</small><br><strong>Resolved</strong> - We have confirmed the fix implemented has been successful and we're seeing stable loading times across the board.<br /><br />If you believe you're still experiencing issues, please submit a ticket at https://support.sensehq.co/hc/en-us/requests/new and our Support team will investigate further"</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>15:11</var> PST</small><br><strong>Monitoring</strong> - Our team has implemented the fix and seeing our servers stabilize across the board. We'll continue to monitor as this takes place and provide updates as we receive them.<br />If you believe you are still seeing issues, please submit a support ticket at https://support.sensehq.co/hc/en-us/requests/new</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>14:00</var> PST</small><br><strong>Update</strong> - Our team is still working to deploy the fix after identifying the block. We'll continue to provide updates as we receive them</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>13:00</var> PST</small><br><strong>Identified</strong> - Our team has identified the block on the backend server and working to deploy a fix that will improve the loading time across the platform. We will provide an update once the fix has been deployed and we will monitor the status.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>12:00</var> PST</small><br><strong>Update</strong> - We are still investigating the root cause of the increased load times. We will continue to provide updates as we receive them</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>11:01</var> PST</small><br><strong>Update</strong> - Our team is continuing to investigate the root cause for the slowness of the platform and working to correct the issue as soon as possible. We do not have an estimated correction time just yet, but will be sure to share this information once we're able to and will provide updates when possible"</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>10:00</var> PST</small><br><strong>Update</strong> - Our team is still investigating the cause for the increased loading times and working to return to the service back to operational. We'll continue to provide updates as we receive them</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>08:51</var> PST</small><br><strong>Investigating</strong> - Sense is experiencing a significant spike in loading times in the portal due to an issue with a backend server. We are actively investigating and will provide an update as soon as possible</p>tag:status.sensehq.com,2005:Incident/160205792023-02-03T15:00:02-08:002023-02-03T15:00:02-08:00Sense Incoming Messaging Delay<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>15:00</var> PST</small><br><strong>Resolved</strong> - We have confirmed the fix implemented has corrected the incoming message delay we were seeing with intermittent users, and our logs are showing the system has been stabilized. We can confirm all incoming messages were delivered and there was no loss of data.<br /><br />If you believe you are still experiencing this issue, or have any questions, please submit a ticket with our Support Team at https://support.sensehq.co/hc/en-us/requests/new</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>14:11</var> PST</small><br><strong>Monitoring</strong> - Our team has implemented a fix to decrease the load on the system and correct the speed of the incoming message display in Sense. We should start seeing messages receive at standard rate, and are monitoring our logs to ensure this does not recur. <br /><br />If you still see this recur, please submit a ticket to our support team at https://support.sensehq.co/hc/en-us/requests/new and we'll investigate further</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>12:31</var> PST</small><br><strong>Identified</strong> - We have been notified of delayed messages being received by Sense users. We have found a strain on the system that is causing these messages to appear later than received. We can confirm all messages are being successfully received in Sense through our downstream carriers and will populate in the UI over time. <br /><br />Our team is working to reduce the strain on the system to have messages come through in real time once more. We'll continue to provide updates as we receive them.</p>