Pulse Software Status - Incident history https://status.pulsesoftware.com Pulse Software Fri, 29 May 2026 01:02:36 +0000 Service outage affecting all platform features Type: Incident Duration: 2 days, 23 hours and 41 minutes Affected Components: Identity Service, Pulse Web Service, Workflow Service, Forms Service May 29, 01:02:36 GMT+0 - Investigating - We are currently investigating an issue causing workflow failures, first observed this morning. Some workflow executions may fail or be delayed during this period. Our engineering team is actively working to identify the root cause. We will post another update within the next 30 minutes, or sooner as new information becomes available. We apologize for the disruption and appreciate your patience. May 29, 02:25:22 GMT+0 - Identified - We have identified the cause of the ongoing service disruption: an expired TLS certificate is causing connection failures across our services. Our engineering team is working to rotate the certificate and restore service. May 29, 02:24:55 GMT+0 - Identified - Escalation update: the workflow failures reported earlier have expanded — we are now seeing impact across all microservices. We are currently experiencing a major service disruption affecting all of our major services. The issue has been identified as related to SSL certificate validation, and our engineering team is actively working on a fix. Customers may experience failures or degraded performance across the platform during this period. May 29, 05:41:48 GMT+0 - Monitoring - A hotfix has been deployed and all services are now restored. Our team is currently validating system health and will be retriggering any failed workflows as soon as possible. We will continue to monitor closely and provide a final update once validation is complete. Thank you for your patience throughout this incident. Jun 1, 00:44:02 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 days, 23 hours and 41 minutes

Affected Components: , , ,

<p><small>May <var data-var='date'> 29</var>, <var data-var='time'>01:02:36</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating an issue causing workflow failures, first observed this morning. Some workflow executions may fail or be delayed during this period. Our engineering team is actively working to identify the root cause. We will post another update within the next 30 minutes, or sooner as new information becomes available. We apologize for the disruption and appreciate your patience..</p> <p><small>May <var data-var='date'> 29</var>, <var data-var='time'>02:25:22</var> GMT+0</small><br><strong>Identified</strong> - We have identified the cause of the ongoing service disruption: an expired TLS certificate is causing connection failures across our services. Our engineering team is working to rotate the certificate and restore service..</p> <p><small>May <var data-var='date'> 29</var>, <var data-var='time'>02:24:55</var> GMT+0</small><br><strong>Identified</strong> - Escalation update: the workflow failures reported earlier have expanded — we are now seeing impact across all microservices. We are currently experiencing a major service disruption affecting all of our major services. The issue has been identified as related to SSL certificate validation, and our engineering team is actively working on a fix. Customers may experience failures or degraded performance across the platform during this period. .</p> <p><small>May <var data-var='date'> 29</var>, <var data-var='time'>05:41:48</var> GMT+0</small><br><strong>Monitoring</strong> - A hotfix has been deployed and all services are now restored. Our team is currently validating system health and will be retriggering any failed workflows as soon as possible. We will continue to monitor closely and provide a final update once validation is complete. Thank you for your patience throughout this incident..</p> <p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>00:44:02</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 29 May 2026 01:02:36 +0000 https://status.pulsesoftware.com/incident/cmpq7wdjg0080qooycteno18e https://status.pulsesoftware.com/incident/cmpq7wdjg0080qooycteno18e Service Disruption — All Sites Currently Unavailable Type: Incident Duration: 7 minutes Affected Components: Forms Service, Email Service, File Service, Public Job Sites, Identity Service, Workflow Service, Pulse Web Service May 1, 05:07:55 GMT+0 - Investigating - We are currently experiencing a service disruption affecting all our sites. Our team is actively investigating the issue and working to restore service as quickly as possible. We will provide an update shortly. We sincerely apologize for the inconvenience and appreciate your patience. May 1, 05:14:31 GMT+0 - Resolved - Earlier today, all our sites became temporarily unavailable due to a faulty infrastructure resource that impacted some of our core services. The issue has been resolved and all sites are now operating normally. **Impact** During the affected window, customers were unable to access our sites as core services were disrupted by the faulty infrastructure resource. **Root Cause** One of our infrastructure resources became faulty, which impacted several core services that our sites depend on, resulting in the outage. **Resolution** Our team identified the faulty resource and remediated it as quickly as possible. Core services were restored and all sites are back online and operating normally. **Next Steps** We are conducting a full review of the faulty resource and the dependent services to understand why this caused a site-wide impact and will implement safeguards to improve resilience and prevent recurrence. We sincerely apologize for the disruption and appreciate your patience while we worked to restore service. Type: Incident

Duration: 7 minutes

Affected Components: , , , , , ,

<p><small>May <var data-var='date'> 1</var>, <var data-var='time'>05:07:55</var> GMT+0</small><br><strong>Investigating</strong> - We are currently experiencing a service disruption affecting all our sites. Our team is actively investigating the issue and working to restore service as quickly as possible. We will provide an update shortly. We sincerely apologize for the inconvenience and appreciate your patience..</p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>05:14:31</var> GMT+0</small><br><strong>Resolved</strong> - Earlier today, all our sites became temporarily unavailable due to a faulty infrastructure resource that impacted some of our core services. The issue has been resolved and all sites are now operating normally. **Impact** During the affected window, customers were unable to access our sites as core services were disrupted by the faulty infrastructure resource. **Root Cause** One of our infrastructure resources became faulty, which impacted several core services that our sites depend on, resulting in the outage. **Resolution** Our team identified the faulty resource and remediated it as quickly as possible. Core services were restored and all sites are back online and operating normally. **Next Steps** We are conducting a full review of the faulty resource and the dependent services to understand why this caused a site-wide impact and will implement safeguards to improve resilience and prevent recurrence. We sincerely apologize for the disruption and appreciate your patience while we worked to restore service. .</p> ]]> Fri, 1 May 2026 05:07:55 +0000 https://status.pulsesoftware.com/incident/cmomgc0m701yjt6ism0b3xzet https://status.pulsesoftware.com/incident/cmomgc0m701yjt6ism0b3xzet Delayed Notifications (Today, 10:00 AM – 12:00 PM) Type: Incident Duration: 53 minutes Affected Components: Email Service Apr 29, 06:17:46 GMT+0 - Investigating - We recently identified and investigating an issue that prevented some notifications from being triggered between 10:00 AM and 12:00 PM 29th April 2026 Apr 29, 07:09:35 GMT+0 - Identified - Due to heavy load on our email service, notifications queued up and were not delivered as expected during this window Apr 29, 07:11:14 GMT+0 - Resolved - We added additional capacity to our email service, which cleared the backlog instantly. Notifications resumed triggering normally. We are also adding monitoring and autoscaling improvements to prevent this from happening again. No action is required on your part We sincerely apologize for any inconvenience this may have caused. If you have any concerns, please reach out to our support team. Type: Incident

Duration: 53 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>06:17:46</var> GMT+0</small><br><strong>Investigating</strong> - We recently identified and investigating an issue that prevented some notifications from being triggered between 10:00 AM and 12:00 PM 29th April 2026.</p> <p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>07:09:35</var> GMT+0</small><br><strong>Identified</strong> - Due to heavy load on our email service, notifications queued up and were not delivered as expected during this window.</p> <p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>07:11:14</var> GMT+0</small><br><strong>Resolved</strong> - We added additional capacity to our email service, which cleared the backlog instantly. Notifications resumed triggering normally. We are also adding monitoring and autoscaling improvements to prevent this from happening again. No action is required on your part We sincerely apologize for any inconvenience this may have caused. If you have any concerns, please reach out to our support team..</p> ]]> Wed, 29 Apr 2026 06:17:46 +0000 https://status.pulsesoftware.com/incident/cmojny59i07t1tpk7i3alg36s https://status.pulsesoftware.com/incident/cmojny59i07t1tpk7i3alg36s Delayed Email Notifications (April 23–24) Type: Incident Duration: 24 days Affected Components: Email Service Apr 23, 03:00:00 GMT+0 - Identified - We recently identified and resolved an issue that prevented some email notifications from being delivered between **1:00 PM on April 23rd and 11:00 AM on April 24th** Apr 24, 01:00:00 GMT+0 - Resolved - **What happened** A configuration issue in our email delivery system caused a subset of notifications to be rejected rather than delivered during 1:00 PM on April 23rd and 11:00 AM on April 24th. **What we've done** Our team identified the root cause and restored normal delivery on April 24th at 11:00 AM. Email notifications are now operating as expected. **What's next** We are in the process of retriggering the notifications that were missed during **1:00 PM on April 23rd and 11:00 AM on April 24th**, so you should receive any pending notifications shortly. No action is required on your part. We sincerely apologise for any inconvenience this may have caused and appreciate your patience while we worked to resolve it. If you have any concerns, please reach out to our support team. Type: Incident

Duration: 24 days

Affected Components:

<p><small>Apr <var data-var='date'> 23</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Identified</strong> - We recently identified and resolved an issue that prevented some email notifications from being delivered between **1:00 PM on April 23rd and 11:00 AM on April 24th**.</p> <p><small>Apr <var data-var='date'> 24</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Resolved</strong> - **What happened** A configuration issue in our email delivery system caused a subset of notifications to be rejected rather than delivered during 1:00 PM on April 23rd and 11:00 AM on April 24th. **What we've done** Our team identified the root cause and restored normal delivery on April 24th at 11:00 AM. Email notifications are now operating as expected. **What's next** We are in the process of retriggering the notifications that were missed during **1:00 PM on April 23rd and 11:00 AM on April 24th**, so you should receive any pending notifications shortly. No action is required on your part. We sincerely apologise for any inconvenience this may have caused and appreciate your patience while we worked to resolve it. If you have any concerns, please reach out to our support team..</p> ]]> Thu, 23 Apr 2026 03:00:00 +0000 https://status.pulsesoftware.com/incident/cmocixtbb001fwy1a218wi44d https://status.pulsesoftware.com/incident/cmocixtbb001fwy1a218wi44d Production Sites Experiencing Timeouts Type: Incident Duration: 15 hours Affected Components: Forms Service, File Service, Public Job Sites, Workflow Service, Pulse Web Service Mar 10, 00:20:00 GMT+0 - Investigating - Multiple production environments are currently experiencing HTTP timeouts when accessing the application. Users are unable to load key pages, and the sites fail with a generic timeout screen. Mar 10, 00:30:00 GMT+0 - Identified - * Engineering team notified and investigating root cause. * Checking: * Application service health across environments * API gateway / load balancer logs * Database performance and connection saturation * Recent deployments or infra changes Mar 10, 00:35:00 GMT+0 - Resolved - A misbehaving Redis cluster caused requests to stall, leading to degraded performance across affected site. The Redis cluster was flushed and restored to a healthy state. All systems have now returned to normal operation and performance has stabilised. Type: Incident

Duration: 15 hours

Affected Components: , , , ,

<p><small>Mar <var data-var='date'> 10</var>, <var data-var='time'>00:20:00</var> GMT+0</small><br><strong>Investigating</strong> - Multiple production environments are currently experiencing HTTP timeouts when accessing the application. Users are unable to load key pages, and the sites fail with a generic timeout screen. .</p> <p><small>Mar <var data-var='date'> 10</var>, <var data-var='time'>00:30:00</var> GMT+0</small><br><strong>Identified</strong> - * Engineering team notified and investigating root cause. * Checking: * Application service health across environments * API gateway / load balancer logs * Database performance and connection saturation * Recent deployments or infra changes.</p> <p><small>Mar <var data-var='date'> 10</var>, <var data-var='time'>00:35:00</var> GMT+0</small><br><strong>Resolved</strong> - A misbehaving Redis cluster caused requests to stall, leading to degraded performance across affected site. The Redis cluster was flushed and restored to a healthy state. All systems have now returned to normal operation and performance has stabilised. .</p> ]]> Tue, 10 Mar 2026 00:20:00 +0000 https://status.pulsesoftware.com/incident/cmmk03fzz08yfs8507mx92iuc https://status.pulsesoftware.com/incident/cmmk03fzz08yfs8507mx92iuc Notifications were failing for some customers Type: Incident Duration: 3 days, 7 hours and 57 minutes Affected Components: Email Service Feb 19, 16:00:00 GMT+0 - Investigating - We have identified an issue where notifications were failing to send for a subset of customers. Our team is currently investigating the root cause Feb 22, 23:56:57 GMT+0 - Resolved - The incident has been fully resolved. The faulty instance causing the incorrect parameter handling has been removed, and services are operating normally. As a preventive measure, we are implementing improved automated checks that will detect and evict such unhealthy instances in the future, ensuring greater reliability and consistency across the notification pipeline Type: Incident

Duration: 3 days, 7 hours and 57 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We have identified an issue where notifications were failing to send for a subset of customers. Our team is currently investigating the root cause.</p> <p><small>Feb <var data-var='date'> 22</var>, <var data-var='time'>23:56:57</var> GMT+0</small><br><strong>Resolved</strong> - The incident has been fully resolved. The faulty instance causing the incorrect parameter handling has been removed, and services are operating normally. As a preventive measure, we are implementing improved automated checks that will detect and evict such unhealthy instances in the future, ensuring greater reliability and consistency across the notification pipeline.</p> ]]> Thu, 19 Feb 2026 16:00:00 +0000 https://status.pulsesoftware.com/incident/cmlyfthes02u2m46wyi2b4xyn https://status.pulsesoftware.com/incident/cmlyfthes02u2m46wyi2b4xyn Major disruption to service Type: Incident Duration: 2 hours and 5 minutes Affected Components: Email Service, File Service, Public Job Sites, Identity Service, Workflow Service, Pulse Web Service, Forms Service Feb 9, 10:00:00 GMT+0 - Investigating - A critical response team has been setup and we are investiating the isssue Feb 9, 12:05:00 GMT+0 - Resolved - A denial of service attack has been mitigated Type: Incident

Duration: 2 hours and 5 minutes

Affected Components: , , , , , ,

<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:00:00</var> GMT+0</small><br><strong>Investigating</strong> - A critical response team has been setup and we are investiating the isssue.</p> <p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>12:05:00</var> GMT+0</small><br><strong>Resolved</strong> - A denial of service attack has been mitigated.</p> ]]> Mon, 9 Feb 2026 10:00:00 +0000 https://status.pulsesoftware.com/incident/cmlf3hrvx08jb10n5u5garg13 https://status.pulsesoftware.com/incident/cmlf3hrvx08jb10n5u5garg13 Forms service is down Type: Incident Duration: 18 minutes Affected Components: Forms Service Feb 9, 06:08:43 GMT+0 - Identified - We have identified the issue as a disk issue on one of our database services. We are adding additional resources to mitigate the issue. Feb 9, 06:27:02 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 18 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>06:08:43</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue as a disk issue on one of our database services. We are adding additional resources to mitigate the issue..</p> <p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>06:27:02</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 9 Feb 2026 06:08:43 +0000 https://status.pulsesoftware.com/incident/cmleru6b60qov11tv3vv9w1za https://status.pulsesoftware.com/incident/cmleru6b60qov11tv3vv9w1za Advisory: Unintended Emails Sent During Internal Testing Type: Incident Duration: 6 hours and 45 minutes Feb 8, 22:00:00 GMT+0 - Monitoring - Over the weekend, while our team was running internal security testing, emails were unintentionally sent from a non‑production system. As a result, customers may have received multiple unexpected emails. We identified the issue today, stopped it immediately, and confirmed that our live services and customer data were not affected. We understand that receiving repeated, unexpected emails can be frustrating or concerning, and we’re genuinely sorry for the disruption this caused. Our team is reinforcing safeguards between testing and customer communications to ensure this doesn’t happen again. Thank you for your patience and understanding Feb 9, 04:44:38 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 6 hours and 45 minutes

<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Monitoring</strong> - Over the weekend, while our team was running internal security testing, emails were unintentionally sent from a non‑production system. As a result, customers may have received multiple unexpected emails. We identified the issue today, stopped it immediately, and confirmed that our live services and customer data were not affected. We understand that receiving repeated, unexpected emails can be frustrating or concerning, and we’re genuinely sorry for the disruption this caused. Our team is reinforcing safeguards between testing and customer communications to ensure this doesn’t happen again. Thank you for your patience and understanding.</p> <p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>04:44:38</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sun, 8 Feb 2026 22:00:00 +0000 https://status.pulsesoftware.com/incident/cmleqgztj0lrg5q0gktsnd1ff https://status.pulsesoftware.com/incident/cmleqgztj0lrg5q0gktsnd1ff Degreaded system performance Type: Incident Duration: 52 minutes Affected Components: Pulse Web Service Jan 13, 22:00:00 GMT+0 - Investigating - We are currently investigating this incident. Jan 13, 22:52:00 GMT+0 - Resolved - The issues has not been resolved Jan 14, 04:26:51 GMT+0 - Postmortem - At approximately 8:00 AM, degraded performance in the production system was detected by our alerting system and reported by several customers. A critical response team was immediately assembled to investigate the issue. The initial investigation identified the in-memory cache service as the primary bottleneck, as it was operating beyond expected capacity. The underlying cause of the increased load was not immediately clear. To mitigate customer impact and allow additional time for analysis, the decision was made to scale up the cache service. The scale-up alleviated the bottleneck almost immediately, and system performance returned to normal operating levels. As a follow-up action, we will integrate the in-memory cache service into our APM observability platform to improve visibility, monitoring, and early detection of similar issues in the future. Type: Incident

Duration: 52 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 13</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jan <var data-var='date'> 13</var>, <var data-var='time'>22:52:00</var> GMT+0</small><br><strong>Resolved</strong> - The issues has not been resolved.</p> <p><small>Jan <var data-var='date'> 14</var>, <var data-var='time'>04:26:51</var> GMT+0</small><br><strong>Postmortem</strong> - At approximately 8:00 AM, degraded performance in the production system was detected by our alerting system and reported by several customers. A critical response team was immediately assembled to investigate the issue. The initial investigation identified the in-memory cache service as the primary bottleneck, as it was operating beyond expected capacity. The underlying cause of the increased load was not immediately clear. To mitigate customer impact and allow additional time for analysis, the decision was made to scale up the cache service. The scale-up alleviated the bottleneck almost immediately, and system performance returned to normal operating levels. As a follow-up action, we will integrate the in-memory cache service into our APM observability platform to improve visibility, monitoring, and early detection of similar issues in the future..</p> ]]> Tue, 13 Jan 2026 22:00:00 +0000 https://status.pulsesoftware.com/incident/cmkdib4xg02dr405kqe21yujo https://status.pulsesoftware.com/incident/cmkdib4xg02dr405kqe21yujo An issue has been reported loading workflow screens Type: Incident Duration: 6 hours and 47 minutes Affected Components: Workflow Service Dec 10, 16:00:00 GMT+0 - Investigating - We are currently investigating this incident. Dec 10, 22:47:00 GMT+0 - Resolved - This incident has been resolved. Post moretom to follow. Type: Incident

Duration: 6 hours and 47 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>22:47:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Post moretom to follow..</p> ]]> Wed, 10 Dec 2025 16:00:00 +0000 https://status.pulsesoftware.com/incident/cmj0mu0jz02fbzs4famfscw1e https://status.pulsesoftware.com/incident/cmj0mu0jz02fbzs4famfscw1e Cloudflare outage impacting Pulse Software services Type: Incident Affected Components: Email Service, File Service, Public Job Sites, Identity Service, Workflow Service, Pulse Web Service, Forms Service Nov 18, 14:40:00 GMT+0 - Resolved - This incident has been resolved. Nov 18, 12:30:00 GMT+0 - Resolved - This incident has been resolved. Nov 18, 20:16:32 GMT+0 - Postmortem - Pulse Software relies on Cloudflare to enhance the performance, security, and reliability of its infrastructure. Cloudflare is a leading provider in this space, powering an estimated 20% of global internet traffic. On the date of the incident, a major Cloudflare outage disrupted network connectivity to Pulse Software’s systems, resulting in a period of service unavailability. Details of the upstream outage can be found in Cloudflare’s incident report: <https://www.cloudflarestatus.com/incidents/8gmgl950y3h7> Type: Incident

Affected Components: , , , , , ,

<p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>14:40:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>12:30:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>20:16:32</var> GMT+0</small><br><strong>Postmortem</strong> - Pulse Software relies on Cloudflare to enhance the performance, security, and reliability of its infrastructure. Cloudflare is a leading provider in this space, powering an estimated 20% of global internet traffic. On the date of the incident, a major Cloudflare outage disrupted network connectivity to Pulse Software’s systems, resulting in a period of service unavailability. Details of the upstream outage can be found in Cloudflare’s incident report: <https://www.cloudflarestatus.com/incidents/8gmgl950y3h7>.</p> ]]> Tue, 18 Nov 2025 12:30:00 +0000 https://status.pulsesoftware.com/incident/cmi50ghek00zap72nz8b7h2yk https://status.pulsesoftware.com/incident/cmi50ghek00zap72nz8b7h2yk Job listing page not always loading correctly Type: Incident Duration: 8 minutes Affected Components: Public Job Sites Nov 9, 22:14:41 GMT+0 - Investigating - We are currently investigating this incident. Nov 9, 22:22:49 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 8 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>22:14:41</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>22:22:49</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sun, 9 Nov 2025 22:14:41 +0000 https://status.pulsesoftware.com/incident/cmhs9u1uk01wtwnvwm57m9qka https://status.pulsesoftware.com/incident/cmhs9u1uk01wtwnvwm57m9qka An issue with Forms was detected by automated alarms and confirmed by support staff Type: Incident Duration: 10 hours and 44 minutes Affected Components: Forms Service Sep 28, 09:35:00 GMT+0 - Investigating - We are currently investigating this incident. Sep 28, 20:19:00 GMT+0 - Resolved - This incident has been resolved. Sep 29, 02:26:49 GMT+0 - Postmortem - ### **Incident Summary** During a scheduled Azure update, the Forms service was automatically restarted. Upon restarting, the service failed to establish a connection with the database. As a result, the Forms service remained in a failed state until it was manually restarted and successfully reconnected to the database. ### **Root Cause** The outage was triggered by the Forms service being unable to connect to the database immediately after the Azure-driven restart. The lack of sufficient database connection resilience mechanisms caused the service to fail rather than recover automatically. ### **Resolution** The service was restored by performing an additional restart, at which point it successfully re-established the database connection and resumed normal operations. ### **Preventative Actions** To improve resilience and prevent recurrence, the following work items have been defined: * **Enhanced database connection resilience**: Implement improvements to ensure the Forms service can recover from temporary database connectivity issues. * **Health check enhancements:** Update the Forms service health check to fail explicitly when the database connection is not healthy, enabling faster detection and automated recovery. Type: Incident

Duration: 10 hours and 44 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 28</var>, <var data-var='time'>09:35:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 28</var>, <var data-var='time'>20:19:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>02:26:49</var> GMT+0</small><br><strong>Postmortem</strong> - ### **Incident Summary** During a scheduled Azure update, the Forms service was automatically restarted. Upon restarting, the service failed to establish a connection with the database. As a result, the Forms service remained in a failed state until it was manually restarted and successfully reconnected to the database. ### **Root Cause** The outage was triggered by the Forms service being unable to connect to the database immediately after the Azure-driven restart. The lack of sufficient database connection resilience mechanisms caused the service to fail rather than recover automatically. ### **Resolution** The service was restored by performing an additional restart, at which point it successfully re-established the database connection and resumed normal operations. ### **Preventative Actions** To improve resilience and prevent recurrence, the following work items have been defined: * **Enhanced database connection resilience**: Implement improvements to ensure the Forms service can recover from temporary database connectivity issues. * **Health check enhancements:** Update the Forms service health check to fail explicitly when the database connection is not healthy, enabling faster detection and automated recovery..</p> ]]> Sun, 28 Sep 2025 09:35:00 +0000 https://status.pulsesoftware.com/incident/cmg4brpsr054ylegcl565kuvr https://status.pulsesoftware.com/incident/cmg4brpsr054ylegcl565kuvr Emails are currently delayed Type: Incident Duration: 43 minutes Affected Components: Email Service Sep 24, 04:46:29 GMT+0 - Investigating - We are currently investigating this incident. Sep 24, 05:29:41 GMT+0 - Resolved - This incident has been resolved. The email queue is up to date. Sep 25, 07:40:27 GMT+0 - Postmortem - **Incident Summary** On 24 Septmber, a significant spike in email volume caused delivery delays of up to 30 minutes. The root cause was the current email service reaching its throughput limits under increased load. **Root Cause** The existing architecture was not designed to scale dynamically to handle large, sudden increases in outbound email traffic. As a result, the system became a bottleneck during high-volume events. **Remediation & Next Steps** The email service is currently undergoing a major re-architecture focused on scalability and performance. The new system is expected to support an order of magnitude higher throughput than the current implementation. We are targeting a full cutover to the new service in October 2025. **Status** Mitigation measures have been applied to reduce the likelihood of recurrence before the new architecture is deployed. Monitoring and alerting thresholds have also been adjusted to provide earlier detection of delays. Type: Incident

Duration: 43 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>04:46:29</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>05:29:41</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. The email queue is up to date..</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>07:40:27</var> GMT+0</small><br><strong>Postmortem</strong> - **Incident Summary** On 24 Septmber, a significant spike in email volume caused delivery delays of up to 30 minutes. The root cause was the current email service reaching its throughput limits under increased load. **Root Cause** The existing architecture was not designed to scale dynamically to handle large, sudden increases in outbound email traffic. As a result, the system became a bottleneck during high-volume events. **Remediation & Next Steps** The email service is currently undergoing a major re-architecture focused on scalability and performance. The new system is expected to support an order of magnitude higher throughput than the current implementation. We are targeting a full cutover to the new service in October 2025. **Status** Mitigation measures have been applied to reduce the likelihood of recurrence before the new architecture is deployed. Monitoring and alerting thresholds have also been adjusted to provide earlier detection of delays..</p> ]]> Wed, 24 Sep 2025 04:46:29 +0000 https://status.pulsesoftware.com/incident/cmfxi4vnv00f011v5k2vafush https://status.pulsesoftware.com/incident/cmfxi4vnv00f011v5k2vafush Some intermittant timeouts being reported Type: Incident Duration: 5 hours and 11 minutes Affected Components: Pulse Web Service, Forms Service, Workflow Service, File Service, Identity Service, Public Job Sites, Email Service Sep 23, 06:00:50 GMT+0 - Investigating - We are currently investigating this incident. Sep 23, 11:11:21 GMT+0 - Resolved - This incident has been resolved. Sep 23, 06:02:30 GMT+0 - Identified - We have identified the issue is related to imports. A fix will be deployed tonight. Sep 23, 10:51:59 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Sep 24, 00:02:09 GMT+0 - Postmortem - ### Incident Summary A Microsoft component responsible for extracting data from Excel began causing application pool crashes. When an application pool crashes, it impacts all users connected to that pool. Each crash resulted in temporary request timeouts for those users until the pool automatically recovered, typically within seconds. As a result, the issue only persisted for less than a minute and only affected users tied to the impacted pool. ### Impact * Users connected to the affected app pool experienced brief timeouts (under one minute). * No lasting or widespread outages occurred, as app pools automatically restarted and recovered within seconds. ### Root Cause The failure was traced to a Microsoft Excel data extraction component, which triggered repeated crashes in the application pool. ### Resolution * All servers were patched and verified on the morning following the incident. * A longer-term remediation has been expedited: migrating the Excel data extraction functionality to a dedicated microservice. This change will reduce dependency on the main web server and improve overall system resilience. ### Next Steps * Continue monitoring the updated servers for stability. * Complete and deploy the microservice migration to fully eliminate reliance on the problematic component. * Review alerting and recovery procedures to ensure faster detection and mitigation in future pool crashes. Type: Incident

Duration: 5 hours and 11 minutes

Affected Components: , , , , , ,

<p><small>Sep <var data-var='date'> 23</var>, <var data-var='time'>06:00:50</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 23</var>, <var data-var='time'>11:11:21</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Sep <var data-var='date'> 23</var>, <var data-var='time'>06:02:30</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue is related to imports. A fix will be deployed tonight..</p> <p><small>Sep <var data-var='date'> 23</var>, <var data-var='time'>10:51:59</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>00:02:09</var> GMT+0</small><br><strong>Postmortem</strong> - ### Incident Summary A Microsoft component responsible for extracting data from Excel began causing application pool crashes. When an application pool crashes, it impacts all users connected to that pool. Each crash resulted in temporary request timeouts for those users until the pool automatically recovered, typically within seconds. As a result, the issue only persisted for less than a minute and only affected users tied to the impacted pool. ### Impact * Users connected to the affected app pool experienced brief timeouts (under one minute). * No lasting or widespread outages occurred, as app pools automatically restarted and recovered within seconds. ### Root Cause The failure was traced to a Microsoft Excel data extraction component, which triggered repeated crashes in the application pool. ### Resolution * All servers were patched and verified on the morning following the incident. * A longer-term remediation has been expedited: migrating the Excel data extraction functionality to a dedicated microservice. This change will reduce dependency on the main web server and improve overall system resilience. ### Next Steps * Continue monitoring the updated servers for stability. * Complete and deploy the microservice migration to fully eliminate reliance on the problematic component. * Review alerting and recovery procedures to ensure faster detection and mitigation in future pool crashes..</p> ]]> Tue, 23 Sep 2025 06:00:50 +0000 https://status.pulsesoftware.com/incident/cmfw5cmv400bw11noy5fq1lkw https://status.pulsesoftware.com/incident/cmfw5cmv400bw11noy5fq1lkw An issue has been reported with Job listings and onboarding across all customers Type: Incident Duration: 3 hours and 16 minutes Affected Components: Pulse Web Service, Forms Service, Workflow Service Aug 2, 20:01:00 GMT+0 - Investigating - We are currently investigating this incident. Aug 2, 23:17:00 GMT+0 - Resolved - This incident has been resolved. Aug 7, 04:28:40 GMT+0 - Postmortem - A network issue between Pulse Web Services and Forms resulted in job listings and the offer process to fail. Restarting some of the impacted services resolved the issue. An additional alarm will be created to pickup so that the on call team can respond and rectify the issue in a shorter timeframe. Type: Incident

Duration: 3 hours and 16 minutes

Affected Components: , ,

<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>20:01:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>23:17:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>04:28:40</var> GMT+0</small><br><strong>Postmortem</strong> - A network issue between Pulse Web Services and Forms resulted in job listings and the offer process to fail. Restarting some of the impacted services resolved the issue. An additional alarm will be created to pickup so that the on call team can respond and rectify the issue in a shorter timeframe..</p> ]]> Sat, 2 Aug 2025 20:01:00 +0000 https://status.pulsesoftware.com/incident/cme0wa3dg00aw48pdf12mumcb https://status.pulsesoftware.com/incident/cme0wa3dg00aw48pdf12mumcb Major outage Type: Incident Duration: 14 minutes Affected Components: Pulse Web Service, Forms Service, Workflow Service, Email Service, Identity Service, Public Job Sites Jul 17, 03:59:00 GMT+0 - Resolved - This incident has been resolved. Jul 17, 03:45:00 GMT+0 - Investigating - A major outage impacting all services Aug 4, 22:00:49 GMT+0 - Postmortem - | **Date and Time of Incident** | 2025-07-17 13:45 | | ----------------------------- | ------------------------------ | | **Incident Type** | Major Outage | | **Reported By** | Reported Internally | | **Location/System Affected** | All Australian Sites/Customers | | **Prepared By** | Site Reliability Engineer | | **Acknowledged By** | | ## Description of the Incident On 17 June 2025 at approximately 13:43 AEST, a sudden spike in resource consumption was detected across our infrastructure. This surge posed a potential performance risk to all customers. In response, our team initiated standard mitigation procedures to stabilize the environment and maintain service quality. During this process, an engineer made a configuration error while addressing an overloaded server. This error inadvertently caused the platform to go offline for a duration of 16 minutes. ## Root Cause Analysis The incident was caused by a manual configuration process that should have been automated. The lack of automation introduced the possibility of human error, which ultimately led to the misconfiguration and temporary service disruption. ## Timeline of Event · 13:43 – Overloaded server identified · 13:45 – Configuration error occurred · 14:00 – Alternative Server Cluster configured · 14:01 – Platform restored, and services resumed ## Post incident review Following the incident, a comprehensive review was conducted. Key findings and actions include: · Automation Improvements: Plans are underway to automate the configuration process to eliminate manual intervention and reduce the risk of human error · Monitoring Enhancements: Resource monitoring tools will be refined to provide earlier alerts and more granular diagnostics Type: Incident

Duration: 14 minutes

Affected Components: , , , , ,

<p><small>Jul <var data-var='date'> 17</var>, <var data-var='time'>03:59:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 17</var>, <var data-var='time'>03:45:00</var> GMT+0</small><br><strong>Investigating</strong> - A major outage impacting all services.</p> <p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>22:00:49</var> GMT+0</small><br><strong>Postmortem</strong> - | **Date and Time of Incident** | 2025-07-17 13:45 | | ----------------------------- | ------------------------------ | | **Incident Type** | Major Outage | | **Reported By** | Reported Internally | | **Location/System Affected** | All Australian Sites/Customers | | **Prepared By** | Site Reliability Engineer | | **Acknowledged By** | | ## Description of the Incident On 17 June 2025 at approximately 13:43 AEST, a sudden spike in resource consumption was detected across our infrastructure. This surge posed a potential performance risk to all customers. In response, our team initiated standard mitigation procedures to stabilize the environment and maintain service quality. During this process, an engineer made a configuration error while addressing an overloaded server. This error inadvertently caused the platform to go offline for a duration of 16 minutes. ## Root Cause Analysis The incident was caused by a manual configuration process that should have been automated. The lack of automation introduced the possibility of human error, which ultimately led to the misconfiguration and temporary service disruption. ## Timeline of Event · 13:43 – Overloaded server identified · 13:45 – Configuration error occurred · 14:00 – Alternative Server Cluster configured · 14:01 – Platform restored, and services resumed ## Post incident review Following the incident, a comprehensive review was conducted. Key findings and actions include: · Automation Improvements: Plans are underway to automate the configuration process to eliminate manual intervention and reduce the risk of human error · Monitoring Enhancements: Resource monitoring tools will be refined to provide earlier alerts and more granular diagnostics.</p> ]]> Thu, 17 Jul 2025 03:45:00 +0000 https://status.pulsesoftware.com/incident/cmdxx16ts0099azc5oi49onmq https://status.pulsesoftware.com/incident/cmdxx16ts0099azc5oi49onmq Network issues (524) were reported by customers Type: Incident Duration: 3 minutes Affected Components: Pulse Web Service Jul 16, 07:10:00 GMT+0 - Resolved - The service has automatically marked a node as unhealthy and recovered. Jul 16, 07:07:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 3 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>07:10:00</var> GMT+0</small><br><strong>Resolved</strong> - The service has automatically marked a node as unhealthy and recovered..</p> <p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>07:07:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Wed, 16 Jul 2025 07:07:00 +0000 https://status.pulsesoftware.com/incident/cme0w4maa00jf7voc6atqpbkq https://status.pulsesoftware.com/incident/cme0w4maa00jf7voc6atqpbkq Major outage across all services except notifications Type: Incident Duration: 1 hour and 5 minutes Affected Components: Pulse Web Service, Forms Service, Workflow Service, File Service, Identity Service, Public Job Sites Jun 12, 04:50:00 GMT+0 - Investigating - We are currently investigating this incident. Jun 12, 05:55:00 GMT+0 - Resolved - A service node crashed and failed to automatically recover. Engineers had to manually recover the node. Aug 7, 04:18:24 GMT+0 - Postmortem - ## Description of the Incident On 12th June 2025, a major outage of the Pulse Software system was reported. This impacted all services and all customers. The critical response process was followed, this including setting up a critical response team and sending communications internally. It was found that our microservice infrastructure had failed and did not automatically recover. The critical response team intervened adding additional infrastructure and removing the unhealthy infrastructure. This is the first time we have had a complete failure on our microservices infrastructure. The outage lasted for a total of 65 minutes. Over the last six months the uptime for Pulse Software Platform has been exceptional at 99.98% with no maintenance windows. After this event this has dropped to 99.96% after the event. This is well above 99.5% SLA. ## Root Cause Analysis ### Technical Cause: The incident was traced to a very high peak workload impacting the change log auditing service. This service was overwhelmed and crashed the service infrastructure preventing users from accessing the system. This is the first time the entire service has crashed at Pulse Software. ## Timeline of Event * **13 June 2025** * 14:50: Issue was reported internally within Pulse * 14:50: Critical response team setup * 14:53: A series of uptime alerts were triggered * 14:53: Critical response team found the Microservice infrastructure was in an unhealthy state * 14:54: Communications were sent internally of a major service disruption impacting all services and all customers * 15:05: Redirected all sites to maintenance mode * 15:10: Attempts to recover the infrastructure failed * 15:11: Started to setup new infrastructure to support microservices * 15:55 All services returned to normal operations ### ### Post incident review · Add additional limits to the services that will protect the infrastructure from failing completely · Expedite the micro service scaling and resilience project. Include into 2025 Q3 plan · Change Log Audit export requires further analysis and performance optimisation Type: Incident

Duration: 1 hour and 5 minutes

Affected Components: , , , , ,

<p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>04:50:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>05:55:00</var> GMT+0</small><br><strong>Resolved</strong> - A service node crashed and failed to automatically recover. Engineers had to manually recover the node..</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>04:18:24</var> GMT+0</small><br><strong>Postmortem</strong> - ## Description of the Incident On 12th June 2025, a major outage of the Pulse Software system was reported. This impacted all services and all customers. The critical response process was followed, this including setting up a critical response team and sending communications internally. It was found that our microservice infrastructure had failed and did not automatically recover. The critical response team intervened adding additional infrastructure and removing the unhealthy infrastructure. This is the first time we have had a complete failure on our microservices infrastructure. The outage lasted for a total of 65 minutes. Over the last six months the uptime for Pulse Software Platform has been exceptional at 99.98% with no maintenance windows. After this event this has dropped to 99.96% after the event. This is well above 99.5% SLA. ## Root Cause Analysis ### Technical Cause: The incident was traced to a very high peak workload impacting the change log auditing service. This service was overwhelmed and crashed the service infrastructure preventing users from accessing the system. This is the first time the entire service has crashed at Pulse Software. ## Timeline of Event * **13 June 2025** * 14:50: Issue was reported internally within Pulse * 14:50: Critical response team setup * 14:53: A series of uptime alerts were triggered * 14:53: Critical response team found the Microservice infrastructure was in an unhealthy state * 14:54: Communications were sent internally of a major service disruption impacting all services and all customers * 15:05: Redirected all sites to maintenance mode * 15:10: Attempts to recover the infrastructure failed * 15:11: Started to setup new infrastructure to support microservices * 15:55 All services returned to normal operations ### ### Post incident review · Add additional limits to the services that will protect the infrastructure from failing completely · Expedite the micro service scaling and resilience project. Include into 2025 Q3 plan · Change Log Audit export requires further analysis and performance optimisation.</p> ]]> Thu, 12 Jun 2025 04:50:00 +0000 https://status.pulsesoftware.com/incident/cme0vz1wz009r9a0t86lnfxoi https://status.pulsesoftware.com/incident/cme0vz1wz009r9a0t86lnfxoi Four sites having issues with file service Type: Incident Duration: 30 minutes Affected Components: File Service Apr 7, 05:30:00 GMT+0 - Investigating - We are currently investigating this incident. Apr 7, 06:00:00 GMT+0 - Resolved - This incident has been resolved. Aug 7, 04:14:57 GMT+0 - Postmortem - An issue was identified as an issue in an update. The issue was resolved within 30 minutes and no more action is required. Type: Incident

Duration: 30 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>05:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>04:14:57</var> GMT+0</small><br><strong>Postmortem</strong> - An issue was identified as an issue in an update. The issue was resolved within 30 minutes and no more action is required..</p> ]]> Mon, 7 Apr 2025 05:30:00 +0000 https://status.pulsesoftware.com/incident/cme0ucpq603osta47dskznkqc https://status.pulsesoftware.com/incident/cme0ucpq603osta47dskznkqc Major outage across all services Type: Incident Duration: 20 minutes Affected Components: Pulse Web Service, Forms Service, Workflow Service, Identity Service, Public Job Sites Apr 1, 02:09:00 GMT+0 - Investigating - We are currently investigating this incident. Apr 1, 02:29:00 GMT+0 - Resolved - The system has automatically recovered Aug 7, 03:14:35 GMT+0 - Postmortem - All Database resources were consumed as a result of a very high number of notifications. The root cause was found to be a query inside the notification engine that consumed significant resources. A priority work item was created to optimise the process and assigned as the highest priority to resolve. Type: Incident

Duration: 20 minutes

Affected Components: , , , ,

<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>02:09:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>02:29:00</var> GMT+0</small><br><strong>Resolved</strong> - The system has automatically recovered .</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>03:14:35</var> GMT+0</small><br><strong>Postmortem</strong> - All Database resources were consumed as a result of a very high number of notifications. The root cause was found to be a query inside the notification engine that consumed significant resources. A priority work item was created to optimise the process and assigned as the highest priority to resolve..</p> ]]> Tue, 1 Apr 2025 02:09:00 +0000 https://status.pulsesoftware.com/incident/cme0tms9w05p8sjiwfkn0o827 https://status.pulsesoftware.com/incident/cme0tms9w05p8sjiwfkn0o827 Timeout issues Type: Incident Duration: 30 minutes Affected Components: Pulse Web Service Jan 29, 03:30:00 GMT+0 - Investigating - Users were occasionaly receiving timeout errors Jan 29, 04:00:00 GMT+0 - Resolved - This incident has been resolved. Aug 7, 03:08:35 GMT+0 - Postmortem - An attempt to diagnose a system issue while it was under heavy load resulted in database resource exhaustion, causing occasional timeouts. The diagnostics process has been changed to prevent this issue in the future. Type: Incident

Duration: 30 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 29</var>, <var data-var='time'>03:30:00</var> GMT+0</small><br><strong>Investigating</strong> - Users were occasionaly receiving timeout errors.</p> <p><small>Jan <var data-var='date'> 29</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>03:08:35</var> GMT+0</small><br><strong>Postmortem</strong> - An attempt to diagnose a system issue while it was under heavy load resulted in database resource exhaustion, causing occasional timeouts. The diagnostics process has been changed to prevent this issue in the future..</p> ]]> Wed, 29 Jan 2025 03:30:00 +0000 https://status.pulsesoftware.com/incident/cme0tcqqa002r12jcfcaqke7m https://status.pulsesoftware.com/incident/cme0tcqqa002r12jcfcaqke7m