Duration: 2 days, 23 hours and 41 minutes

Affected Components: Identity Service, Pulse Web Service, Workflow Service, Forms Service

May 29, 01:02:36 GMT+0
Investigating - We are currently investigating an issue causing workflow failures, first observed this morning. Some workflow executions may fail or be delayed during this period. Our engineering team is actively working to identify the root cause. We will post another update within the next 30 minutes, or sooner as new information becomes available. We apologize for the disruption and appreciate your patience..

May 29, 02:25:22 GMT+0
Identified - We have identified the cause of the ongoing service disruption: an expired TLS certificate is causing connection failures across our services. Our engineering team is working to rotate the certificate and restore service..

May 29, 02:24:55 GMT+0
Identified - Escalation update: the workflow failures reported earlier have expanded — we are now seeing impact across all microservices. We are currently experiencing a major service disruption affecting all of our major services. The issue has been identified as related to SSL certificate validation, and our engineering team is actively working on a fix. Customers may experience failures or degraded performance across the platform during this period. .

May 29, 05:41:48 GMT+0
Monitoring - A hotfix has been deployed and all services are now restored. Our team is currently validating system health and will be retriggering any failed workflows as soon as possible. We will continue to monitor closely and provide a final update once validation is complete. Thank you for your patience throughout this incident..

Jun 1, 00:44:02 GMT+0
Resolved - This incident has been resolved..

Duration: 7 minutes

Affected Components: Forms Service, Email Service, File Service, Public Job Sites, Identity Service, Workflow Service, Pulse Web Service

May 1, 05:07:55 GMT+0
Investigating - We are currently experiencing a service disruption affecting all our sites. Our team is actively investigating the issue and working to restore service as quickly as possible. We will provide an update shortly. We sincerely apologize for the inconvenience and appreciate your patience..

May 1, 05:14:31 GMT+0
Resolved - Earlier today, all our sites became temporarily unavailable due to a faulty infrastructure resource that impacted some of our core services. The issue has been resolved and all sites are now operating normally. **Impact** During the affected window, customers were unable to access our sites as core services were disrupted by the faulty infrastructure resource. **Root Cause** One of our infrastructure resources became faulty, which impacted several core services that our sites depend on, resulting in the outage. **Resolution** Our team identified the faulty resource and remediated it as quickly as possible. Core services were restored and all sites are back online and operating normally. **Next Steps** We are conducting a full review of the faulty resource and the dependent services to understand why this caused a site-wide impact and will implement safeguards to improve resilience and prevent recurrence. We sincerely apologize for the disruption and appreciate your patience while we worked to restore service. .

Duration: 53 minutes

Affected Components: Email Service

Apr 29, 06:17:46 GMT+0
Investigating - We recently identified and investigating an issue that prevented some notifications from being triggered between 10:00 AM and 12:00 PM 29th April 2026.

Apr 29, 07:09:35 GMT+0
Identified - Due to heavy load on our email service, notifications queued up and were not delivered as expected during this window.

Apr 29, 07:11:14 GMT+0
Resolved - We added additional capacity to our email service, which cleared the backlog instantly. Notifications resumed triggering normally. We are also adding monitoring and autoscaling improvements to prevent this from happening again. No action is required on your part We sincerely apologize for any inconvenience this may have caused. If you have any concerns, please reach out to our support team..

Duration: 24 days

Affected Components: Email Service

Apr 23, 03:00:00 GMT+0
Identified - We recently identified and resolved an issue that prevented some email notifications from being delivered between **1:00 PM on April 23rd and 11:00 AM on April 24th**.

Apr 24, 01:00:00 GMT+0
Resolved - **What happened** A configuration issue in our email delivery system caused a subset of notifications to be rejected rather than delivered during 1:00 PM on April 23rd and 11:00 AM on April 24th. **What we've done** Our team identified the root cause and restored normal delivery on April 24th at 11:00 AM. Email notifications are now operating as expected. **What's next** We are in the process of retriggering the notifications that were missed during **1:00 PM on April 23rd and 11:00 AM on April 24th**, so you should receive any pending notifications shortly. No action is required on your part. We sincerely apologise for any inconvenience this may have caused and appreciate your patience while we worked to resolve it. If you have any concerns, please reach out to our support team..

Duration: 15 hours

Affected Components: Forms Service, File Service, Public Job Sites, Workflow Service, Pulse Web Service

Mar 10, 00:20:00 GMT+0
Investigating - Multiple production environments are currently experiencing HTTP timeouts when accessing the application. Users are unable to load key pages, and the sites fail with a generic timeout screen. .

Mar 10, 00:30:00 GMT+0
Identified - * Engineering team notified and investigating root cause. * Checking: * Application service health across environments * API gateway / load balancer logs * Database performance and connection saturation * Recent deployments or infra changes.

Mar 10, 00:35:00 GMT+0
Resolved - A misbehaving Redis cluster caused requests to stall, leading to degraded performance across affected site. The Redis cluster was flushed and restored to a healthy state. All systems have now returned to normal operation and performance has stabilised. .

Duration: 3 days, 7 hours and 57 minutes

Affected Components: Email Service

Feb 19, 16:00:00 GMT+0
Investigating - We have identified an issue where notifications were failing to send for a subset of customers. Our team is currently investigating the root cause.

Feb 22, 23:56:57 GMT+0
Resolved - The incident has been fully resolved. The faulty instance causing the incorrect parameter handling has been removed, and services are operating normally. As a preventive measure, we are implementing improved automated checks that will detect and evict such unhealthy instances in the future, ensuring greater reliability and consistency across the notification pipeline.

Duration: 2 hours and 5 minutes

Affected Components: Email Service, File Service, Public Job Sites, Identity Service, Workflow Service, Pulse Web Service, Forms Service

Feb 9, 10:00:00 GMT+0
Investigating - A critical response team has been setup and we are investiating the isssue.

Feb 9, 12:05:00 GMT+0
Resolved - A denial of service attack has been mitigated.

Duration: 18 minutes

Affected Components: Forms Service

Feb 9, 06:08:43 GMT+0
Identified - We have identified the issue as a disk issue on one of our database services. We are adding additional resources to mitigate the issue..

Feb 9, 06:27:02 GMT+0
Resolved - This incident has been resolved..

Duration: 6 hours and 45 minutes

Feb 8, 22:00:00 GMT+0
Monitoring - Over the weekend, while our team was running internal security testing, emails were unintentionally sent from a non‑production system. As a result, customers may have received multiple unexpected emails. We identified the issue today, stopped it immediately, and confirmed that our live services and customer data were not affected. We understand that receiving repeated, unexpected emails can be frustrating or concerning, and we’re genuinely sorry for the disruption this caused. Our team is reinforcing safeguards between testing and customer communications to ensure this doesn’t happen again. Thank you for your patience and understanding.

Feb 9, 04:44:38 GMT+0
Resolved - This incident has been resolved..

Duration: 52 minutes

Affected Components: Pulse Web Service

Jan 13, 22:00:00 GMT+0
Investigating - We are currently investigating this incident..

Jan 13, 22:52:00 GMT+0
Resolved - The issues has not been resolved.

Jan 14, 04:26:51 GMT+0
Postmortem - At approximately 8:00 AM, degraded performance in the production system was detected by our alerting system and reported by several customers. A critical response team was immediately assembled to investigate the issue. The initial investigation identified the in-memory cache service as the primary bottleneck, as it was operating beyond expected capacity. The underlying cause of the increased load was not immediately clear. To mitigate customer impact and allow additional time for analysis, the decision was made to scale up the cache service. The scale-up alleviated the bottleneck almost immediately, and system performance returned to normal operating levels. As a follow-up action, we will integrate the in-memory cache service into our APM observability platform to improve visibility, monitoring, and early detection of similar issues in the future..

Duration: 6 hours and 47 minutes

Affected Components: Workflow Service

Dec 10, 16:00:00 GMT+0
Investigating - We are currently investigating this incident..

Dec 10, 22:47:00 GMT+0
Resolved - This incident has been resolved. Post moretom to follow..

Affected Components: Email Service, File Service, Public Job Sites, Identity Service, Workflow Service, Pulse Web Service, Forms Service

Nov 18, 14:40:00 GMT+0
Resolved - This incident has been resolved..

Nov 18, 12:30:00 GMT+0
Resolved - This incident has been resolved..

Nov 18, 20:16:32 GMT+0
Postmortem - Pulse Software relies on Cloudflare to enhance the performance, security, and reliability of its infrastructure. Cloudflare is a leading provider in this space, powering an estimated 20% of global internet traffic. On the date of the incident, a major Cloudflare outage disrupted network connectivity to Pulse Software’s systems, resulting in a period of service unavailability. Details of the upstream outage can be found in Cloudflare’s incident report: <https://www.cloudflarestatus.com/incidents/8gmgl950y3h7>.

Duration: 8 minutes

Affected Components: Public Job Sites

Nov 9, 22:14:41 GMT+0
Investigating - We are currently investigating this incident..

Nov 9, 22:22:49 GMT+0
Resolved - This incident has been resolved..

Duration: 10 hours and 44 minutes

Affected Components: Forms Service

Sep 28, 09:35:00 GMT+0
Investigating - We are currently investigating this incident..

Sep 28, 20:19:00 GMT+0
Resolved - This incident has been resolved..

Sep 29, 02:26:49 GMT+0
Postmortem - ### **Incident Summary** During a scheduled Azure update, the Forms service was automatically restarted. Upon restarting, the service failed to establish a connection with the database. As a result, the Forms service remained in a failed state until it was manually restarted and successfully reconnected to the database. ### **Root Cause** The outage was triggered by the Forms service being unable to connect to the database immediately after the Azure-driven restart. The lack of sufficient database connection resilience mechanisms caused the service to fail rather than recover automatically. ### **Resolution** The service was restored by performing an additional restart, at which point it successfully re-established the database connection and resumed normal operations. ### **Preventative Actions** To improve resilience and prevent recurrence, the following work items have been defined: * **Enhanced database connection resilience**: Implement improvements to ensure the Forms service can recover from temporary database connectivity issues. * **Health check enhancements:** Update the Forms service health check to fail explicitly when the database connection is not healthy, enabling faster detection and automated recovery..

Duration: 43 minutes

Affected Components: Email Service

Sep 24, 04:46:29 GMT+0
Investigating - We are currently investigating this incident..

Sep 24, 05:29:41 GMT+0
Resolved - This incident has been resolved. The email queue is up to date..

Sep 25, 07:40:27 GMT+0
Postmortem - **Incident Summary** On 24 Septmber, a significant spike in email volume caused delivery delays of up to 30 minutes. The root cause was the current email service reaching its throughput limits under increased load. **Root Cause** The existing architecture was not designed to scale dynamically to handle large, sudden increases in outbound email traffic. As a result, the system became a bottleneck during high-volume events. **Remediation & Next Steps** The email service is currently undergoing a major re-architecture focused on scalability and performance. The new system is expected to support an order of magnitude higher throughput than the current implementation. We are targeting a full cutover to the new service in October 2025. **Status** Mitigation measures have been applied to reduce the likelihood of recurrence before the new architecture is deployed. Monitoring and alerting thresholds have also been adjusted to provide earlier detection of delays..

Duration: 5 hours and 11 minutes

Affected Components: Pulse Web Service, Forms Service, Workflow Service, File Service, Identity Service, Public Job Sites, Email Service

Sep 23, 06:00:50 GMT+0
Investigating - We are currently investigating this incident..

Sep 23, 11:11:21 GMT+0
Resolved - This incident has been resolved..

Sep 23, 06:02:30 GMT+0
Identified - We have identified the issue is related to imports. A fix will be deployed tonight..

Sep 23, 10:51:59 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Sep 24, 00:02:09 GMT+0
Postmortem - ### Incident Summary A Microsoft component responsible for extracting data from Excel began causing application pool crashes. When an application pool crashes, it impacts all users connected to that pool. Each crash resulted in temporary request timeouts for those users until the pool automatically recovered, typically within seconds. As a result, the issue only persisted for less than a minute and only affected users tied to the impacted pool. ### Impact * Users connected to the affected app pool experienced brief timeouts (under one minute). * No lasting or widespread outages occurred, as app pools automatically restarted and recovered within seconds. ### Root Cause The failure was traced to a Microsoft Excel data extraction component, which triggered repeated crashes in the application pool. ### Resolution * All servers were patched and verified on the morning following the incident. * A longer-term remediation has been expedited: migrating the Excel data extraction functionality to a dedicated microservice. This change will reduce dependency on the main web server and improve overall system resilience. ### Next Steps * Continue monitoring the updated servers for stability. * Complete and deploy the microservice migration to fully eliminate reliance on the problematic component. * Review alerting and recovery procedures to ensure faster detection and mitigation in future pool crashes..

Duration: 3 hours and 16 minutes

Affected Components: Pulse Web Service, Forms Service, Workflow Service

Aug 2, 20:01:00 GMT+0
Investigating - We are currently investigating this incident..

Aug 2, 23:17:00 GMT+0
Resolved - This incident has been resolved..

Aug 7, 04:28:40 GMT+0
Postmortem - A network issue between Pulse Web Services and Forms resulted in job listings and the offer process to fail. Restarting some of the impacted services resolved the issue. An additional alarm will be created to pickup so that the on call team can respond and rectify the issue in a shorter timeframe..

Duration: 14 minutes

Affected Components: Pulse Web Service, Forms Service, Workflow Service, Email Service, Identity Service, Public Job Sites

Jul 17, 03:59:00 GMT+0
Resolved - This incident has been resolved..

Jul 17, 03:45:00 GMT+0
Investigating - A major outage impacting all services.

Aug 4, 22:00:49 GMT+0
Postmortem - | **Date and Time of Incident** | 2025-07-17 13:45 | | ----------------------------- | ------------------------------ | | **Incident Type** | Major Outage | | **Reported By** | Reported Internally | | **Location/System Affected** | All Australian Sites/Customers | | **Prepared By** | Site Reliability Engineer | | **Acknowledged By** | | ## Description of the Incident On 17 June 2025 at approximately 13:43 AEST, a sudden spike in resource consumption was detected across our infrastructure. This surge posed a potential performance risk to all customers. In response, our team initiated standard mitigation procedures to stabilize the environment and maintain service quality. During this process, an engineer made a configuration error while addressing an overloaded server. This error inadvertently caused the platform to go offline for a duration of 16 minutes. ## Root Cause Analysis The incident was caused by a manual configuration process that should have been automated. The lack of automation introduced the possibility of human error, which ultimately led to the misconfiguration and temporary service disruption. ## Timeline of Event · 13:43 – Overloaded server identified · 13:45 – Configuration error occurred · 14:00 – Alternative Server Cluster configured · 14:01 – Platform restored, and services resumed ## Post incident review Following the incident, a comprehensive review was conducted. Key findings and actions include: · Automation Improvements: Plans are underway to automate the configuration process to eliminate manual intervention and reduce the risk of human error · Monitoring Enhancements: Resource monitoring tools will be refined to provide earlier alerts and more granular diagnostics.

Duration: 3 minutes

Affected Components: Pulse Web Service

Jul 16, 07:10:00 GMT+0
Resolved - The service has automatically marked a node as unhealthy and recovered..

Jul 16, 07:07:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 5 minutes

Affected Components: Pulse Web Service, Forms Service, Workflow Service, File Service, Identity Service, Public Job Sites

Jun 12, 04:50:00 GMT+0
Investigating - We are currently investigating this incident..

Jun 12, 05:55:00 GMT+0
Resolved - A service node crashed and failed to automatically recover. Engineers had to manually recover the node..

Aug 7, 04:18:24 GMT+0
Postmortem - ## Description of the Incident On 12th June 2025, a major outage of the Pulse Software system was reported. This impacted all services and all customers. The critical response process was followed, this including setting up a critical response team and sending communications internally. It was found that our microservice infrastructure had failed and did not automatically recover. The critical response team intervened adding additional infrastructure and removing the unhealthy infrastructure. This is the first time we have had a complete failure on our microservices infrastructure. The outage lasted for a total of 65 minutes. Over the last six months the uptime for Pulse Software Platform has been exceptional at 99.98% with no maintenance windows. After this event this has dropped to 99.96% after the event. This is well above 99.5% SLA. ## Root Cause Analysis ### Technical Cause: The incident was traced to a very high peak workload impacting the change log auditing service. This service was overwhelmed and crashed the service infrastructure preventing users from accessing the system. This is the first time the entire service has crashed at Pulse Software. ## Timeline of Event * **13 June 2025** * 14:50: Issue was reported internally within Pulse * 14:50: Critical response team setup * 14:53: A series of uptime alerts were triggered * 14:53: Critical response team found the Microservice infrastructure was in an unhealthy state * 14:54: Communications were sent internally of a major service disruption impacting all services and all customers * 15:05: Redirected all sites to maintenance mode * 15:10: Attempts to recover the infrastructure failed * 15:11: Started to setup new infrastructure to support microservices * 15:55 All services returned to normal operations ### ### Post incident review · Add additional limits to the services that will protect the infrastructure from failing completely · Expedite the micro service scaling and resilience project. Include into 2025 Q3 plan · Change Log Audit export requires further analysis and performance optimisation.

Duration: 30 minutes

Affected Components: File Service

Apr 7, 05:30:00 GMT+0
Investigating - We are currently investigating this incident..

Apr 7, 06:00:00 GMT+0
Resolved - This incident has been resolved..

Aug 7, 04:14:57 GMT+0
Postmortem - An issue was identified as an issue in an update. The issue was resolved within 30 minutes and no more action is required..

Duration: 20 minutes

Affected Components: Pulse Web Service, Forms Service, Workflow Service, Identity Service, Public Job Sites

Apr 1, 02:09:00 GMT+0
Investigating - We are currently investigating this incident..

Apr 1, 02:29:00 GMT+0
Resolved - The system has automatically recovered .

Aug 7, 03:14:35 GMT+0
Postmortem - All Database resources were consumed as a result of a very high number of notifications. The root cause was found to be a query inside the notification engine that consumed significant resources. A priority work item was created to optimise the process and assigned as the highest priority to resolve..

Duration: 30 minutes

Affected Components: Pulse Web Service

Jan 29, 03:30:00 GMT+0
Investigating - Users were occasionaly receiving timeout errors.

Jan 29, 04:00:00 GMT+0
Resolved - This incident has been resolved..

Aug 7, 03:08:35 GMT+0
Postmortem - An attempt to diagnose a system issue while it was under heavy load resulted in database resource exhaustion, causing occasional timeouts. The diagnostics process has been changed to prevent this issue in the future..