Pulse Software - Notice history

Incident Summary

During a scheduled Azure update, the Forms service was automatically restarted. Upon restarting, the service failed to establish a connection with the database. As a result, the Forms service remained in a failed state until it was manually restarted and successfully reconnected to the database.

Root Cause

The outage was triggered by the Forms service being unable to connect to the database immediately after the Azure-driven restart. The lack of sufficient database connection resilience mechanisms caused the service to fail rather than recover automatically.

Resolution

The service was restored by performing an additional restart, at which point it successfully re-established the database connection and resumed normal operations.

Preventative Actions

To improve resilience and prevent recurrence, the following work items have been defined:

• Enhanced database connection resilience: Implement improvements to ensure the Forms service can recover from temporary database connectivity issues.

• Health check enhancements: Update the Forms service health check to fail explicitly when the database connection is not healthy, enabling faster detection and automated recovery.

Incident Summary

On 24 Septmber, a significant spike in email volume caused delivery delays of up to 30 minutes. The root cause was the current email service reaching its throughput limits under increased load.

Root Cause

The existing architecture was not designed to scale dynamically to handle large, sudden increases in outbound email traffic. As a result, the system became a bottleneck during high-volume events.

Remediation & Next Steps

The email service is currently undergoing a major re-architecture focused on scalability and performance. The new system is expected to support an order of magnitude higher throughput than the current implementation.

We are targeting a full cutover to the new service in October 2025.

Status

Mitigation measures have been applied to reduce the likelihood of recurrence before the new architecture is deployed. Monitoring and alerting thresholds have also been adjusted to provide earlier detection of delays.

This incident has been resolved. The email queue is up to date.

Incident Summary

A Microsoft component responsible for extracting data from Excel began causing application pool crashes. When an application pool crashes, it impacts all users connected to that pool. Each crash resulted in temporary request timeouts for those users until the pool automatically recovered, typically within seconds. As a result, the issue only persisted for less than a minute and only affected users tied to the impacted pool.

Impact

• Users connected to the affected app pool experienced brief timeouts (under one minute).

• No lasting or widespread outages occurred, as app pools automatically restarted and recovered within seconds.

Root Cause

The failure was traced to a Microsoft Excel data extraction component, which triggered repeated crashes in the application pool.

Resolution

• All servers were patched and verified on the morning following the incident.

• A longer-term remediation has been expedited: migrating the Excel data extraction functionality to a dedicated microservice. This change will reduce dependency on the main web server and improve overall system resilience.

Next Steps

• Continue monitoring the updated servers for stability.

• Complete and deploy the microservice migration to fully eliminate reliance on the problematic component.

• Review alerting and recovery procedures to ensure faster detection and mitigation in future pool crashes.

We have identified the issue is related to imports. A fix will be deployed tonight.

A network issue between Pulse Web Services and Forms resulted in job listings and the offer process to fail. Restarting some of the impacted services resolved the issue. An additional alarm will be created to pickup so that the on call team can respond and rectify the issue in a shorter timeframe.