The situation has remained stable for several hours, the internal monitoring doesn’t show any problem. Therefore, we are closing the incident.
The root cause has been identified: a 3rd party tool was upgraded and it introduced a bug that caused networking issue on the nodes of several clusters.
The Bloomreach team has identified a list of potential improvement to prevent or mitigate the impact of similar situation in the future.
Kind regards, Bloomreach Support team.
Posted Mar 13, 2024 - 11:27 CET
Monitoring
Dear Customer,
Further mitigation has been applied and the situation is currently stable. However, the team is still observing and closely monitoring the clusters while working on a permanent fix that we expect to rollout tomorrow.
As soon as we have any new updates, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 12, 2024 - 18:11 CET
Update
Dear Customer,
We are continuing to work with our upstream provider to investigate the internal DNS issue.
We do not have an ETA for a resolution at the moment.
In the meantime, the Bloomreach SRE team is putting all the effort in order to mitigate the impact on the affected clusters. As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 12, 2024 - 15:46 CET
Update
Dear Customer,
We are continuing to work with our upstream provider to investigate the internal DNS issue.
We do not have an ETA for a resolution at the moment.
In the meantime, the Bloomreach SRE team is putting all the effort in order to mitigate the impact on the affected clusters. As soon as we have any new updates, or an estimated resolution time, we will share with you.
Posted Mar 12, 2024 - 12:58 CET
Identified
Dear Customer,
Unfortunately the issue the internal DNS has become apparent again this morning and we are working again with our upstream provider on a resolution of the problem. However, we do not have an ETA at the moment.
In the meantime, the Bloomreach SRE team is putting all the effort in order to mitigate the impact on the affected clusters. As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 12, 2024 - 11:47 CET
Update
Dear Customer,
A fix has been rolled out and the team is monitoring the situation.
As soon as we have any new updates, or an estimated resolution time, we will be sure to share it with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 22:02 CET
Update
Dear Customer,
A mitigation action is being applied and the team is monitoring the situation.
As soon as we have any new updates, or an estimated resolution time, we will be sure to share it with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 20:43 CET
Update
Dear Customer,
A mitigation action is being applied and the team is monitoring the situation.
As soon as we have any new updates, or an estimated resolution time, we will be sure to share it with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 19:33 CET
Monitoring
Dear Customer,
A mitigation action is being applied and the team is monitoring the situation.
As soon as we have any new updates, or an estimated resolution time, we will be sure to share with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 18:25 CET
Update
Dear Customer,
The issue has been identified as an internal DNS issue and we are working with our upstream provider on a resolution of the problem. However, we do not have an ETA at the moment.
In the meantime, the Bloomreach SRE team is putting all the effort in order to mitigate the impact on the affected clusters. As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 17:12 CET
Identified
Dear Customer,
The issue has been identified as an internal DNS issue and we are working with our upstream provider on a resolution of the problem. However, we do not have an ETA at the moment.
In the meantime, the Bloomreach SRE team is putting all the effort in order to mitigate the impact on the affected clusters. As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 16:09 CET
Update
Dear Customer,
Our internal monitoring is reporting unavailability of nodes on multiple clusters. We are in close contact with our upstream provider to further investigate the problem.
As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 15:06 CET
Update
Dear Customer,
Our internal monitoring is reporting unavailability of nodes on multiple clusters. We are in close contact with our upstream provider to further investigate the problem.
As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 14:00 CET
Update
Dear Customer,
Our internal monitoring is reporting unavailability of nodes on multiple clusters. We are in close contact with our upstream provider to further investigate the problem.
As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 13:00 CET
Update
Dear Customer,
Our internal monitoring is reporting unavailability of nodes on multiple clusters. We are in close contact with our upstream provider to further investigate the problem.
As soon as we have any new updates, or an estimated resolution time, we will share with you.
Kind regards, Bloomreach Support team.
Posted Mar 11, 2024 - 11:55 CET
Investigating
Dear Customer,
Our internal monitoring is reporting unavailability of nodes on multiple clusters. We are in close contact with our upstream provider to further investigate the problem.
As soon as we have any new updates, or an estimated resolution time, we will be sure to share with you.