Client has been advised not to use a bad design of the campaigns. The lags are now processed and operation back in normal
Posted Feb 05, 2024 - 06:10 CET
Monitoring
Internal brokers and replicas were upscaled to help the lag processing. We are currently monitoring the situation.
Posted Feb 05, 2024 - 05:38 CET
Identified
The issue has been identified at 4:30 CET. The issue has been caused by a huge property update campaign that was manually triggered by one of the clients. Lag is still being processed. There is no data loss.
Posted Feb 05, 2024 - 05:37 CET
Investigating
Campaing event triggers crashlooping on memory. The memory usage is huge. It's being overly processed but still useing huge amount of memory The massive property update is also causing IMF oplog lag Tracking lag currently on 50m