All services have returned to normal operation. We will continue to monitor for issues throughout the holiday weekend and remain on high alert.
Posted Nov 27, 2020 - 12:01 MST
Monitoring
We have re-enabled recommendations for all clients and are still seeing normal error rates and latency continues to improve to near normal levels. We will continue to monitor the situation. All clients remain on a 5 minute cache, any indexing or merchandising changes may take up to 5 minutes to propagate through this cache. We will update again by 2pm EST.
Posted Nov 27, 2020 - 11:46 MST
Update
The steps we have taken to remediate the problem have begin to show great improvement. Our error rates have returned to normal levels and while latency is still higher than normal it is trending back towards normal. We will provide another update within 15 minutes.
Posted Nov 27, 2020 - 11:30 MST
Update
We are disabling our recommendations service to help alleviate issues with our core search and category features. There were some issues causing some clients to not have their requests use our cache. We are currently working to fix this issue to ensure they are properly hitting the cache to help alleviate load. Our next update will be at or before 1:30pm.
Posted Nov 27, 2020 - 11:21 MST
Update
We are continuing to work to resolve this issue. We have temporarily suspended indexing as well to help alleviate some of the problems. Next update will be at 1:25pm EST.
Posted Nov 27, 2020 - 11:10 MST
Update
We are currently working to roll out additional resources to help mitigate the traffic issues. We will provide another update at 1:10pm EST.
Posted Nov 27, 2020 - 10:54 MST
Update
We are still seeing large traffic anomalies and are exploring steps to mitigate them.
Posted Nov 27, 2020 - 10:42 MST
Investigating
It appears that we are experiencing another high traffic event, we are currently exploring how to resolve this issue.
Posted Nov 27, 2020 - 10:10 MST
Monitoring
We are seeing a return to normal and continuing to monitor the situation.
Posted Nov 27, 2020 - 09:49 MST
Update
We are working to institute a more aggressive caching system across all clients in order to reduce the effects of this traffic event.
Posted Nov 27, 2020 - 09:34 MST
Identified
We saw an extremely high traffic event effecting a limited number of our customers, we are taking steps to limit the impact of this now.
Posted Nov 27, 2020 - 09:13 MST
Investigating
We are seeing slightly higher than normal API response times for a subset of our customers. We are exploring possible causes and solutions.
Posted Nov 27, 2020 - 08:34 MST
This incident affected: Search, Indexing, Autocomplete, and Product Recommendation Widgets.