Paul Norton: April 2008 Archives

Several ProStores hosting customers experienced a critical error early Monday morning (04/07/2008). An error message appeared on the Store Front and in the Store Administration for affected customers, causing the store to be inoperable. We believe the issue was triggered by our regularly scheduled maintenance, but the error didn't occur until several hours after the maintenance was completed. We have not experienced any issues previously with regards to ProStores hosting or any of our other offerings due to routine server maintenance.

Since this was a software error and not an actual server/service down event, our automated monitoring systems were unable to detect the issue and alert our technical staff.

In addition, our telephone paging service was unavailable during the early morning on Monday. Therefore, the technical staff was not alerted to this issue until we opened our office at the start of the business day. Upon discovery of the problem, the issue was immediately escalated to top tier and the ProStores application was restored.

The application vendor of ProStores, Ebay, was alerted shortly thereafter and a proposed fix was sent to Neoverve. According to Ebay, there is a bug in the configuration where a section for multi-cast clustering is enabled by default.

An updated configuration fix will be completed today on all ProStores servers. We are also investigating a new paging system with multiple escalation support to be implemented immediately.

We fully understand the severity of the service interruption and apologize for its effect. All measures will be taken to ensure continuous service for our customers.