2018.06.20 - Incident Report: Decreased Performance and Unplanned Downtime

Summary

On May 20 - May 21, 2018, emsCharts users experienced degraded system performance and an unplanned downtime.

Contents


Summary of Events

  • On the evening of June 20, 2018 beginning around 5:30 EDT performance of our production database began to degrade
  • Staff Database Administrators worked with Oracle support through the night to identify the process consuming large amounts of Disk I/O which was competing with our standard activity. While web traffic was able to pass to the database, performance was poor.
  • The issue was resolved a little after 11:00 am EDT, June 21, 2018.
  • No data loss occurred, and all systems have returned to normal operations.


If applicable, this page will be updated with additional information.