City of Paducah Recovers Information Systems and Resumes Secure IT Operations

Date of release:  February 21, 2020

Statement from Paducah City Manager Jim Arndt
City of Paducah Recovers Information Systems and Resumes Secure IT Operations

generic computer codingThe City of Paducah has regained access to the servers and records impacted by our recent IT security incident.  City IT systems and file storage are back online and operating securely.  Our security team has identified and remediated the point of entry used for unauthorized access, and multiple security scans by outside experts did not detect any active malicious activity within our network.  Furthermore, a thorough forensic analysis did not find any evidence of efforts to remove files or data from our systems.  I want to stress that there is no indication any information has been misused as a result of this incident. 

We treated this incident with the highest priority and appreciate everyone’s patience as we worked through a complex, sensitive, and time-consuming process to confirm system security and resume normal operations.  We have now reached a point where we are able to provide a more thorough explanation of what happened and the steps we have taken in response to this incident.

On Saturday, February 1, we became aware of an unauthorized intrusion into our IT network by an unknown third-party who used malicious software to compromise our systems and encrypt numerous data files.  The threat actor subsequently contacted the city demanding a payment in exchange for decryption keys to restore access.  Upon discovery of the incident, we immediately disconnected impacted servers and initiated a comprehensive investigation and response with the assistance of independent IT security and computer forensic specialists.  We coordinated with local and federal law enforcement and our insurance provider.  

After thoroughly investigating all options for restoring our IT systems, the City team in consultation with outside security experts, the Kentucky League of Cities, and our insurance provider ultimately decided to pursue a multi-pronged approach of rebuilding certain systems from scratch and unlocking others by purchasing decryption keys from the threat actor for a payment of approximately $30,000.  This was a carefully considered decision that we determined to be in the best interest of our citizens and our ongoing data security.  Decryption not only was the most expeditious and cost-effective way to restore access to our technology and important records but also enabled the most thorough forensic review of our systems, so that we could best understand the impact of this incident.

Our recovery included a methodical process of restoring and performing security inspections on individual servers before bringing them back online one-by-one.  In addition, our IT team completed the testing and restoration of nearly 300 individual IT machines which have been loaded with advanced active threat detection software.  We have also reconfirmed the security of our email system and are confident our mail server was not compromised by this incident. 

We have already implemented measures to enhance security – including systemwide password resets and use of advanced active threat detection – and we are also using this as an opportunity to replace some of our older IT equipment.  We will continue working with outside experts to identify and implement new security measures to strengthen our defenses and protocols going forward.  These measures will include new cybersecurity training for city employees which had already been slated for our 2020 Strategic Plan prior to this incident.

Fortunately, the temporary outage of some IT systems had only a minor impact on our ability to conduct city business as usual.  We are deeply sorry for any inconvenience this incident may have caused citizens or our staff and are grateful for the resourcefulness and resilience of our many dedicated employees, who continued to provide city services at a high level and meet the needs of our citizens throughout our recovery.