Disaster Tolerance Response ed-dt/r-01

1.0 Purpose

The purpose of this policy is to document the processes and responsibilities of the College faculty and staff in responding to security incidents disaster recovery (security compromises, virus infected systems, and events that render one or more network closets inoperable). This policy is intended to provide all users of College-based personal computers, servers, and networking hardware with information pertaining to how the Education Technology Center and the University Security Operation Services are expected to respond in the event of a system compromise or disaster within a College-operated network closet. 

2.0 Background

2.1 Incident Response

Incident response applies to the actions taken at all levels within Penn State University when a user's computer or any server is compromised for one or more of the following reasons:

  • Compromise by brute-force attacks from inside or outside the College.
  • Downloading of any virus that threatens continuous College communications and computing services.
  • Connecting any non-patched or compromised system to the College's hardwired or wireless network.
  • Participating in any computing practices that are unlawful, against College or University computing policies.
  • Participating in any computing activities that prevent or have the potential to prevent others from carrying out the College's academic and administrative missions.

Incident response also applies to any reported or discovered illegal activities on any computer used on University premises, where illegal activities are defined by University policies and laws established by local, state, or federal governments.

2.2 Disaster Tolerance/Recovery

Disaster Tolerance and Recovery are two distinctly different issues. Disaster Tolerance applies to actions taken by the College, a department, and users to insure that computing operations and network services are maintained or at worst case, gracefully degraded and terminated during a disaster. Disaster Recovery are those actions taken by the College, departments, and users to recover from events that render computing operations and network services inoperable. Events that initiate actions to maintain or restore computing operations and network services include but are not limited to momentary/long-term power outages, hardware failures, fire, natural disasters, and malicious attacks that render servers or systems inoperable or degraded.

3.0 Scope

All College departments, centers, and operational units are responsible for developing and implementing Disaster Tolerance/Recovery plans.

4.0 Policy

4.1 Incident Response

Any desktop, laptop, workstation, server or other network capable device found or suspected of violating any College or University policy focusing on ensuring secure and safe communications and computing will summarily and immediately be disconnected from the College of Education's and University's data backbone.

Users will be notified as quickly as possible of such action once the Education Technology Center or University Security personnel are satisfied that a real or potential threat to other users or the Internet in general has been mitigated. Individuals at any level (users, the Education Technology Center, and University Security) have the obligation to report any real or potential computer operational activities that may detract from normal computing activities.

4.2 Disaster Tolerance/Recovery

4.2.1 Disaster Tolerance

Disaster Tolerance is a result of planned actions, policies, hardware deployments, and any other efforts aimed at preventing limited to momentary/long-term power outages, hardware failures, fire or natural disasters from causing long-term disruptions of College academic or administrative activities. The Education Technology Center assumes the responsibility for Disaster Tolerance in networking operations throughout College and Univeristy-maintained Telecommunications Closets. The Education Technology Center is also responsible for these activities as they relate to maintenance and operations of core College servers (e.g., email, web, data, etc.) and departmental servers maintained by the Education Technology Center personnel.

In an effort to achieve Disaster Tolerance within the aforementioned operations and services, the Education Technology Center has implemented the following procedures:

  • Maintains spare components for critical networking hardware operations (as budget permits)
  • Maintains spare components for critical servers (as budget permits)
  • Provides and maintains Uninterruptible Power Supplies (UPS) for network equipment deployed in Telecommunications Closets maintained by the College
  • Provides and maintains UPS for all core College servers and departmental servers maintained by the Education Technology Center
  • Mirrors College email servers and deploys them in a secondary Telecommunications Closet so that they can assume email services in the event of a primary email cluster failure
  • Provides and maintains at least two months of data backup for core TSM backups for all College and departmental servers maintained by the Education Technology Center.


4.2.2 Disaster Recovery

Disaster Recovery encompasses all those activities and steps necessary to restore personnel and system services that have been interrupted by an unforeseen event(s) that may include but are not limited to momentary/long-term power outages, hardware failures, fire, natural disasters, and malicious attacks that render servers or systems inoperable/degraded. It necessarily includes making plans to relocate personnel in order to effectively reconstitute personnel and system services along with academic and administrative services.

It is neither economical nor practical to maintain 100% redundant hardware in preparation for any and all potential disasters. Therefore, as soon as conceivably possible and approved by appropriate University or other authority, the Education Technology Center personnel will enter building Telecommunications Closets for the purpose of assessing damage and serviceability of network hardware and core/departmental server effected by a disaster. All equipment will be inventoried and categorized according to its serviceability. Steps will immediately be taken to procure and receive replacements for unserviceable equipment.

  • In the event that offices and equipment used daily by the College's networking, computing and training personnel are rendered uninhabitable, personnel will work from their homes or other locations where connectivity is available.
  • Office or laboratory lab space will be made available to displaced personnel based on a separate agreement made by the Facility Manager, the College's Associate Dean for Technology (IT Director)
  • Replacement computing assets will be made available through emergency local purchases.
  • The IT Director will work with the College's Financial Officer to establish emergency procurement procedures.

In the event of a minor disaster such as a long-term electrical power outage, the Education Technology Center will work with the College's Facilities Coordinator and the Office of Physical Plant (OPP) to have power generation equipment installed to restore critical networking services. Naturally, this process assumes that a building remains serviceable and is approved for use by OPP or the appropriate authorities.

Reconstitution of networking operations and computing services will receive the highest priority. Initially, only that equipment and tools that are absolutely required to support reestablishment of reliable/sustainable services will be procured under the aforementioned emergency procurement process.

Departments are responsible for establishing and implementing Disaster Recovery policies and procedures that will enable them to reconstitute operations and continue their academic and administrative missions.

5.0 Incident Response Enforcement

University Security Operations (SOS) personnel have the right and responsibility to identify and take immediate action to curtail any computing operation that violates University Policies. They have the right and responsibility to intentionally or randomly scan any systems on the University's backbone. Furthermore, they have the right and obligation to summarily curtail a system's computing activities that disrupt or are suspected of negatively impacting secure computing activities on University property or beyond.

Education Technology Center personnel have the right and responsibility to identify and take immediate action to curtail any computing operation that violates College or University Policies. They have the right and responsibility to intentionally or systematically scan any systems on the College's network. Furthermore, they have the right and obligation to summarily curtail a system's computing activities that disrupt or are suspected of negatively impacting secure computing activities on College, University or beyond.

Illicit and illegal activities are forbidden on the College and University networks. Illicit activities are those which are expressly prohibited by Department, University and/or College policies and are not illegal as defined by local, state, or federal laws; they include but are not limited to operating business for personal gains. It is the responsibility of a department head to ensure that individuals within their departments abstain from such practices. Should someone outside or within the department report such activities to a department head or the Education Technology Center, it is the Education Technology Center's responsibility to advise the offending party of the offence and to ensure that all remnants of such activities are removed immediately from the College's network and the computer or server on which it resides. Questions concerning illicit activities may be directed to the Information Technology Manager, at 865-0474 and the University's Security Officers at security@psu.edu.

Illegal activities are those that are contrary to local, state, or federal laws. Anyone becoming aware of such activities must immediately contact the College's Information Technology Manager, at 865-0474 and University's Security Officers at security@psu.edu. No further actions are to be taken at the department level until and when either the Information Technology Manager or the University Security Officer notifies the department head. No one in a department is to discuss their knowledge or suspicion of illegal activities with individuals suspected of participating in such activities; this is ultimately the responsibility of the University's Security Officer (SOS).

Any faculty or staff member has responsibility to identify and take immediate action to curtail any computing operation that violates departmental, College or University Policies. At the department level and other than prescribed above, faculty, staff and students are explicitly prohibited from scanning systems on the College's network or University's data backbone.

Faculty, staff or students that have a compromised or suspected compromised system identified, which is owned by Penn State, are obligated to repair the system or refer the system to the Education Technology Center for repair.

Faculty, staff or students that have a personally owned system identified which is compromised or suspected of being compromised, are obligated to repair the system immediately.

In either case, systems must be validated as having been patched with the latest operating system (OS) updated and cleansed of any virus-laden or disruptive software before being reinstated on any Penn State network.

6.0 Revision History

Rev. 5-3-2016, David Cochrane

Addendum to be added under 4.0 Policy, 4.1 Incident Response

4.1a The following procedure outlines the College’s initial response to a local security incidence and reporting out from the College Carrara Education Technology Center (CETC) to the Office of Information Security (OIS) is as follows.  

CETC’s internal procedure requires communicating locally discovered incidents and reporting them via email to the following email locations in the Office of Information Security (OIS).

  • Security issues can be reported to security@psu.edu.  
  • Phishing threats can be reported to phishing@psu.edu.

This procedure allows CETC and the OIS to react quicker when communicating discovered incidents. This provides a communication feedback loop on all incidents generated from the College or to the College by OIS to the College IT persons through current systems. In addition  it provides a means of reporting out on local incidences.  Once reported OIS will create a ticket in their ticketing system (SNOW) to address the nature and outcome of the challenge.