At approximately 0900 on Dec 25, 2008 email and home directory services on geosc.psu.edu became unavailable. Investigation revealed multiple disk failures in the redundant disk array. Failed drives were replaced with available spare units. File restoration is currently underway and is expected to continue through Dec 26. The backup set used was taken at Dec 24 2100 - 2300.
Email received and files created or modified since Dec 24 2100 are lost. Email sent after Dec 25 1000 will be queued on the sending server for later delivery.
Email services are disabled until file restoration completes. This is to prevent any additional data loss during the restore process.
Updates will be posted to this article and an announcement will be made to the newsletter mailing list upon completion of the restore process.
UPDATE Dec 28 1030: The restoration process is still underway. Because the restore will take a very long time to complete (possibly up to 2 weeks) we have changed strategies. Email is now back in service and the restore process will now NOT overwrite existing files. Once all directories are restored, we will restore the old inboxes from the backups to a new file. Upon request we will merge the old inbox with the new inbox.
Email is back online but many files remain to be restored. If you are missing items, please contact us (support@geosc.psu.edu) with the name and location of the files. We can tell you if your directory has been completely restored.
