How I recovered Exchange Server 2003

This is probably not very interesting to sports card collectors, but if you recently had a power failure or just got impatient with your Exchange server and did a hard reboot, you might find this useful.

Now that we have our new servers up and running, I needed to move our old server to be next to the new ones. The old server is just used as our mail server running Exchange Server 2003. Well, after giving it a good 10 minutes to shut down, it still hadn’t completely shut down, and I was getting impatient (it was 2:30 AM). So, I just pulled the power cable, and moved the server.

When the server finally came back up Exchange complained with this error in the event log.

Information Store (3812) First Storage Group: Database G:\Program Files\Exchsrver\MDBDATA\priv1.edb: Page 3031 (0x00000bd7) failed verification due to a flush-order dependency mismatch. This page should have flushed before page 7216 (0x00001c30), but the latter page has instead flushed first. Recovery/restore will fail with error -255. If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware “losing” one or more flushes on one or both of these pages sometime in the past. Please contact your hardware vendor for further assistance diagnosing the problem.

I have a default Microsoft Windows Small Business Server 2003 install with the Exchange data files in “G:\Program Files\Exchsrver\MDBDATA\”.

To fix the issue I did the following:

  • Made a backup of the entire “G:\Program Files\Exchsrver\MDBDATA\” directory
  • Found this article about eseutil
  • From “G:\Program Files\Exchsrver\bin” I ran the following command.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA”

  • This complained about priv1.edb being corrupt, so I ran the following commands to repair and defragment the edb files.

    eseutil.exe /p “G:\Program Files\Exchsrver\MDBDATA\priv1.edb”
    eseutil.exe /d “G:\Program Files\Exchsrver\MDBDATA\priv1.edb”
    eseutil.exe /p “G:\Program Files\Exchsrver\MDBDATA\pub1.edb”
    eseutil.exe /d “G:\Program Files\Exchsrver\MDBDATA\pub1.edb”

  • This fixed the corruption, but when tried to restart the “Microsoft Exchange Information Store” service, I got this error in the event log.

    Information Store (3600) First Storage Group: Database recovery failed with error -1216 because it encountered references to a database, ‘G:\Program Files\Exchsrver\MDBDATA\priv1.edb’, which is no longer present. The database was not brought to a Clean Shutdown state before it was removed (or possibly moved or renamed). The database engine will not permit recovery to complete for this instance until the missing database is re-instated. If the database is truly no longer available and no longer required, please contact PSS for further instructions regarding the steps required in order to allow recovery to proceed without this database.

  • This was very confusing because the file actually was present, but it turns out that you need to run a recovery to bring the database back to a clean state. So, I ran the recover command again.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA”

  • This time it complained with the following error.

    Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch, An outstanding database attachment has been detected at the start or end of recovery, but database is missing or does not match attachment info) after 130.0 seconds.

  • This was the result of having to run the repair command without doing a clean shutdown. To resolve that issue I had to run the same recover command with the /i switch to ignore the inconsistencies.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA” /i

  • Finally I had to mount the mailbox and public folder stores by doing the following.
    • Open Server Management (Start -> Server Management)
    • Expand the Advanced Management node
    • Expand the <DomainName> (Exchange) node
    • Expand the Servers node
    • Expand the <ServerName> node
    • Expand the First Storeage Group node
    • Right-click Mailbox Store (<ServerName>) node
    • Click the Mount Store node
    • Right-click Public Folder Store (<ServerName>) node
    • Click the Mount Store node

At this point I was able to connect to Exchange with Outlook and everything appears to be back to normal.