Customer Service

We’ve recently switched to a new server specifically for customer service, and this morning we discovered that some of our replies weren’t being sent out and some were being caught in “spam” filters upon arrival to the customer’s email inbox.  The issue should be fixed now, but if you’ve emailed us in the last few days and haven’t received a reply please email us again and let us know so we can take care of you, and please also take a look at your spam email folder to see if our reply was mistakenly filtered out.

Thanks!

2010 Q3 Stats & Performance Notes

Another quarter with steady growth on all fronts.

Website Growth & Performance Demands

It is interesting to follow the average page views per month from year to year for .
2007 Q3 – 0.1 million views/month
2008 Q3 – 1.1 million views/month
2009 Q3 – 2.5 million views/month
2010 Q3 – 4.4 million views/month

As the traffic continues to grow we need to make performance optimizations to handle the workload. In a post on Monday I alluded to some changes that might be noticeable by repeat visitors. The major update that I made was a performance optimization that affected nearly every page of the website. As a result many pages started loading significantly faster.

We migrated from using ASP.Net’s stored procedure profile provider for custom user information to using direct queries with Microsoft’s LINQ to SQL. This made it very easy for us to make highly targeted, optimized, and secure queries just when we needed the information.

This is just the first of many performance improvements we plan to make to the site. Over the next few weeks I will discuss a monumental change that we have in the works. Hint: The future is cloudy.

Getting email to go to aol.com addresses

At last… I found the needle in the haystack. Wow, what a pain. We got our new email server all configured about 10 days ago, and it seemed to be working great for just about everyone. However, no one from with an aol.com email address was getting our emails.

After many many hours of searching I finally traced down what the issue was. This forum post from tek-tips.com was very helpful.

Quick Summary
AOL required reverse DNS to be configured for the IP address that is sending the email messages.

Full Details
This had all been configured correctly with our old email server, but when we migrated the the new email server, I had to change some of our firewall settings. The email server happens to accept request from a couple different IP addresses. So it was successfully receiving emails that were routed through mx1.checkoutmycards.com, but when it was sending emails it happen to go out through a different IP address.

I realized this by doing a “What is my IP” search and seeing that it didn’t match mx1.checkoutmycards.com. All I had to do was change one firewall setting so that outgoing emails got sent through the same IP address as the one that had reverse DNS configured.

I even created an @aol.com email address just to test the scenario and make sure things worked. Hopefully all of you at aol.com can now receive our emails.

Preparing for Growth

At COMC.com we have been working hard to strategically prepare for growth.

Better Storage System
We recently invented a new storage system to complement our inventory management system. This new system will allow us to easily grow from 80,000 to 1 Million+ cards in our current facilities. The storage system is fire retardant, theft deterring, earth quake resilient, and massively scalable. All this and we can still access any card in seconds.

Better Servers
We recently installed new servers that have more than 10 times the horse power of our previous server, and we are ready to scale with more servers as traffic increases.

Better Database Performance
We recently improved indexes used to perform searches so that all searches are fast, no matter how you choose to sort them. Nearly every search is now being completed in less than 0.20 seconds.

Better HTML Markup
We recently reduced the markup used on our search pages so that the views that have hover pop-ups (all views other than the details view) render in 1/10th the time they used to.

Better Page Response Time
We recently enabled gzip compression on our server so that the total bits transferred to the browser were reduced by 1/2. This makes the site feel a lot more responsive.

How I recovered Exchange Server 2003

This is probably not very interesting to sports card collectors, but if you recently had a power failure or just got impatient with your Exchange server and did a hard reboot, you might find this useful.

Now that we have our new servers up and running, I needed to move our old server to be next to the new ones. The old server is just used as our mail server running Exchange Server 2003. Well, after giving it a good 10 minutes to shut down, it still hadn’t completely shut down, and I was getting impatient (it was 2:30 AM). So, I just pulled the power cable, and moved the server.

When the server finally came back up Exchange complained with this error in the event log.

Information Store (3812) First Storage Group: Database G:\Program Files\Exchsrver\MDBDATA\priv1.edb: Page 3031 (0x00000bd7) failed verification due to a flush-order dependency mismatch. This page should have flushed before page 7216 (0x00001c30), but the latter page has instead flushed first. Recovery/restore will fail with error -255. If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware “losing” one or more flushes on one or both of these pages sometime in the past. Please contact your hardware vendor for further assistance diagnosing the problem.

I have a default Microsoft Windows Small Business Server 2003 install with the Exchange data files in “G:\Program Files\Exchsrver\MDBDATA\”.

To fix the issue I did the following:

  • Made a backup of the entire “G:\Program Files\Exchsrver\MDBDATA\” directory
  • Found this article about eseutil
  • From “G:\Program Files\Exchsrver\bin” I ran the following command.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA”

  • This complained about priv1.edb being corrupt, so I ran the following commands to repair and defragment the edb files.

    eseutil.exe /p “G:\Program Files\Exchsrver\MDBDATA\priv1.edb”
    eseutil.exe /d “G:\Program Files\Exchsrver\MDBDATA\priv1.edb”
    eseutil.exe /p “G:\Program Files\Exchsrver\MDBDATA\pub1.edb”
    eseutil.exe /d “G:\Program Files\Exchsrver\MDBDATA\pub1.edb”

  • This fixed the corruption, but when tried to restart the “Microsoft Exchange Information Store” service, I got this error in the event log.

    Information Store (3600) First Storage Group: Database recovery failed with error -1216 because it encountered references to a database, ‘G:\Program Files\Exchsrver\MDBDATA\priv1.edb’, which is no longer present. The database was not brought to a Clean Shutdown state before it was removed (or possibly moved or renamed). The database engine will not permit recovery to complete for this instance until the missing database is re-instated. If the database is truly no longer available and no longer required, please contact PSS for further instructions regarding the steps required in order to allow recovery to proceed without this database.

  • This was very confusing because the file actually was present, but it turns out that you need to run a recovery to bring the database back to a clean state. So, I ran the recover command again.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA”

  • This time it complained with the following error.

    Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch, An outstanding database attachment has been detected at the start or end of recovery, but database is missing or does not match attachment info) after 130.0 seconds.

  • This was the result of having to run the repair command without doing a clean shutdown. To resolve that issue I had to run the same recover command with the /i switch to ignore the inconsistencies.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA” /i

  • Finally I had to mount the mailbox and public folder stores by doing the following.
    • Open Server Management (Start -> Server Management)
    • Expand the Advanced Management node
    • Expand the <DomainName> (Exchange) node
    • Expand the Servers node
    • Expand the <ServerName> node
    • Expand the First Storeage Group node
    • Right-click Mailbox Store (<ServerName>) node
    • Click the Mount Store node
    • Right-click Public Folder Store (<ServerName>) node
    • Click the Mount Store node

At this point I was able to connect to Exchange with Outlook and everything appears to be back to normal.