If the 3rd time is the charm, what is the 3,333rd?

Version 3333
I just released version 3333 of our internal application today. This is the software that is used by our 50+ employees to manage all of our processing and shipping. Yes, that means that I release more than once a day on average. This is part of our competitive advantage. We are able to quickly adapt to the changing needs of the business, and boy, have there been a lot of those lately.

Thank you for your patience as we continue to catch up with all of the changes. We are rapidly adapting and evolving to meet our new needs. Good thing I put in my 10,000 hours with Microsoft before launching this business. I couldn’t imagine trying to tackle these challenges without so much technology at my fingertips.

We’re Back…in Card Show Mode!

First, a sleepy, furry word from Tim:

Thank you for your patience while we transitioned to our new catalog.  Welcome to a new era for COMC.com!  We are grateful for the outpouring of support we are receiving, and we appreciate feedback as we continue to work together, building our new catalog.

As we mentioned previously, the next few weeks will be a bit rocky with missing data, errors, and many opportunities for improvement.  Everything submitted is for sale as usual, but items that are missing data are harder to browse/search. The current state of the site is “Card Show” mode.  Just like when you go to a card show and browse tables and bins looking for those rare items and special deals, that’s a bit what shopping on COMC will be like for the near future while we rapidly build our database.  We are optimistic about the speed at which you will see improvement, as our data increased by 10% in one day alone!  Your contributions have been significant.  You’ve made a big difference!

Ways To Make It Better

  • The best way to help populate data is still by playing The Challenges, as the data entered there is normalized. When you assign a team name like New York Yankees to a card, you’re essentially bucketing that card with all the other Yankees cards. Be on the look out for new challenges and prizes!
  • You can submit missing data or corrections for existing data for specific cards in your inventory or for sale on the site:
    • Go to the Inventory Manager in your Dashboard.  When you find missing/incorrect data in your inventory, click the Edit link next to the item description:

Inventory Manager

    • When you find missing data in the Search Results, click on the item and, at the top of the page next to the description, click the Edit link:

Search Results

    • From Edit (from your Search or Inventory), you will get a popup where you can enter data that is missing or needs correction:

Catalog Details

    • Please keep in mind, edits you make to a card are not normalized, meaning if you enter the correction to Teams: “New York Yankees” or “yankees” or “new yyok yankz”, the system won’t bucket it with other Yankees cards. It is still very helpful to use the new Edit feature, as it flags the card as incorrect and lets us know your correction suggestion.  It is just a less efficient process than playing the Challenges and will take more time to correct, as someone here will need to review each correction.
    • Please DO NOT USE the Report Error function, as that is used for reporting ID errors. These will go to the wrong team.

As we said previously, we won’t charge storage fees for the month of April.  We expect to have player and team search functionality fully restored by the end of April.  In May and June, we will be working on more advanced search functionality, such as the type of memorabilia.

Suggested List Price (SLP)

We were hoping to have the beta version of our pricing suggestions live today, but we ran out of time.  We will release it soon and let you know more as it develops.  Once our SLPs are live, contributors who earned at least 20K in Challenge points will receive free SLPs while it is in beta.  In the meantime, we have turned on the “Going Rate” feature.  This feature may help you price your cards by showing you how many like items are on the site and the range at which they are being priced.  While we have previously had quite a few requests for this feature, we have hesitated to turn it on to prevent a race-to-the-bottom with pricing.  We will be monitoring how this feature is being used, and if it remains good for the COMC ecosystem, we may choose to keep it enabled.

Going Rate

Going Rate Feature

The other way to receive helpful pricing information is to click Sales History to the right of the items in your inventory. This activates a popup which tells you the condition, how many are for sale, how many have sold and shipped, and at what price.

Sales History Feature

Sales History Feature

We hope you enjoy COMC while it’s in Card Show mode, and we will continue to let you know as things develop here.  Please continue to contribute to the catalog and let us know what we can do to create the best collecting experience!

10 X the traffic = some site hiccups

Let’s start with the good news: we are getting TEN TIMES our normal traffic today!

Image

This means we have the necessary traffic to reach our goal of increasing sales by 10 times (as part of the previously announced “Perfect 10 Black Friday Special”).

As for the not so good news, as you may have noticed, there have been some site issues today.  Please know that we are working very hard to resolve them all and expect to be fully up to speed very soon!

Even so, please don’t hesitate to contact our Customer Relations Department if you need help with anything.  You can reach us until midnight tonight at staff@comc.com or by 1-800-517-3540. Here are our Extended Customer Service Hours. Thank you for your patience!

Customer Service

We’ve recently switched to a new server specifically for customer service, and this morning we discovered that some of our replies weren’t being sent out and some were being caught in “spam” filters upon arrival to the customer’s email inbox.  The issue should be fixed now, but if you’ve emailed us in the last few days and haven’t received a reply please email us again and let us know so we can take care of you, and please also take a look at your spam email folder to see if our reply was mistakenly filtered out.

Thanks!

2010 Q3 Stats & Performance Notes

Another quarter with steady growth on all fronts.

Website Growth & Performance Demands

It is interesting to follow the average page views per month from year to year for .
2007 Q3 – 0.1 million views/month
2008 Q3 – 1.1 million views/month
2009 Q3 – 2.5 million views/month
2010 Q3 – 4.4 million views/month

As the traffic continues to grow we need to make performance optimizations to handle the workload. In a post on Monday I alluded to some changes that might be noticeable by repeat visitors. The major update that I made was a performance optimization that affected nearly every page of the website. As a result many pages started loading significantly faster.

We migrated from using ASP.Net’s stored procedure profile provider for custom user information to using direct queries with Microsoft’s LINQ to SQL. This made it very easy for us to make highly targeted, optimized, and secure queries just when we needed the information.

This is just the first of many performance improvements we plan to make to the site. Over the next few weeks I will discuss a monumental change that we have in the works. Hint: The future is cloudy.

Getting email to go to aol.com addresses

At last… I found the needle in the haystack. Wow, what a pain. We got our new email server all configured about 10 days ago, and it seemed to be working great for just about everyone. However, no one from with an aol.com email address was getting our emails.

After many many hours of searching I finally traced down what the issue was. This forum post from tek-tips.com was very helpful.

Quick Summary
AOL required reverse DNS to be configured for the IP address that is sending the email messages.

Full Details
This had all been configured correctly with our old email server, but when we migrated the the new email server, I had to change some of our firewall settings. The email server happens to accept request from a couple different IP addresses. So it was successfully receiving emails that were routed through mx1.checkoutmycards.com, but when it was sending emails it happen to go out through a different IP address.

I realized this by doing a “What is my IP” search and seeing that it didn’t match mx1.checkoutmycards.com. All I had to do was change one firewall setting so that outgoing emails got sent through the same IP address as the one that had reverse DNS configured.

I even created an @aol.com email address just to test the scenario and make sure things worked. Hopefully all of you at aol.com can now receive our emails.

Preparing for Growth

At COMC.com we have been working hard to strategically prepare for growth.

Better Storage System
We recently invented a new storage system to complement our inventory management system. This new system will allow us to easily grow from 80,000 to 1 Million+ cards in our current facilities. The storage system is fire retardant, theft deterring, earth quake resilient, and massively scalable. All this and we can still access any card in seconds.

Better Servers
We recently installed new servers that have more than 10 times the horse power of our previous server, and we are ready to scale with more servers as traffic increases.

Better Database Performance
We recently improved indexes used to perform searches so that all searches are fast, no matter how you choose to sort them. Nearly every search is now being completed in less than 0.20 seconds.

Better HTML Markup
We recently reduced the markup used on our search pages so that the views that have hover pop-ups (all views other than the details view) render in 1/10th the time they used to.

Better Page Response Time
We recently enabled gzip compression on our server so that the total bits transferred to the browser were reduced by 1/2. This makes the site feel a lot more responsive.

How I recovered Exchange Server 2003

This is probably not very interesting to sports card collectors, but if you recently had a power failure or just got impatient with your Exchange server and did a hard reboot, you might find this useful.

Now that we have our new servers up and running, I needed to move our old server to be next to the new ones. The old server is just used as our mail server running Exchange Server 2003. Well, after giving it a good 10 minutes to shut down, it still hadn’t completely shut down, and I was getting impatient (it was 2:30 AM). So, I just pulled the power cable, and moved the server.

When the server finally came back up Exchange complained with this error in the event log.

Information Store (3812) First Storage Group: Database G:\Program Files\Exchsrver\MDBDATA\priv1.edb: Page 3031 (0x00000bd7) failed verification due to a flush-order dependency mismatch. This page should have flushed before page 7216 (0x00001c30), but the latter page has instead flushed first. Recovery/restore will fail with error -255. If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware “losing” one or more flushes on one or both of these pages sometime in the past. Please contact your hardware vendor for further assistance diagnosing the problem.

I have a default Microsoft Windows Small Business Server 2003 install with the Exchange data files in “G:\Program Files\Exchsrver\MDBDATA\”.

To fix the issue I did the following:

  • Made a backup of the entire “G:\Program Files\Exchsrver\MDBDATA\” directory
  • Found this article about eseutil
  • From “G:\Program Files\Exchsrver\bin” I ran the following command.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA”

  • This complained about priv1.edb being corrupt, so I ran the following commands to repair and defragment the edb files.

    eseutil.exe /p “G:\Program Files\Exchsrver\MDBDATA\priv1.edb”
    eseutil.exe /d “G:\Program Files\Exchsrver\MDBDATA\priv1.edb”
    eseutil.exe /p “G:\Program Files\Exchsrver\MDBDATA\pub1.edb”
    eseutil.exe /d “G:\Program Files\Exchsrver\MDBDATA\pub1.edb”

  • This fixed the corruption, but when tried to restart the “Microsoft Exchange Information Store” service, I got this error in the event log.

    Information Store (3600) First Storage Group: Database recovery failed with error -1216 because it encountered references to a database, ‘G:\Program Files\Exchsrver\MDBDATA\priv1.edb’, which is no longer present. The database was not brought to a Clean Shutdown state before it was removed (or possibly moved or renamed). The database engine will not permit recovery to complete for this instance until the missing database is re-instated. If the database is truly no longer available and no longer required, please contact PSS for further instructions regarding the steps required in order to allow recovery to proceed without this database.

  • This was very confusing because the file actually was present, but it turns out that you need to run a recovery to bring the database back to a clean state. So, I ran the recover command again.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA”

  • This time it complained with the following error.

    Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch, An outstanding database attachment has been detected at the start or end of recovery, but database is missing or does not match attachment info) after 130.0 seconds.

  • This was the result of having to run the repair command without doing a clean shutdown. To resolve that issue I had to run the same recover command with the /i switch to ignore the inconsistencies.

    eseutil.exe /r E00 /l”G:\Program Files\Exchsrver\MDBDATA” /i

  • Finally I had to mount the mailbox and public folder stores by doing the following.
    • Open Server Management (Start -> Server Management)
    • Expand the Advanced Management node
    • Expand the <DomainName> (Exchange) node
    • Expand the Servers node
    • Expand the <ServerName> node
    • Expand the First Storeage Group node
    • Right-click Mailbox Store (<ServerName>) node
    • Click the Mount Store node
    • Right-click Public Folder Store (<ServerName>) node
    • Click the Mount Store node

At this point I was able to connect to Exchange with Outlook and everything appears to be back to normal.