We have tracked down some of the root causes of the performance issues while the site is under heavy load. Unfortunately large port purchases need to lock the entire site for too long. We will not be approving any Port Sale classified ads until Tuesday.

None of our Black Friday Weekend promotions apply to Port Sales anyhow. Sorry for the inconvenience. We recommend running Promotions this weekend instead of trying to do any Port Sales.

We hope you had a good Black Friday!  Our Black Friday Weekend Specials are far from over.  We learned a lot today from the increased traffic, made some tweaks, and want the rest of the weekend to be smooth sailing.  In order to best accomplish this, we are taking the site down TONIGHT from 1-4 a.m. PST. 

Sellers, if you are updating your Sales, please make sure you finish BEFORE 1 a.m. PST.

The site will be down so the server can do some optimizations and give good performance throughout the rest of the day.

Peace.

When we originally discussed what kind of promotion to run for the National Collectors Convention at a meeting back in June, we all felt that a theme surrounding the number 5 would be fun to run and easy to explain. It was our 5th anniversary and we’ve traditionally done number-themed specials at the National. A 550-count box for $55 was very similar to what we did the previous year. The only change was that we were removing the “5 product” restriction to make the deal more appealing, so as a safeguard we added a five-box limit just to be careful. We also decided to open the special up to everyone in case there were people who couldn’t make it to Baltimore to drop cards off.

At the 2011 National we took home about 70,000 cards, and we’d substantially increased in size since then so we estimated we’d be shipping back roughly 100,000-120,000 cards from Maryland and probably receive another 200,000-250,000 in the mail. Our estimate for drop-offs was pretty good, as it turned out, but our guess for mail-ins was 400,000 cards off.

The final tally of items we received was about three quarters of a million, and it all arrived within about a two-week period so the due dates were extremely clustered. On three different days there were over 100,000 cards due to go live.

To stay on top of the processing curve we asked for volunteers from our consignors to have their items delayed in exchange for a discount. We also brought on twenty employees in a combination of full time, part time, and temporary positions. Everyone at COMC has been working extremely hard and I am very proud of the dedication and perseverance of all of the teams in the face of an unprecedented quantity of incoming business. The three biggest due dates were Friday September 7th, Monday the 10th, and Wednesday the 12th. Looking at the raw math we expected to be late on some batches from each of those days, but due to the heroic efforts of the Processing and ID teams we actually were successful at hitting the dates for items due for Friday AND for Monday.

Today is Wednesday, the last large due date, and we will be late on some batches beyond those from the customers who volunteered to be delayed. Here is the relevant information:

- If your batch is delayed we will assign it a new due date one week out, and discount the batch by $5. You’ll see the discount reflected in your store credit when the batch goes live.

- Some new customers sent us batches of thick cards that actually would have been less expensive to send in under our Four Week service. After we finish uploading the last of the submissions from the special we’ll be analyzing the batches we’ve uploaded and we’ll figure out who would have been better off sending in their items as Four Week and give them a retroactive price change.

- We always try to maintain our high standard of identification quality, but we encourage consigners to review their uploaded transactions with extra care considering the volume that has just come through our system.

Thanks, everyone, for making this summer our biggest season yet. We’re looking forward to the fourth quarter of 2012.

As a final note, we’re going to start moderating off-topic comments more proactively. Please keep all comments relevant to the post. If there’s a topic you’d like us to post on that would make for positive discussion, shoot us an email at staff@comc.com.

Special thanks to Jeremy Williams for writing this blog post.

First 5 years of service: 7.5 million cards processed
First 2 weeks of August: 0.7 million cards submitted

Normally we receive about 100,000 cards in a 2 week period. Due to our 5th Anniversary Special, we received about 7 times that in the first couple weeks of August. Last year we ran a similar special and only got about 100,000 cards. We figured we would probably get 200,000 – 300,000 cards based on the growth of the site and a slightly better promotion. Oops!

We received more cards in two weeks than we have ever processed in a 3 month period.

We had been hoping to launch the new COMC by the end of this month, but that was before we received 10% as many cards in a two week period as we had received in the previous 5 years combined. Unfortunately, we have had to put all development efforts on hold in order to increase our work force by nearly 50%.

Due Dates

Many of you have been wondering what will happen with due dates. As you may have suspected, it won’t be possible for us to process everything in just 5 weeks. That would have required hiring about 60 temp employees. Instead we have opted for a much more healthy balance of permanent full-time, part-time, and temp employees.

As a result we expect to process about 65-80% of the 5th Anniversary Specials on time. The rest will get processed as soon as we can, but they may get delayed by 2-4 weeks. We will be strategically skipping the most difficult submissions so that we can try to satisfy as many customers as possible.

5 Week Guarantee for the 5th Anniversary Special

If we miss a submission’s due date, we will discount the processing fees $5. Also, for each additional week we miss a due date we will discount an additional $5. If you would prefer to get the discount instead of getting your cards posted on time, please let us know, and we will prioritize your submissions accordingly.

4 Week Processing Blackout for September

Because of the enormous backlog, we will not offer our 4 Week processing service during the month of September. If you have stuff you are planning on sending us we will honor the 4 Week guarantee as long as the package is postmarked by this Saturday, September 1st. Any packages that are postmarked after September 1st will be given a due date of October 29th. Starting October 1st we will only allow the new One Month processing service that will cost $0.25 per item.

1 Week Processing Fee Going Up in October

We will keep the $0.35 processing fee for 1 Week service during the month of September, but on October 1st the One Week processing service will cost $0.50 per item.

Between 9pm and 11pm Pacific Time on Thursday the 19th PayPal will be temporarily down, upgrading some tech stuff.  During that window PayPal transactions will not be able to be completed.

What will be affected on COMC?
During this two hour window the following things will not work: Store credit purchases by PayPal or credit card; shopping cart purchases with PayPal or credit card.

What will not be affected?
You will still be able to spend store credit as normal on the site – buy cards, flip stuff, request shipments etc.  You’ll be able to add things to your shopping cart and your computer will remember your cart contents, but you won’t be able to purchase the cart until after 11pm Pacific.  You will still be able to request cashouts or refunds just like always.  You will also still be able to email us and tell us how great/awful you think COMC is (though we’ll be going to bed so we won’t write you back until the next day).

The PayPal downtime is only scheduled for two hours, and COMC won’t be down at all.
Thanks!

First off, I am sorry for all of the recent website issues. We are actively working on it. Thank you for your patience.

Growth & Limitations
Over the past few years nearly every stat has doubled, and we have finally maxed out our server hardware with the current database architecture.

One of the things that has made us so successful is that our site was built from the ground up specifically for card collectors. We don’t use any pre-canned shopping cart software because none of them are designed to handle millions of unique products or offer all of the specialized functionality that our users have come to love. This also means that we are often pioneering new frontier, and we discover our limitations and issues as we grow.

New Architecture
I started to see the writing on the wall last fall when DDearing was trying to update 50,000 prices every day. So I priced out new server hardware. Unfortunately it was going to cost about $100,000 for us to make a significant hardware upgrade. So I took a very close look at our database architecture to see if there were ways we could be more efficient with our hardware.

It is actually quite amazing that the core of our database architecture has held up from its original design when we had only a few thousand items to now with more than 3.7 million items. Fortunately I found some opportunities for significant savings. For example, we currently store about 1,000 bytes of historical data every time an item is modified (e.g. asking price changes, ownership changes, book price changes…). I designed a new storage schema that could reduce that to only about 50 bytes of data for the most common data changes. This should mean that we can process much larger transactions very efficiently. However, this will be a pretty big change to implement. Since we didn’t want to rock the boat for the critical holiday season, I shelved these efforts until January.

Since mid January I have been finishing the design and porting our data to the new system. Now I am nearly to the point where I can start comparing the performance of the new architecture to the old one. However… issues on the current site continue to mount, and over the last few days it has been virtually unusable for many users. To keep the site limping along, I experimented with some temporary fixes. Some of them have helped a little, but ultimately it has just reconfirmed that with the current architecture our hardware cannot really serve more than our 8,000 daily visitors.

Search Engine Crawler Traffic
It used to be that Google made up about 90% of the crawler traffic on our site, and they were very responsible with how they did it. Now we are getting crawled by yahoo, bing, google, and many others. Unfortunately they are not all as responsible as Google. Today I just discovered that that Yahoo Slurp engine has been paging through our data 12 items at a time. Each request has been pegging our server for about 30 seconds. So Yahoo alone was basically monopolizing 1 of our 8 processors.

To address this issue, I have updated our robots.txt file to tell search engines not to page through our data. Instead they should be using our sitemap.xml to get all of the data in the most efficient manner. This should free up server resources so that more real visitors can use the site.

Search Scalability
Right now the thing that taxes our servers the most is all of the searching and browsing of cards for sale. We are investigating the possibility of offloading the generation of search results to a farm of servers that will have access to their own cache of the live data. If we can move this to the cloud, we should be able to scale the site to handle virtually any amount of traffic.

In summary, there are several opportunities we have to make the site performance much better, but none of them are quick fixes. Our main focus is on these major improvements, but we will continue to make minor adjustments to the current site if people are commonly experiencing website errors.

Thank you for your patience and understanding. We are all experiencing the growing pains of a successful startup.

Tim Getsch
CheckOutMyCards.com Founder, CEO & CTO