We have tracked down some of the root causes of the performance issues while the site is under heavy load. Unfortunately large port purchases need to lock the entire site for too long. We will not be approving any Port Sale classified ads until Tuesday.
We hope you had a good Black Friday! Our Black Friday Weekend Specials are far from over. We learned a lot today from the increased traffic, made some tweaks, and want the rest of the weekend to be smooth sailing. In order to best accomplish this, we are taking the site down TONIGHT from 1-4 a.m. PST.
Sellers, if you are updating your Sales, please make sure you finish BEFORE 1 a.m. PST.
The site will be down so the server can do some optimizations and give good performance throughout the rest of the day.
More good news! Technical issues should be fixed, and we should now be running at full capacity! In light of the technical difficulties, we are EXTENDING the “Get Paid to Buy” Special through tomorrow, Saturday, November 24th.
For this Special, we will pay you 10 cents store credit for every item you purchase & ship from COMC.com.
Please keep in mind that all Black Friday Weekend Specials only apply to www.COMC.com and do NOT apply to www.CheckOutMyCards.com, as they were designed for the functionality of http://www.COMC.com and do not fit with http://www.CheckOutMyCards.com. Also, we are in the process of retiring http://www.CheckOutMyCards.com and want to encourage you to try out http://www.COMC.com if you haven’t already! We hope you will grow to love your experience on http://www.COMC.com as you have on http://www.CheckOutMyCards.com. We are investing a lot to give you many new reasons to love http://www.COMC.com. If you have been confused regarding the Black Friday Weekend Specials, please contact Customer Relations: email@example.com, 1-800-517-3540. Here are our Extended Customer Service Hours. We want to go the extra mile to ensure you are a happy customer.
We regret any inconvenience you’ve had purchasing today! Not only has our system been taxed, but also the systems of a few vendors we rely on, like PayPal.
Hopefully, extending the Special will make things a bit sweeter and offering some clarification will make things more understandable. We hope that erring on the side of getting you new features and specials as quickly as possible pays off more often than it does not. We appreciate you working with us as we refine the experience for you.
The COMC Team
When we originally discussed what kind of promotion to run for the National Collectors Convention at a meeting back in June, we all felt that a theme surrounding the number 5 would be fun to run and easy to explain. It was our 5th anniversary and we’ve traditionally done number-themed specials at the National. A 550-count box for $55 was very similar to what we did the previous year. The only change was that we were removing the “5 product” restriction to make the deal more appealing, so as a safeguard we added a five-box limit just to be careful. We also decided to open the special up to everyone in case there were people who couldn’t make it to Baltimore to drop cards off.
At the 2011 National we took home about 70,000 cards, and we’d substantially increased in size since then so we estimated we’d be shipping back roughly 100,000-120,000 cards from Maryland and probably receive another 200,000-250,000 in the mail. Our estimate for drop-offs was pretty good, as it turned out, but our guess for mail-ins was 400,000 cards off.
The final tally of items we received was about three quarters of a million, and it all arrived within about a two-week period so the due dates were extremely clustered. On three different days there were over 100,000 cards due to go live.
To stay on top of the processing curve we asked for volunteers from our consignors to have their items delayed in exchange for a discount. We also brought on twenty employees in a combination of full time, part time, and temporary positions. Everyone at COMC has been working extremely hard and I am very proud of the dedication and perseverance of all of the teams in the face of an unprecedented quantity of incoming business. The three biggest due dates were Friday September 7th, Monday the 10th, and Wednesday the 12th. Looking at the raw math we expected to be late on some batches from each of those days, but due to the heroic efforts of the Processing and ID teams we actually were successful at hitting the dates for items due for Friday AND for Monday.
Today is Wednesday, the last large due date, and we will be late on some batches beyond those from the customers who volunteered to be delayed. Here is the relevant information:
- If your batch is delayed we will assign it a new due date one week out, and discount the batch by $5. You’ll see the discount reflected in your store credit when the batch goes live.
- Some new customers sent us batches of thick cards that actually would have been less expensive to send in under our Four Week service. After we finish uploading the last of the submissions from the special we’ll be analyzing the batches we’ve uploaded and we’ll figure out who would have been better off sending in their items as Four Week and give them a retroactive price change.
- We always try to maintain our high standard of identification quality, but we encourage consigners to review their uploaded transactions with extra care considering the volume that has just come through our system.
Thanks, everyone, for making this summer our biggest season yet. We’re looking forward to the fourth quarter of 2012.
As a final note, we’re going to start moderating off-topic comments more proactively. Please keep all comments relevant to the post. If there’s a topic you’d like us to post on that would make for positive discussion, shoot us an email at firstname.lastname@example.org.
Special thanks to Jeremy Williams for writing this blog post.
First 5 years of service: 7.5 million cards processed
First 2 weeks of August: 0.7 million cards submitted
Normally we receive about 100,000 cards in a 2 week period. Due to our 5th Anniversary Special, we received about 7 times that in the first couple weeks of August. Last year we ran a similar special and only got about 100,000 cards. We figured we would probably get 200,000 – 300,000 cards based on the growth of the site and a slightly better promotion. Oops!
We received more cards in two weeks than we have ever processed in a 3 month period.
We had been hoping to launch the new COMC by the end of this month, but that was before we received 10% as many cards in a two week period as we had received in the previous 5 years combined. Unfortunately, we have had to put all development efforts on hold in order to increase our work force by nearly 50%.
Many of you have been wondering what will happen with due dates. As you may have suspected, it won’t be possible for us to process everything in just 5 weeks. That would have required hiring about 60 temp employees. Instead we have opted for a much more healthy balance of permanent full-time, part-time, and temp employees.
As a result we expect to process about 65-80% of the 5th Anniversary Specials on time. The rest will get processed as soon as we can, but they may get delayed by 2-4 weeks. We will be strategically skipping the most difficult submissions so that we can try to satisfy as many customers as possible.
5 Week Guarantee for the 5th Anniversary Special
If we miss a submission’s due date, we will discount the processing fees $5. Also, for each additional week we miss a due date we will discount an additional $5. If you would prefer to get the discount instead of getting your cards posted on time, please let us know, and we will prioritize your submissions accordingly.
4 Week Processing Blackout for September
Because of the enormous backlog, we will not offer our 4 Week processing service during the month of September. If you have stuff you are planning on sending us we will honor the 4 Week guarantee as long as the package is postmarked by this Saturday, September 1st. Any packages that are postmarked after September 1st will be given a due date of October 29th. Starting October 1st we will only allow the new One Month processing service that will cost $0.25 per item.
1 Week Processing Fee Going Up in October
We will keep the $0.35 processing fee for 1 Week service during the month of September, but on October 1st the One Week processing service will cost $0.50 per item.
Between 9pm and 11pm Pacific Time on Thursday the 19th PayPal will be temporarily down, upgrading some tech stuff. During that window PayPal transactions will not be able to be completed.
What will be affected on COMC?
During this two hour window the following things will not work: Store credit purchases by PayPal or credit card; shopping cart purchases with PayPal or credit card.
What will not be affected?
You will still be able to spend store credit as normal on the site – buy cards, flip stuff, request shipments etc. You’ll be able to add things to your shopping cart and your computer will remember your cart contents, but you won’t be able to purchase the cart until after 11pm Pacific. You will still be able to request cashouts or refunds just like always. You will also still be able to email us and tell us how great/awful you think COMC is (though we’ll be going to bed so we won’t write you back until the next day).
The PayPal downtime is only scheduled for two hours, and COMC won’t be down at all.
Wow, today was an amazing team effort. Because of the snow last week we have built up the largest backlog of work ever for both shipping order and processing incoming cards.
At the beginning of the day we had about 10,000 cards that were due to be deposited today that hadn’t even been scanned yet. Normally we try to scan the cards at least a week before they are due so that we have time to make sure they get uploaded to the website without having to worry about any hiccups.
The entire team worked together to do what was necessary to deposit every order before it was due and to ship as many cards as possible today. It was quite an amazing feat, and I am so proud of everyone here.
Tomorrow we have 4 new employees starting. They will have our first official new employee orientation. We also have 4 other employees that just started in the last month or so that will be joining them. 7 of these 8 employees will start full-time on our shipping team, and the eighth employee is focusing on our processing team. Our goal is to over staff shipping so that we can offer the fastest service with the highest quality. Any extra resources in our shipping department will help our processing team get ahead.
We still have several job openings available, so please spread the word.
– Order Fulfillment Clerk
– Office Administrator
– Customer Service
– Quality Control Specialist
– Trading Card Identification Specialist
– Coin Identification Specialist
– Software Development Engineer
– Software Test Engineer
– Network Administrator / General IT Person
Our GM is working on the job descriptions. They will be posted here when he finishes them.
Current Job Openings
Standard orders will take 7-10 days for us to package them, so it will take 10-14 days for them to get delivered in the US.
Bulk orders will take 2-3 weeks for us to package them, so it will take 3-4 weeks for them to be delivered in the US.
The Good News
We have added two new shipping options, Rapid & Registered.
Rapid is just like Standard, only we will package it in 1 business day. It only costs $1.99 + $0.10/card extra for this option.
This is a service that we have been testing out for all of our Amazon orders. It has been working very well.
Registered Mail is now available for very expensive orders. It is sent via USPS Priority Mail with the Registered Mail service. The fee is $12.50 more than our existing Priority Mail option.
First off, I am sorry for all of the recent website issues. We are actively working on it. Thank you for your patience.
Growth & Limitations
Over the past few years nearly every stat has doubled, and we have finally maxed out our server hardware with the current database architecture.
One of the things that has made us so successful is that our site was built from the ground up specifically for card collectors. We don’t use any pre-canned shopping cart software because none of them are designed to handle millions of unique products or offer all of the specialized functionality that our users have come to love. This also means that we are often pioneering new frontier, and we discover our limitations and issues as we grow.
I started to see the writing on the wall last fall when DDearing was trying to update 50,000 prices every day. So I priced out new server hardware. Unfortunately it was going to cost about $100,000 for us to make a significant hardware upgrade. So I took a very close look at our database architecture to see if there were ways we could be more efficient with our hardware.
It is actually quite amazing that the core of our database architecture has held up from its original design when we had only a few thousand items to now with more than 3.7 million items. Fortunately I found some opportunities for significant savings. For example, we currently store about 1,000 bytes of historical data every time an item is modified (e.g. asking price changes, ownership changes, book price changes…). I designed a new storage schema that could reduce that to only about 50 bytes of data for the most common data changes. This should mean that we can process much larger transactions very efficiently. However, this will be a pretty big change to implement. Since we didn’t want to rock the boat for the critical holiday season, I shelved these efforts until January.
Since mid January I have been finishing the design and porting our data to the new system. Now I am nearly to the point where I can start comparing the performance of the new architecture to the old one. However… issues on the current site continue to mount, and over the last few days it has been virtually unusable for many users. To keep the site limping along, I experimented with some temporary fixes. Some of them have helped a little, but ultimately it has just reconfirmed that with the current architecture our hardware cannot really serve more than our 8,000 daily visitors.
Search Engine Crawler Traffic
It used to be that Google made up about 90% of the crawler traffic on our site, and they were very responsible with how they did it. Now we are getting crawled by yahoo, bing, google, and many others. Unfortunately they are not all as responsible as Google. Today I just discovered that that Yahoo Slurp engine has been paging through our data 12 items at a time. Each request has been pegging our server for about 30 seconds. So Yahoo alone was basically monopolizing 1 of our 8 processors.
To address this issue, I have updated our robots.txt file to tell search engines not to page through our data. Instead they should be using our sitemap.xml to get all of the data in the most efficient manner. This should free up server resources so that more real visitors can use the site.
Right now the thing that taxes our servers the most is all of the searching and browsing of cards for sale. We are investigating the possibility of offloading the generation of search results to a farm of servers that will have access to their own cache of the live data. If we can move this to the cloud, we should be able to scale the site to handle virtually any amount of traffic.
In summary, there are several opportunities we have to make the site performance much better, but none of them are quick fixes. Our main focus is on these major improvements, but we will continue to make minor adjustments to the current site if people are commonly experiencing website errors.
Thank you for your patience and understanding. We are all experiencing the growing pains of a successful startup.
CheckOutMyCards.com Founder, CEO & CTO
Today all of the headaches are finally gone. Every card will be disappearing from Dan Dearing’s account one by one throughout the afternoon and evening.
If you wanted to purchase one of them at 8% of book value (or 8% of his asking price on N/A cards), you have a few hours left because we do not have the ability to instantly take all the cards off the site. Unfortunately, this is the only window — and the sale ceases a week shy of its last potential ending date.
We apologize for all of the stress the DDearing sale may have caused with the site slowdown, but are very relieved that the slowdowns due to this will not hamper any of your sales over this holiday season. To all of you that participated in the sale, our hope is that you received some sort of benefit and that you enjoyed participating. Once the site is updated and can handle these types of sales, everyone — including the staff — will be very happy!
Warm wishes throughout the season,
The Staff at CheckOutMyCards.com