The team bonding isn’t limited to after-hours; several different groups of engineers get together at lunch throughout the week to play board and card games – Tuesday is usually dominated by Dominion, and Thursday is all about Bridge, Wizard, and other trick-taking games.

It doesn’t take a Wayfair engineer to see that there’s some synergy between our small lunch gatherings and our big after-hours blowouts, so we’re putting together our first Wayfair Engineering Open Board Game Night. We’re reaching out to the Boston engineering community to join us on our 24th floor (check out the sweet view) on Wednesday May 23rd from 6:30 to 9:00 pm. There will be free food and brew, along with a selection of our favorite games.

Register here to spend some quality time with the Wayfair engineering team. Feel free to bring your own favorite games and to bring your friends – but make sure to register them too, so we’re sure to have plenty of pizza to go around.  Since we will be ordering food and beverages based upon the number of registered users, Please don’t sign up unless you will make a best effort to attend.

Nothing happened to it for about a year, until one day in early November of 2011. The port to PHP had been successful and we were gradually moving traffic from our ASP scripted web pages to the new PHP platform. The holiday season was approaching and the PHP platform was taking more and more traffic every day. The web platform started showing lots of faults and things were getting worse with the increase in traffic. Turns out that the FreeTDS driver we use in PHP scripts to access our MSSQL databases had trouble making connections under high traffic conditions. The efforts to resolve the issues with the driver produced no immediate results, and we started to brainstorm for alternative ways to solve the problem. Wayfair Engineering had to come up with something soon. The biggest chunk of the DB queries affected were identified, and it looked like our inventory lookup query was the easiest one to divert to an alternative solution. Without inventory lookups, the FreeTDS driver seemed capable of handling the expected traffic.

A few ideas came up. One of them was to take that year-old WCF prototype and turn it into a proxy service for MSSQL inventory lookups. So I started adding the necessary code to implement the inventory query as a pass-through operation. So, essentially, the PHP client script would make a REST API call to the new service over HTTP passing the query as a parameter and get the query execution result in the response. I added code to implement the database connection logic from our standard method and tested that. I also needed to provide a good format for the response and wrote a function that formats a dataset object directly into a JSON object to facilitate the use of the response in PHP. Once all this was working I had to test it under heavy traffic conditions, so our DBAs provided captured query traffic from the production databases to use as realistic load. I wrote a simple client to call the service with the captured queries and monitored the performance, memory and CPU. At first it looked promising, but eventually the memory use rose to high levels, so I checked the code and found ways to eliminate potential memory leaks. After that, the service was ready for a trial in the production environment, so the PHP team built a client function and prepared to send the traffic of one of the web servers to the new service. The service performed reasonably well and more and more servers were added to the trial until all the inventory calls were using the service. The response time wasn’t as good as the one from the FreeTDS driver’s call, but it was close enough and it didn’t cause connection faults. The new service was named “http query”. Once the service was running in its own application server, it took care of our high traffic worries and sailed through “Black Friday”, “Cyber Monday” and the rest of the holiday season workloads without any major issues.

This experience made Wayfair Engineering aware of alternative ways to solve scalability problems. Since then, a few new service oriented solutions have spawned from the little program that could. One solution uses the same concept to process order data to identify fraudulent trends. Another effort by the Recommendations team built upon the same foundations to implement a more efficient way to retrieve inventory status for multiple products in a single request.

The Wayfair Engineering Express has left the station, and I’m so glad I got a great window seat. Long live the little program that could!

Around 10 a.m., we started to receive alerts from our internal and external monitoring indicating that something out of the ordinary was up with the site.  Response times were spiking and, within 15 minutes, we were returning error pages apparently caused by timeouts from our back-end.  Everyone scrambled to determine what was causing the site outage:

• Did we just push bad code?
• Did something break?
• Was this a DOS attack?

Our analytics team was able to determine that referral traffic from Yahoo! had spiked by over 500%.  And they quickly realized that the Inc. story had been picked up by Yahoo! – thus leading us to conclude that the resulting interest was causing the capacity issues we were seeing.

Our site is designed to try to handle large traffic loads and we successfully handled the past holiday rush, which was significantly higher than our typical April load.  However, within the first 15 minutes, we had sustained a forty-fold increase in the amount of traffic to certain pages which brought our total load well beyond what we experienced on Cyber Monday, which had been our biggest day to date.  We had reached our capacity limit which caused the site to slow down for everyone and, at the peak, to return error pages for about half of all requests.

We had sufficient web server capacity, so that wasn’t the issue.  As we looked closer, we found that we had reached a connection pool limit within our load balancing layer, something that we had not experienced previously, so we quickly increased the cap.  By around 10:45 a.m., the error pages had stopped and the site started returning to normal.

Or so we thought…

Traffic continued to grow and we quickly reached another plateau, at twice our normal levels.  We were no longer returning error pages, but load times were still high.  We could see that something was overloading our front-end caching databases, causing application calls to queue and eventually time out.

Over the next hour, we determined that one routine in particular was causing most of our database problems.  The particular call provides us with geo-location information that we use, amongst other things, to provide localized content for our Get it Near Me! service.  Fortunately, our application makes use of internal “feature knobs”, which allow us to dial up and down the percentage of users who see elements on our site without requiring code changes, so we simply set this function to appear for zero user.  With this feature turned off, site performance returned to normal, but this time at four times our normal traffic volume.  About 90 minutes later, we had new code in production that resolved the issue and allowed us to turn the feature back on.

Overall, site traffic stayed up for the rest of the day and through 11am the following morning.  At that point, we saw a sharp drop down in traffic with levels returning to what we would expect for a normal April morning.  Our news story had cycled off of Yahoo!’s news page, thus ending the 24-hour news cycle and distinguishing Tuesday as having the largest overall traffic volume in our history, beating the previous Cyber Monday by over 20%.

We are now officially a member of a somewhat exclusive club of large web companies who have fallen victim to their own success.   The real test is to learn from the past, avoid the same mistakes, and continue to improve.  Yahoo! posted our story again on Sunday as part of their week in review, causing that day to have our fourth highest traffic count behind only Tuesday’s spike and the previous two Cyber Monday’s.  The site worked fine.

With Sunday’s spike in traffic successfully behind us, we’re encouraged by our team’s ability to adapt quickly and improve, but by no means are we thinking our job is done – instead, we’re now ready and eager to face our next challenge!

(Requests per second over time)

The hardware came in, we loaded up a FreeBSD 9.0 Release install CD, and the RAID controller was actually detected by an entirely different driver, the mpt driver.  It turns out there is a bug in the release version of that driver, which we were able to get around fairly easily (more on that later).  Still, none of the drivers in the release version would detect the raid controller. Eventually we were able to find a project on FreeBSD’s SVN site where a new version of the mfi driver was being developed that supports the new line of Dell PERC cards.  Below is a quick how-to on making an install CD that will work perfectly with the new RAID controllers.  After we were able to get a clean install, we then discovered that the Broadcom LOM NIC is detected, but doesn’t function, so we just installed an Intel Pro1000 PCI NIC.  We later discovered that Intel has source code for a newer version of the igb driver for the optional Intel LOM NICs available here.  We haven’t tested this yet, but the same method of injecting new driver code we use below should work for this driver, as well.

Ok, start with a clean install of FreeBSD 9.0 Release. (We used a VM, but physical hardware is fine, too.)  Just make sure you have plenty of drive space, as we ended up using about 20GB by the time we were done

1. Download the stable version of source code, which we’ll use when building world and our kernel to build our install CD

csup stable-supfile

2. Do a standard buildworld.  We don’t need to install it, but the output of this is used when building the install CD

cd /usr/src/

make buildworld

3. Now we need to get the new driver source code.  The SVN checkout link is here, and here is the link to the websvn page if you want to take a look at the commit logs.  Once you have all the files downloaded, put them in /usr/src/sys/dev/mfi folder, overwriting all existing files.

4. The new code actually added 2 new C files, so we need to add them to one of the included make files so when we build our kernel, it will see these new files.

Fire up your favorite text editor and edit /usr/src/sys/conf/files

Find the lines with the mfi driver files by searching for mfi, and add the following lines after those:

dev/mfi/mfi_syspd.c     optional mfi

dev/mfi/mfi_tbolt.c       optional mfi

5. Now, since we are using stable source files, we will get the updated mpt driver to get around the bug were this driver incorrectly tries to attach to our RAID controller   Alternatively, you could edit the GENERIC kernel config file and comment out the mpt driver altogether.  Now, just build the kernel

cd /usr/src

make buildkernel KERNCONF=GENERIC

6. Now we just need to make our release, which will generate our install ISO files.

cd /usr/src/release

make release

7. ISO files are in /usr/obj/usr/src/release, pull them off then either just boot from an iDRAC with it, or burn to a CD.

For good measure, we installed iozone on the server in an attempt to stress test the RAID controller and drivers, and had no issues with the drivers or errors in the messages.log file. It’s worth noting that the stats below are based on an 8x 15k SAS disk RAID 5 array.

We hope that you are as excited about the new line of servers as we are, and that this article is helpful to not only those trying to get FreeBSD running on Dell’s new hardware, but also for those that need a custom install CD for other reasons as well.  As always, thanks for reading.

At the onset of this project our cache system was essentially a two-tiered system with local APC caching on all servers, backed up by remote memcached servers.  One benefit of APC is that during code execution and page loads we can very easily set and get values in cache, which makes its use very dynamic.  CDB works very well on heavy load for just read operations, but fails if multiple processes attempt to update or write new key/value pairs on the fly.  Therefore we envisioned a system in which both APC and CDB were utilized as local cache options, with memcache as a backup for both.  CDB would handle our constants, which in general are static and do not change often.  We update them once an hour with an asynchronous job, which keeps them fresh enough for our needs.    If CDB could handle the constant load well, we planned to move our language resources (how we localize our site for other countries) over from APC to CDB as well and observe how performance was affected.

While the initial move of constants to CDB did not show a huge performance improvement s, we did notice some small gains once we added language resources to CDB.  On top of a marginal decrease in load time, CDB performed much better under load, showing significantly less periodic behavior than what we had seen previously with APC.

Below are graphs of homepage performance over an 8 hour window, before and after our change.  The huge drop seen in the first graph is due to clearing APC on our servers.  Clearing APC on a somewhat regular basis became necessary to keep our site performance stable under load.  With CDB taking over a huge portion of the storage that lived in APC previously, we have achieved a level of stability that was not possible with the APC user cache.

Before (November):

After (February):

Ignoring the blue line from the initial graph, what we see here is twofold.  First off, over the course of a couple months, our team has made huge strides to improve the performance of our servers and our code, decreasing the overall load time for the Wayfair.com homepage.  CDB contributed to this vast decrease in load times, but many other factors/projects contributed as well.  CDB’s main contribution comes with improved stability and reliability for our sites.  Moving forward, we have more ideas in the works to make our pages faster and more stable, and we plan to continue our efforts in this area well into the future.  As one of our mottos states very clearly, we are never done.

The biggest challenge we face is the number of different ways of communication and data formats we have to support to accomplish the integration with our partners. For example, with Amazon we build XML messages and send them using web services, for Walmart we create XML transaction files to upload via FTP, for Sears we submit XML transactions via REST API services, for eBay we use SOAP messages over HTTPS, and for Buy.com we generate tab-delimited text files and send them via FTP.

We’ve established a baseline of data structures and processing steps that we can use as a starting point for each new partner integration. But we didn’t get there overnight or even close to the beginning of this endeavor. For our first few partnerships we tried to accommodate their specifications independently, and that’s when we realized we wouldn’t get too far if we kept doing it that way forever. So we started building a common set of rules and data structures to use as a foundation for future integrations.

We also started building consolidated services designed to provide the necessary data for all our partners without duplicating efforts. For example, when we had about six partnerships already integrated, we had six different systems for preparing inventory information, and in many cases we were retrieving the same exact data six times every night. This was obviously overwhelming our servers and limiting our ability to scale and grow our partnerships. So we created a single system to function as an inventory data service for all partners. Before the improvements the individual inventory processes took between 8 and 16 hours to run. After the consolidation, we got the whole process done in under 4 hours. Eventually we created similar services for price change tracking, product variations, kits, etc. We now have a collection of services available for any new retail partner integration. Bring them on!

Needless to say, in an in-house development environment like ours, it is difficult to hammer in the point about matching column types. Especially when there is no fool proof method in SQL Server, like Oracle’s PL/SQL %TYPE syntax which guarantees that the parameters’ type matches the type of the corresponding column.

We looked at methods to find these troublemakers, and developed a query which uses SQL Server’s Dynamic Management Views (DMVs) – sys.dm_exec_cached_plans and sys. dm_exec_query_stats. The query can find a query in the cache that performs implicit conversions, and gives us all of the below mentioned metrics along with the physical operation being performed due to the implicit conversion, whether it does an index scan, full table scans, etc. We were only interested in insert, update, delete, select, and execute proc operations that cause a convert_implicit. We also filtered out any implicit conversions on the same data type but on different lengths. For example, implicit conversions from varchar(256) to varchar(200) were filtered.

Metrics from the Query:

-  SQL Query – the sql statement that causes the convert implicit to happen.
-  SQL Type – the kind of SQL – whether ‘SELECT’, ‘INSERT’, ‘INSERT EXEC’, ‘UPDATE’, ‘EXECUTE PROC’, ‘DELETE’
-  Databasename where the implicit conversion took place
-  Schemaname – the schema where the table is stored
-  Tablename – the table involved in the implicit conversion
-  Columnname – the column involved in the implicit conversion
-  ConvertFrom – tells you the implicit conversion, converts the column from this datatype
-  ConvertTo – tells you the implicit conversion, converts the column to this datatype
-  ConvertFromLength – tells you the implicit conversion, converts from this datatype length
-  ConvertToLength – tells you the implicit conversion, converts to this datatype length
-  Scalarstring – gives the internal convert implicit statement, that does the conversion.
-  PhysicalOp – tells you whether it was a Clustered Index Scan, Index Scan, Seek, TableScan
-  EstimateRows – estimated # of rows that the query affects
-  EstimateIO – estimated IO
-  EstimateCPU – estimated CPU
-  AvgRowSize – average number of rows the query expects to return
-  EstimatedTotalSubtreeCost – estimated cost of the query
-  Usecounts – the number of times the query plan/cache object is used since its inception
-  size_in_bytes – number of bytes consumed by the cache object
-  Object Type – whether the sql query is a proc or SQL statement
-  exec_dbname – database where the query was actually run
-  exec_schemaname – schema where the query was actually run
-  exec_objectname – object name (stored procedure) where the query was actually run
(for eg: a stored procedure in one database DB_1, can run a query against a table in another database DB_2, where the implicit conversion takes place)
-  query_plan – compile time show plan representation of the query execution plan.

We’ve been able to identify several queries in our environment that were doing implicit conversions and saw a 1000x performance improvement in query execution times, with CPU usage humming back to normal after tuning them.

Implicit conversions probably aren’t the reason your site is having performance problems, but once you’ve taken care of the low hanging fruit (index maintenance, statistics, etc.), you need to take DB optimization to the next level and look for these types of queries which hopefully can be easily refactored (perhaps using temp tables) to give you that extra bit of speed.

Download SQL Script Here

We’re always looking to squeeze out that last bit of performance, so we’ll let you know what needle in the haystack we find next.

Flash forward to late 2011.  It turns out that MCL is pretty useful if you are trying to sell home goods on the internet, and perhaps other types of goods as well.   The search and recommendations team at Wayfair has just launched a simple recommender component, as described here.  Our system is working pretty well and giving the people something like what they want, but we suspect we can find more interesting connections among people and things than the ones we are finding. Greg and I are reading academic and industry research papers, when Greg finds Stijn’s research and MCL. We give it a try, and our recommendations improve.

That’s the short version of the story.  But we think it may interest some readers if we describe our thought process a little further.

There’s a passage in Philipp Janert‘s book Data Analysis with Open Source Tools, where he’s describing the state of machine learning, classification schemes in particular:

The difficulty of developing some recommendations that work in general and for a broad range of application domains may also explain one particular observation regarding classification: the apparent scarcity of spectacular, well-publicized successes. Spam filtering seems to be about the only application that clearly works and affects many people directly. Credit card fraud detection and credit scoring are two other widely used (if less directly visible) applications. But beyond those two, I see only a host of smaller, specialized applications. This suggests again that every successful classifier implementation depends strongly on the details of the particular problem—probably more so than on the choice of algorithm. (page 424)

More generally, all this machine learning, when you first hear about it, sounds as if it’s going to solve all your problems by crunching unattended through a bunch of simulations, cluster-finders and test/training sequences on a big computer in your back room.  But unless you’re working in one of the magical domains where the textbook techniques ‘just work’, you’re probably not going to be able to simply download a software library and plug your data in.  You’re going to need to get your hands into the seemingly shapeless muck, and try different approaches until you’ve molded it into a form where it does something useful for you.  The people who won the Netflix contest had an ‘everything-but-the-kitchen-sink’ approach.  I suspect in the end, so will we.  Be warned, though: once you get past the naive approaches, you can try some very sophisticated techniques and move the needle of recommendations quality only a little.  But don’t be discouraged.  The sense of the community, and certainly our own experience, is that with a large data set, small improvements can make a big difference in relevance and ultimately profit.

So, the kitchen sink it is.  But where to start?  We can’t just choose our techniques at random.  We need to make some educated guesses as to what might be helpful.  Off to the library!  Or at least the internet.  There’s a pretty good general discussion of advanced techniques in the recommendations space here, where the authors divide the families of approaches into the Pandora and Netflix types: content filtering and collaborative filtering, respectively.  Translated into our world, let’s call these Homeflix and Homedora (h/t John Mulliken for those labels).  Which one?   No doubt some sort of hybrid in the end, as these nice people suggest.  For now let’s start with collaborative filtering, so Homeflix.  It would be great if we could use one of the already well-researched recommendations models based on singular value decomposition (SVD), which take explicit customer ratings of products and in some cases implicit customer preferences, to learn latent customer preferences and product attributes. Then it would just be a matter of running stochastic gradient descent or alternating least squares on top of a good linear algebra package and waiting a few hours for the customer preference and product attribute matrices to converge.  Yehuda Koren and Robert Bell describe the relevant ideas in this article, which is chapter 5 of this excellent book. Something along those lines is apparently state of the art for Netflix. IBM has even worked out a way to distribute this type of work if your data is too big to fit in memory, at least for SVD models based on purely explicit ratings. But all that gives pretty poor results when we try to shoehorn our data into it.

Our data just doesn’t seem to be shaped like the data that works well with these approaches.  How is it shaped?  Well, it turns out, a lot like the proteins Stijn van Dongen was analyzing, which he depicts like so:

As you can see from that progression, he’s gradually eliminating the wispy, more tenuous connections between clusters of proteins defined by the sturdier, weightier links.  We need to do something a lot like that.  To be specific, if you recall our definition of ‘flagging’ an item in the previous article, then you can model a flag as an edge of weight ’1′ in a graph like the one depicted.

And lo and behold, when we started recommending things based on connections as established by this process, our results improved again.  Better furniture shopping through protein analysis!  Who would have figured that?  This is perfect for e-commerce people in a hurry.  Both the science and the primary engineering (the toolkit with all its switches) are done!  Greg just had to realize it would help.  Of course, that’s a bit of a black art in itself.

I wonder if the author of the toolkit might take a dim view of all this, based on what he says here. Where is the line between helpfully eliminating the useless irrelevant nonsense from the lists of things we present to our users, and pre-emptively discarding things they might have been interested in?  We’ll have to be careful about how much if any of all this we use in our internal search results, I suppose.  But for showing similar items in scrollers down the page, or cross-selling at judiciously chosen spots, this will be a big help.

What to do? I decided to start with something simple, before our inevitable trek into the forests of matrix factorization, stochastic gradient descent, Markov clusters and other impressive-sounding stuff: more on all that in subsequent posts.

So George and I began with the most obvious available literature, the O’Reilly Book Programming Collective Intelligence by Toby Segaran (if there’s an O’Reilly book on it, you must be able to just do it, right?), and with the simplest data set we could imagine: a set of relations between users and items, which we interpret as the user’s preference for the item.  This might be a item-view, an item-purchase (unless closely followed by a return), or any other event we think might come in handy.  We need a general term for this activity, for discussion purposes, not ‘view’ or ‘purchase’: let’s call it ‘flagging’.  This data is most like the book’s del.icio.us example: people either linked to something or did not.  We’re also going to limit ourselves to the simplest possible tool, a sql interface to something that is more or less a bunch of tables.  We have tried the following, or at least parts of it, on MS SQL Server, Netezza and Hive.

It’s impractical for us to load our entire data set into memory, or even to represent the relationships of all users and all items as explicit data, so we look for a sparse durable representation: a relational table in which a record means that a user flagged an item within a certain context. The context depends on the data source: a user viewed an item in a particular month/week/day/session, a user purchased an item in a particular month/week/day/order, etc.

Now let’s compute the following:

1. For each user-item pair: how many times did the user flag the item?
2. For each user: how many items were flagged? how many total flag events?
3. For each pair of items flagged by at least one user: how many users flagged both items? We call this ‘overlap’.  We exclude outliers at this point: users with too few items flagged, or too many flag events.
4. For each item: how many users flagged it?  We call this ‘popularity’.

Now we’re ready to compute any of the correlation metrics in the book.  Which ones make sense?  Not Pearson correlation or Euclidean distance.  You can compute them well enough, but try to imagine what they mean in the context of this data. What kind of straight line, or triangle’s hypotenuse, are you fitting these data points to?  None that I can picture.  The data is too much of a degenerate case of anything to which those concepts might usefully apply: you get a lot of scores of exactly ’1′ or ’0′ or  .  Raw frequency makes sense, but it’s a bit of a blunt instrument.  Prior to our setting up this system, there was something on the site that essentially used frequency along the lines we’re talking about here.  It overvalued very popular things, to be sure, but in the end people clicked and bought things off those recommendations, so it wasn’t terrible.  But what about Jaccard coefficient (sometimes called Tanimoto distance)?  J(A,B) =   .  Sounds plausible.  We’ll interpret the Jaccard coefficient of our items A and B as 1 minus (overlap of A and B)/(popularity of A + popularity of B – overlap of A and B).  Makes sense to me, and it’s straightforward in sql!   Our final table (let’s call it ‘item_affinity_jaccard’) will have at least 3 columns: the id of A, the id of B, and the coefficient.

We placed those results in a test harness, and the results were visibly, obviously better than the frequency-based thing that was there before.  But could we trust our eyes?  Hard to know without trying it.  We replaced it on the site, and clickthrough rose 18%.  That we can trust!  If you’re starting out with recommendations, I’d say give that a try.

For extra points, let’s move on to a less degenerate case.  We’ll add a new product C to our A and B, and observe that if A is connected to B, and B is connected to C, then A is connected, in a way, to C.  This will be quick, dirty, and not scalable at all (scalable in the sense that, if you wanted to add a D, E or F, you would quickly be out of luck). But if you’ve got this far, and you’ve never gone through an exercise to convince yourself that graph processing gets ugly fast when your only tools are bunch of relational tables, try the following:

1. Summarize previous results in a table: for each item, compute the count of users who flagged it, total flags, and the number of other items for which you can compute the Jaccard coefficient (let’s call this the ‘recommendation count’, and these items ‘recommendable items’).
2. Make an item_relationship_step2 table that contains all the connected pairs.  Avoid combinatorial explosion by only including items where recommendation count is greater than 0 and less than something that excludes items for which you already have so many direct pairs that you don’t really need the farther-away things.
3. Join item_affinity_jaccard to itself and then to item_relationship_step2, and compute the two-hop distance in whatever way you think best.

A while back a fellow name Richard Crowley wrote a puppet-pip provider, which seems to have been folded into Puppet 2.7, or replaced by a module in Puppet 2.7, or something like that. So in a sense his little project is dead. But Karthick on our team has resurrected a fork of it, a hybrid provider using subcommands of setuptools (easy_install) and pip for different aspects of installation, version checking and uninstallation. We call it easypip (easypip.rb), and the forked project containing it is now up on github.  Enjoy!