Webops for Python, part 2: the how-to

In part 1 of this 2-part series we used a comic strip to depict Python programmers and web operations folk working together to figure out how to deploy some scientific computing to an e-commerce site.  Joking aside, let’s describe exactly what were were trying to accomplish, and how we did it. Continue reading

WebOps for Python, part 1: the comic strip

Python is my favorite computer language for data science, but it is a poorly standardized beast when it comes to packaging, deployment, web operations, etc.  There are plenty of people who are deploying Python code to the web effectively, but especially in the data science area, there is no equivalent of the LAMP stack that you can just plug in and start coding against.  We have a way, among other possible ways, of solving these problems, that we think people might find useful, and I am going to describe our methods in a couple of blog posts.  The first one will tell the story as a comic strip.  The next one will have the code and instructions. Continue reading

REST, REST, REST, you can never get enough rest.

Representational State Transfer (REST) has been around for more than a decade and continues to grow in popularity. There are dozens of frameworks that can expedite implementing a solution; 16 OpenSource PHP alone. So when Wayfair Engineering decided to evaluate migrating to Web Services (WS), it was a no-brainer. REST, right? Wait; there are other more mature frameworks. Continue reading

Wayfair Engineering Open Board Game Night

At Wayfair Engineering, we’re not just proud of the elegant technical solutions we implement, but we’re also proud of our team.  As part of our team bonding, we have frequent “Pod Outings,” activities that can be organized by any member of the Engineering team. Some recent Pod Outings have included a trip to play aerial dodgeball at SkyZone, a paintball outing where we honed our squad tactics, a relaxing day of golf, and our recurring breakfast club before work.  Sometimes the best outings are the ones we host at our offices, as we recently did when we decked out our 24th floor with food, beer, and every gaming system we could lay our hands on. Continue reading

The little program that could

It started as a proof of concept prototype in the fall of 2010. The idea came from a meeting where we were discussing the porting of our storefront codebase from classic ASP to PHP. One of the discussion points was how to avoid simply porting the same logic from one scripting language to another, but rather finding ways to move some of that logic to other more suitable platforms, including service oriented solutions. The little program was written as a self-hosted WCF service written in VB.NET running on my Windows XP box as a console application. It implemented a RESTful API service that returned the number of products in a customer’s shopping cart. Really simple and modest in scope, the thing worked like a charm. I made a presentation about it during one of our Wayfair Engineering Lunch and Learn sessions about a week later. So far, so good. Continue reading

The 24-Hour News Cycle

A few weeks ago, we celebrated Inc! Magazine’s great cover story about us, including an internal poll of our favorite item from the photo shoot (results: tie between the purple dragon and giant giraffe).  Unbeknownst to us, however, the story was later picked up by Yahoo!’s news feed on April 10th and posted to the scroller on their homepage.  This is where our story begins… Continue reading

FreeBSD 9.0 on Dell PowerEdge 12G servers

At Wayfair, we are big fans of Dell’s server platform, so naturally we were excited when their 12G line of servers started shipping.  We are also very big fans of FreeBSD.  Ahead of our first order for a new Dell PowerEdge r720, we did some research and found that the new PERC H710 RAID Controllers used the LSI SAS 2208 controller chip.  A quick look at the FreeBSD hardware compatibility list for 9.0 Release showed that this chip was supported by the mps driver, knowing that PERC cards are usually supported by the mfi driver, we thought it was a bit weird, but weren’t that concerned. Continue reading

Moving Constants out of APC and into CDB

As has been discussed at great length on this blog recently, performance is a key part of the work we do here at Wayfair.  Recently, we’ve been putting a lot of extra effort into our technology developments to make our pages load faster and more reliably.  One of the most recent releases to this end was an update to how we store a lot of our data in our various caching systems.  I am going to focus this post on the introduction of a new technology to our caching layers: CDB (Constant Database). Continue reading

Software Adaptations – Wayfair’s Partners

You would think data replication is a piece of cake these days given all the advances in database technology, and that’s true for the most part when you’re dealing with databases of the same type, but when you have to replicate parts of your product catalog with other companies, things get a bit tricky. At Wayfair Engineering we’ve figured out how to make it happen by creating a great software solution that keeps our retail partner operations working like a well-oiled machine. Continue reading

Eliminate Implicit Conversions

At Wayfair, we are never done. And the DBA team here is a true example of it. We are constantly looking to improve performance and we rigorously tune our databases on a daily basis. We are always looking at ways to have our queries run faster – by maintaining indexes, optimizing queries and procedures, creating any missing indexes based on query usage, generating statistics on currently running queries, and filtering out queries with top CPU usage, among other improvements. Of late, we’ve been trying to eliminate any implicit data type conversions that happen at runtime. Implicit data type conversions come with cost, especially when the conversion is performed at the column side of the query – not the literal side. We have had scenarios where for high volume processing jobs (processing millions of records) we had index scan execution on queries due to implicit conversions. A simple demonstration of an implicit conversion is: WHERE a.OrderID = b.OrderNo; a.OrderID being varchar(30) and b.OrderNo is nvarchar(30). Here the execution plan would do an implicit cast to nvarchar(30) and would perform an index scan operation on the millions of records – with you waiting endlessly for the query or job to finish. Continue reading