Why not give Code Deploy Clients access to the repository?

We’ve received a few online, and in person questions like this, so i figured it was probably worth explaining in a little more detail.

On the Deployment server, we have a variety of applications that we deploy. From Windows .Net Services, Python, Classic ASP, CSS/JS and PHP to name a few.

We chose to standardize the interface to the Deployment server to make creating new code deployment clients simpler. Our Deployment server is essentially an on demand package creation and deployment system. Continue reading

Webops for Python, part 2: the how-to

In part 1 of this 2-part series we used a comic strip to depict Python programmers and web operations folk working together to figure out how to deploy some scientific computing to an e-commerce site.  Joking aside, let’s describe exactly what were were trying to accomplish, and how we did it. Continue reading

WebOps for Python, part 1: the comic strip

Python is my favorite computer language for data science, but it is a poorly standardized beast when it comes to packaging, deployment, web operations, etc.  There are plenty of people who are deploying Python code to the web effectively, but especially in the data science area, there is no equivalent of the LAMP stack that you can just plug in and start coding against.  We have a way, among other possible ways, of solving these problems, that we think people might find useful, and I am going to describe our methods in a couple of blog posts.  The first one will tell the story as a comic strip.  The next one will have the code and instructions. Continue reading

Puppet provider for python library installation

We run a python/Tornado-based recommendations service behind the scenes at Wayfair.  As part of our code deployments, we need to install various third-party libraries to our Tornado servers. The python tools that do this kind of thing are a bit half-baked, so we paper over their inadequacies with puppet.

A while back a fellow name Richard Crowley wrote a puppet-pip provider, which seems to have been folded into Puppet 2.7, or replaced by a module in Puppet 2.7, or something like that. So in a sense his little project is dead. But Karthick on our team has resurrected a fork of it, a hybrid provider using subcommands of setuptools (easy_install) and pip for different aspects of installation, version checking and uninstallation. We call it easypip (easypip.rb), and the forked project containing it is now up on github.  Enjoy!

Wayfair Code Deployment, Part 3

If you haven’t already read the overview of our deployment system or the architecture of our deployment server you might want to check those posts out first.

In this article I will discuss how we deploy code in a unique way that gives us all the flexibility of the symlink method without requiring a symlink.

Part of the decision on how to setup our deployment system was determined by the need to roll forward and roll back our deployments as quickly as possible. We also needed a way to do this atomically, so that no errors occurred when the code was physically being copied to the webservers.

Continue reading

Wayfair Code Deployment, Part 2

The architecture of the deployment system

In the last post I explained our deployment system goals and the basic architecture.

Now that stage is set, let’s get into the details of the architecture from the deployment server perspective. As we designed the deployment server we wanted to keep the interface and the logic and requirements of the server itself as simple as possible.

The deployment server is running on our standard FLAMP stack. (FreeBSD, Lighttpd, APC, MSSQL/MySQL/Memcache, PHP). We’ll save more details on those decisions for a later post.

On the deployment server there is a configuration file that defines each application and the required details to deploy it. It defines the basic things like friendly name, allowed code reviewers, repository name and repository folders to be included. Continue reading

Continuous Improvement

The Wayfair Engineering team isn’t really on the continuous integration bandwagon, but our deployment model comes pretty close naturally because of our desire for continuous improvement.

In the block to the right of the homepage we are displaying the number of code deployments that engineers at Wayfair have done over the last 7 business days. This is a count of deployments to our new environment running our customer facing websites. This represents only a small portion of our total code base, but is a good display of how often we are improving the storefront for our customers. Internally we track a lot of other statistics about our deployments as well, like who did the deployment, the internal ticket number that the deployment was for, the SVN revision that was deployed, and who reviewed the code. Continue reading

Wayfair Code Deployment

Part 1 of a series on Code Deployment at Wayfair

Code Deployment at Wayfair has always been about creating the
fastest and most friction free deployment process possible .

Wayfair.com Architecture History
For the last 9 years our platform has been primarily a Classic ASP
environment on a Windows Server stack. In order to facilitate code deployment in that environment we wrote a script that replicated file changes out to the webfarm once they were FTPed to a central server.  Unfortunately as the webfarm grew, so did the amount of time it took to push code, and in the event of an issue, the time it took to roll back code. In this environment it took about 15 minutes to replicate code out to all of our servers.

From the chatter on the inter-webs and talks at Velocity and other conferences I’m sure a lot of people would think that is “fast enough”. For us this was unacceptable, since changes that had to go out together were not guaranteed to replicate at the same time, and with upwards of 50 deployments a day we needed to see our changes more quickly. Continue reading