Everything comes with trade-offs, and the word on the street is that /dev/null is so heavily optimized for write performance that ‘read’ reliability can be less than ideal.  But who knows?  Maybe that’s the balance we want for our write-heaviest workloads.

So we got out our testing tools and went to work on a bake-off.  We wanted to simulate real-world conditions as much as possible, so we wrote some PHP scripts that connected to our sharded development Mongo cluster.  On the /dev/null side, configuration of a cluster was pretty easy, as long as you start from a standard posix-style system.

After function-testing the PHP, we wrote a quick Apache Bench script to bake off the two systems.  The results speak for themselves:

Past the upper 90%, MongoDB is a classic hockey stick.  /dev/null starts fast and stays flat, out to the far horizon.  Love it.  I don’t know that we’re ready to switch right away: I’ll be a little uncomfortable while it’s still in beta, and I’ll have to get to the bottom of those unreliable read operations.  But this is looking *very* promising.

For now, I’m going to follow our internal process for new technologies like this, which is to email around a Wayfair Technical Finding (‘WTF’) to all the senior software engineers and architects, so we can put our heads together, evaluate further, and eventually make a plan to roll this out across all our data centers.

Hat tip gar1t on xtranormal.

My plan is to provide feedback about the wearer’s surroundings, including obstacles and suggested paths, and enable him or her to control the wheelchair with eye movements. The original student project used patterns of a user’s eyes being opened or closed to change between types of motion. For my project, I want to use subtle yet deliberate movements of the eye to let the user interact seamlessly with the surrounding environment. I think this technology could be life-changing for persons with disabilities. I hope that being able to work on this project with the support of the Google Glass Explorer program will help make it a reality.

I wrote up my idea, posted it on Google Plus with the #ifihadglass hashtag, and some fellow Wayfairians tweeted about it. Walter Frick of BostInno saw a tweet, did an interview with me, and then wrote an article about it. You can read the full story at these links:

BostInno: http://bit.ly/X7nD6b
Popular Science write-up: http://bit.ly/16unjVS

In 2009 Wayfair’s database infrastructure was based almost entirely on Microsoft SQL Server. Our Business Intelligence team was using a SQL Server data warehouse to prepare a large amount of data for import into Analysis Services (SSAS) each day. We populated our data warehouse using transaction log shipping from production servers, which required about 3 hours of downtime on the data warehouse at midnight each night to restore the previous day’s logs. Once that was done, a series of stored procedures were kicked off by jobs that would crunch through data from several different servers to produce a star schema that could be pulled into SSAS. Wayfair was scaling rapidly, and this approach started to become painfully slow, often taking 10-16 hours to crunch through the previous day’s data.

The BI team decided to look into other solutions for data warehousing, and ultimately purchased a Netezza appliance. Netezza is essentially a fork of PostgreSQL that takes a massively parallel cluster of nodes (24 in our case) and makes them look like one database server to the client. In our tests, Netezza could crunch through our data in roughly a quarter of the time, bringing 10-16 hours down to a much more reasonable 2-4 hours. The dream of updating our data warehouse multiple times each day was starting to look feasible. The feedback loop on business decisions would become dramatically shorter, enabling us to iterate more quickly and make well informed decisions at a much faster pace. There was just one glaring problem.

Great, But How Are We Going to Get Data Into It?

As soon as the DBA team heard that the Netezza purchase had been finalized, our first question was “great, but how are we going to get data into it?” The folks at Netezza didn’t have an answer for us, but they did send us an engineer to help devise a solution. As it turned out, the problem of how to incrementally replicate large amounts of data into a data warehouse was a common one, and there were surprisingly few open source solutions. Google it, and most people will tell you that they just reload all their data every day, or that they only have inserts so they can just load the new rows each day. “Great, but what if you want incremental replication throughout the day? What if you have updates or deletes? How do you deal with schema changes?” Crickets.

The First Solution

The solution we arrived upon was to use SQL Server Change Tracking to keep track of which rows had changes on each table, and we built a replication system around that. We created stored procedures for each table that contained the commands required to use the CHANGETABLE() function to generate change sets, dump those to flat files on a network share using bcp, pipe them through dos2unix to fix the line endings, and load them into netezza using the proprietary nzload command. Over the course of a few months we came up with an elaborate series of REPLACE() functions for text fields to escape delimiters, eliminate line breaks and clean up other data anomalies that had the potential to break the nzload. The whole process was driven by SSIS packages.

This solution worked, but it was a maintenance nightmare. We frequently had to edit stored procedures when adding new columns, and we had to edit the SSIS packages to add new tables. SSIS uses GUI based programming, and the editor for it (Business Intelligence Development Studio) is extremely slow and clunky, so even making simple changes was a painful process. Adding a new table into change tracking was a 14-step process that took over an hour of DBA time, and setting up a new database took roughly 28 steps and around two days of DBA time. We also had no solution for schema changes – we needed to manually apply them to the Netezza server, and if we forgot to do so the change tracking jobs would fail.

Release Early, Then Iterate Like Hell

Over the next few years, we iterated on this solution and added a number of useful features. We got rid of the stored procedures per table and switched to a single stored procedure that used dynamic SQL instead. We created a solution for automated schema changes based off of DDL triggers. We created a single stored procedure to handle adding new tables into change tracking, turning it into a one-step process. We added features to publish a subset of a table’s columns, because Netezza had a fixed row size limit that some of our tables exceeded. We added a feature to trim the length of text fields, because large blobs of text usually aren’t needed on the data warehouse and they slowed down the process. We added logging of performance and health metrics to statsD with alerts in Tattle. We added the ability to replicate changes from sharded master databases and consolidate them into one database on the slave. We added the ability to replicate to multiple SQL Server data warehouses in addition to Netezza. We had data on our masters that was moved into archive tables when certain criteria were met, so we added a feature to apply changes to a table and its archive in one transaction on the slave to eliminate the temporary appearance of duplicate data.

Not Good Enough

Ultimately, we were still unhappy with the solution. It was too heavily based on stored procedures, functions, configuration via database tables, xp_cmdshell, and worst of all - linked servers. It was still a nightmare to set up new databases, and when wanted to make changes we had to edit the same stored procedures in 20+ different places.  It was still single threaded. Worst of all, it was tightly coupled. If one slave server fell behind, the others suffered for it. It was also extremely specific to the use case of replicating data from SQL Server to either SQL Server or Netezza, and Wayfair was beginning to make much more use of open source databases like MySQL and MongoDB. In early 2012, we realized this solution wasn’t going to scale any further through iteration. We needed a redesign. We needed something fresh.

Enter Tesla

Redesigned from the ground up and inspired by the Tesla Replicator in The Prestige, Tesla was the solution to our data warehousing woes. We completely avoided stored procedures, functions, configuration tables, SSIS, dynamic SQL, xp_cmdshell and linked servers. Instead, we wrote Tesla in C# (primarily due to one incredibly useful .NET class for copying data between SQL servers) and moved all the logic into the application. Tesla is a single console application that takes care of everything we were doing with stored procedures and SSIS before. Its configuration is based on files rather than tables, which we can version control and deploy using our push tool. It’s multi-threaded and uses a configurable number of threads, allowing us to replicate the most important databases as quickly as possible. It’s completely decoupled, meaning that if one slave falls behind it doesn’t impact the others. It was also designed to be extensible to other data technologies, both as sources and destinations.

Tesla’s Design

Tesla is built into a few agents such as Master and Slave. These agents are run as scheduled tasks in the scheduler of your choice, and they each have their own configuration files. They are completely decoupled and can be run on separate servers and at separate times.

The design for Tesla was inspired by LinkedIn’s article about DataBus. Specifically, the idea of a master server publishing its change sets to a relay server and the slaves polling the relay for those changes was appealing to us. It meant less load on the masters, and it also meant we could store the change sets in such a way that if a slave fell behind it would be able to get consolidated deltas to more efficiently catch up. The biggest difference between Tesla and DataBus is that we focus on batch-based change sets, rather than streaming. Batches are captured on the master as one semi-consistent view of a database at a given point in a time, reducing the chance of orphaned or incomplete data on the data warehouse. It also makes the most sense for a technology like Netezza, which is terrible at small transactions and great at large batches.

Open Source

Tesla is fully open source and available on github. It currently supports SQL Server as a master, slave and relay server, and Netezza as a slave. It was designed with extensibility in mind, so we expect to add more technologies on both sides over time. We already have a slave adapter for Hive in the works. Feel free to hack away, add features, and submit pull requests!

For those unfamiliar, a hackathon is a fantastic display of creativity, technical skills, team work, problem solving, and time management, all compressed into a single marathon event. The beanpot hackathon produced 17 demos, impressive considering the event only lasted about 24-hours.

As the event got underway, the dinner area was buzzing with excitement.  Groups of people informally huddled together, some with a white board to their side, drawing sketches and getting feedback, others researching stuff on their laptops, everyone engaged in the discussion bouncing ideas back and forth.  As different teams solidified, they moved to the main conference room to start building their project.  The one theme that was consistent across all groups was passion for technology, and enthusiasm to get something ready for demo.

Most groups worked through the night, taking short naps between bursts of coding.  We had some of our engineers available as mentors, although most groups seemed to be heads down and not looking for outside assistance.  Near the entrance to the conference room, Wayfair setup a duck pond, available for those needing a fun distraction from their project.  There was a fishing pole, and you could pull a duck from the pond to win a prize.  The rubber duck also serves as a good sounding board for ideas, or debugging code when you are stuck. (link: http://en.wikipedia.org/wiki/Rubber_duck_debugging/)

By the time Saturday evening arrived, I was blown away by some of the projects that teams put together.  Not only were the demos some cool application, or something that solved a problem, but the presentations were well done.  In many cases, the presenters talked about their inspiration, thought process, and where they saw the idea going next.  Questions from the audience were often constructive and suggested improvements.

Looking back on the event, I think one of the reasons we aligned so well with this particular event is because of the similarities to our work environment — Smart people using technology to solve problems quickly and get things done.  I see that demonstrated every day in our engineering department, and it was refreshing to see so many talented students come together for an event like this.  On a related note, we are hiring for summer internships in our software development group.  If you were a participant at the beanpot hackathon, or this type of environment sounds good to you, please get in touch with us (link: eomeara@wayfair.com)

Brad S.

(contributors: Elias Y., Nishan S.)

On the Deployment server, we have a variety of applications that we deploy. From Windows .Net Services, Python, Classic ASP, CSS/JS and PHP to name a few.

We chose to standardize the interface to the Deployment server to make creating new code deployment clients simpler. Our Deployment server is essentially an on demand package creation and deployment system.

For example when we push CSS/JS we minify the contents prior to packaging it up to be picked up by the static content servers. Same goes for things like the python deployments, where we actually build a virtualenv on the deploy server and package that up prior to deploying. This allows the code deploy clients to just download a ready to go package.

Other side benefits are that this allows us to cache the packages with varnish to offload the push server during deploys. It also allows us to have a cached version of the packages in each data center to increase the speed of deployments locally. We also get a little bit of increased security. Really more the Onion / Layers reasoning here, but every little bit helps.

A con that should be noted is that unlike an rsync or direct repo deployment, it’s a full package, not an incremental. At this point in time our largest application deployment is the CSS/JS application which is still only about ~70M when compressed. We may need to move to a different model when bandwidth/download time of packages become the bottleneck, but we have a lot of breathing room for now.

Some stats on the move:

• Data migrated for NAS and SAN block storage: 161 TB
• Network cables plugged in: 798
• Physical servers moved or newly installed: 99 rack mount and 50 blades
• Physical servers decommissioned to save power and simplify our environment: 49
• VMs newly stood up or migrated: 619

It’s worth noting that the physical moves were done over the course of 2 months.  Why so long?  Unlike many companies that can have a weekend to bring things down, we aren’t afforded that luxury.  We have customer service working in our offices 7 days a week both in the US as well as Europe, and we have our website to think about, which never closes.  In fact, we were able to pull this off with only a single 4-hour outage to our storefront, and several very small outages to our internal and backend systems during weeknights throughout the project.

Lessons Learned:

No matter how good your documentation is, it’s probably not good enough.  Most folks documentation concentrates on break/fix and general architecture of a system, what’s installed, how it’s configured, etc.  Since we drastically changed our network infrastructure, we had to re-ip every server when it was moved.  We had to go through and come up with procedures for what else needed to happen when a machine suddenly had a new IP address.  We use DNS for some things, but not everything, so we had to ensure that inter-related systems were also updated when we moved things.

Get business leads involved in the timeline.  This sounds funny, but one of the biggest metrics in measuring the success of a project like this is the perception of the users.  Since a good percentage of the systems moved had certain business units as the main “customers”, we worked with leaders from these business units to ensure we understood  their use of the systems, what days or times of day were they using it the most, or if they had any concerns over off-hours operations during different times of the week.  Once we had this info from many different groups, we sat down in a big room with all the engineers responsible for these systems, and came up with a calendar for the move, then got final approval for dates from the business leads.  This was probably the smarted thing we did, and went a long way in helping our “customer satisfaction”.

Another thing we learned early on was to divide the work of the physical moving of equipment and the work done by the subject matter experts to make system changes and ensure things are working properly after the physical move.  This freed the subject matter expert to get right to work, and not have to worry about other, non-related systems that were also being moved in the same maintenance window.  How did we pull this off?  Again, include everyone.  We have a large Infrastructure Engineering team, 73 people as of this writing.  We got everyone involved, from our frontline and IT Support groups, all the way up to directors; even Steve Conine, one of our co-founders did an overnight stint at the datacenter helping with the physical move of servers.  It was an amazing team effort, and we would never have had such a smooth transition if everyone didn’t step up in a big way.

I hope these little tidbits are helpful to anyone taking on such a monumental task as moving an entire data center.  As always, thanks for reading.

]]> http://engineering.wayfair.com/lessons-from-a-datacenter-move/feed/ 0 http://engineering.wayfair.com/restwayfair-com/?utm_source=rss&utm_medium=rss&utm_campaign=restwayfair-com http://engineering.wayfair.com/restwayfair-com/#comments Tue, 06 Nov 2012 17:36:03 +0000 Daniel B. http://engineering.wayfair.com/?p=1467 Continue reading →]]>

Some Background

At Wayfair, we are working on a next generation of systems to power our business. The decade old  systems that currently keep us running in stride have allowed Wayfair.com to vault from nothing to where it is today. But as with all systems, they have started to show their age.

Up until just a few years ago our entire engineering team was less than 40 people.  At this team size, we could all easily work on a single shared code base that our entire business ran on. This monolithic architecture was a conscious decision given our bootstrapped startup beginnings.

The good news is that the ‘single code base’ approach successfully bootstrapped our way to millions of customers and products, 400 million dollars in revenue, and ‘a zillion things home’ (wink) over 9 years. We’re proud to say that we accomplished all of this with a small, sharp team of engineers.  We are a successful startup, and it is only a matter of time until any startup needs to evolve through technical step function changes. For Wayfair.com, that time is now.  You can call it technical debt, re-architecture, version 2.0, whatever you want.  The reality is we need to change our approach. If we want our large business to grow even more, we need to have teams who work in parallel, and in order for that to work, we need to decouple and distribute our organization.

One of our initial steps was to reorganize the Engineering department into ‘platform teams’ – small and efficient groups of domain-specific spec op developers. These platform teams are responsible for the maintainability, scalability, and evolution of discrete segments of the business.  We are excited about this change, and can already see the scaling advantages to this organization model.

This change has exposed an issue that gets to our core software architecture.  Prior to the Great Platforms Restructure of 2012, we could think of our code base as one big system that was either working or not working . The complexities of graceful degredation and multi modes of failure could be ignored. But no more! We need to ‘un-ignore’ these complexities and take them into account as we develop.

We can look around at all sorts of large internet success stories and see the obvious solution is to move to a distributed services architecture.  The high level plan turns out to be the easy part.  The hard part is the detail.

Look at What Others Have Done

There are two leading paradigms of web services in production today:  SOAP (Simple Object Access Protocol) and REST (REpresentational State Transfer).

SOAP

SOAP is tantamount to Remote Procedure Call (RPC), a concept in computer science that allows a machine to invoke a function or subroutine on a separate machine. With RPC, programmers create innumerable specific procedures that are accessible at any node in a distributed system – a truly attractive concept. Despite the physical gap between two machines, RPC hides the ‘distributed’ part of a ‘distributed system’ through a highly malleable, customized protocol. One downside to this customized protocol is the tight coupling of the client and service, which reduces serendipitous reuse and impedes total scalability of a system.

We all felt that SOAP was too heavy weight and came with too much overhead. Plus, what is this, the 20^th century? Just kidding. Wayfair is a fast-paced business, so this client-service coupling could come at a very high price. Since these interfaces are specialized, they would constantly need to change! ‘RPC impedes total scalability of a system’ is a bold conclusion to make, but also is a conclusion partially provided to us by the largest distributed system known to man-kind: the World Wide Web.

REST

REST is an architectural style that characterizes what makes the WWW the world’s largest distributed system.  To our team, it felt lightweight and approachable. We did, though, struggle with the notion of a resource in the realm of our application, and keep in mind that resources are core to REST architectures.

Any information that can be named can be a resource: a document or image, a temporal service (e.g. “today’s weather in Los Angeles”), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author’s hypertext reference must fit within the definition of a resource. A resource is a conceptual mapping to a set of entities, not the entity that corresponds to the mapping at any particular point in time.

-Fielding

With this in mind our initial reaction to implement REST, given our database centric history, was to make every database table a resource.  Historically we’d been extremely thoughtful about our database structure.  We had very clean data, with very minimal duplication.  However, we’ve got 2 Million lines of code, and over 6,000 database tables.  ‘Resource’-ing our application with a relational database attitude quickly started to feel overwhelming and misdirected.  It quickly became apparent that we do a ton of complex joins in SQL. So a resource-to-table approach would mean we’d be trying to re-produce the highly tuned join capabilities of the best SQL databases in PHP.  Not a good idea.

Go with your Gut

Our thinking was something to the effect of:  Let’s start by putting our core functionality (mainly Classic ASP functions and Stored Procedures in SQL Server) behind URLs.

We’ll exemplify with one of the first services we created, the inventory service.

Millions of times a day we will lookup stock status of products in our catalog.  The basic use-case is given a SKU, return stock status.  Our service-less version of this was done as a SQL Procedure called like this:

EXECUTE spGetInventory @SKU=’TH1254’

This methodology that we grew comfortable with is super simple to use as in-line code.

Our first pass at moving this behind a service looked like this:

http://services.wayfair.com/get_inventory.php?sku=TH1254&format=json

At this point we were pretty proud of ourselves.  We felt like we were moving in the services direction. We agreed, as a team, that any services would support some common query string parameters like: format, debug, verbosity, logging, help, etc…

Fast forward a few months and we start to adopt this approach in earnest.  As we move from talking directly to the database, to talking through a service we find ourselves basically wrapping what had previously been stored procedures.  This starts to look like this:

http://services.wayfair.com/create_customer.php
http://services.wayfair.com/get_customer.php
http://services.wayfair.com/update_customer.php
http://services.wayfair.com/delete_customer.php
http://services.wayfair.com/create_basket.php
http://services.wayfair.com/get_basket.php
http://services.wayfair.com/update_basket.php
http://services.wayfair.com/delete_basket.php
http://services.wayfair.com/create_order.php
….

You get the gist of it.  We provide the standard CRUD operations through URLs. However, our business is complicated, so on top of the CRUD operations we start doing things like this:

http://services.wayfair.com/does_order_have_returns.php
http://services.wayfair.com/is_order_cancellable.php
http://services.wayfair.com/apply_promotion_code.php

On one hand, we were happy because we were clearly separating our data storage from our application logic. We were creating a path where we could at some point change our database structure without affecting the application directly.  On the other hand, we weren’t actually making the code any simpler, improving the organization, or separating of functionality into more discrete areas.  We were just moving complexity to different areas.

So we decided to give our services a pinch of RESTful-ness.

One of the stages of adopting something new seems to always involve fighting to make it look like the old thing you were comfortable with. Refer to our first step towards a more RESTful approach in our past blog: “REST, REST, REST, you can never get enough REST”.

The path we were on was “REST like” because it sent HTTP messages, was stateless, and returned loosely structured data. We were still not happy that we were on the right long term path with our “REST variant”. We did not comply to the RFC that formally describes HTTP so we weren’t RESTful.  We then re-visited our approach and as we constantly do in software engineering, we evolved.

We have evolved

At some point ‘comfortable’ paradigms wear you out, and you have that ah-ha moment of clarity where you start to understand a different paradigm.

We’re happy to say we’re over the hump of fighting some of the “pure” concepts of REST and chose to imbibe them.  In particular, we’ve made a few very important changes from our home-grown REST approach.  We’ve embraced HTTP and are using a broader set of the functions it was intended to enable.  Namely we’ve standardized on:

1. Identification of resources in our system (Resource Addressing): We have started to create resources that conform to our business domain. E.g. Orders, Order Products etc. That is, resources have started to represent our business domain objects and business logic.
2. Uniform Interface (GET, POST, PUT, DELETE): We are conforming to the boundaries provided to us by HTTP/1.1
3. HATEOAS (Linking it all together):  Resources are now defined across platform boundaries and links to these resources insures that we are not duplicating data when there isn’t any need (e.g. Order representation links to customer resource representation).
4. Self-descriptive messages (Headers, Response Codes): Utilizing properties of HTTP allows our messages to describe themselves.

Doug: Solving the problem of class struggle with one of Greg‘s classifiers.

Me:  Karl Marx should call his office.  What do you mean by that?

Doug: Let me explain…

Class struggle at Wayfair search used to manifest itself as searches for ‘red cups’ that returned lists of beer pong tables, and other unlikely and embarrassing results.  In this particular example, our prospective customer was clearly looking for something in the ‘cups’ class of things, but the pack of search monkeys behind the scenes were interpreting this request wrong.    Why, you may ask?  Easy answer.  There happens to be a ‘Red Cup’ brand of beer pong tables, which forms a serendipitous linguistic pairing with these search terms, of a type that bedevils search engineers the world over.  But not at Wayfair, not any more.

When you type something into the search box on Wayfair or AllModern, your request will most likely be processed by the Solr search platform.  It’s a great platform, but in some cases vanilla Solr doesn’t do exactly what we want.   In some of these situations we hack Solr, and in others we augment its functionality with a pre-processor.  This blog post is about a classifier that we use as a pre-processor, which has dramatically improved the quality of our Solr search results.

The way we did this is not the only way to do such things.  If you want to do everything in Java, there’s an emerging pattern (perhaps already common?) of Hadoop+Mahout+Solr, which Grant Ingersoll describes here (Powerpoint) and here (techie how-to), with particular attention to the index-time aspects of such setups.  We don’t currently have a lot of needs in that area, because our catalogue is pretty well classified before it makes its way to Solr. Trey Grainger of CareerBuilder gives a good overview of various plain-Solr and search-time machine-learning techniques here.  I think you could reasonably work along those lines by integrating some Mahout libraries into a custom search handler.   You might get the same results we did.  To us, that seemed like a lot of trouble.  We already have some machine-learning components, which we wrote for non-search purposes, in C, C++ and Python.  That stack is closer to the metal than Java anyway, and for most of the mathematical operations that we need in the machine-learning area we prefer to stick with it if possible.  When we’re writing our own compiled code, or when we need C or C++ libraries like BLAS, GSL, etc., for the hard or fast math in our processing stream, an expressive scripting language like Python feels to us like a better choice for wrapper programs than Java.  In a Cython/JNI bake-off, Cython wins hands down.

But enough about platform choice.  The flow of control for the pre-processors is this:

1. User types search terms into box.
2. Php code pre-processes search terms, sometimes, as in this case, by calling out to a Python service.
3. Php code generates Solr request, executes it, displays results.

Let’s return to our prospective customer who has typed ‘red cups’ into the search box, because he wants to buy red cups.  Our unclassified search would reach for the ‘Red Cup’ beer pong tables, but we can school it a bit. We use the pre-processing step to prioritize items in the ‘cups’ class, or whatever we have that most closely resembles a ‘cups’ class.  We send ‘red+cup’ to a naive Bayesian classifier trained on our product catalogue, and we get this: “Confidence: 0.605732999499 4087, Plates, Bowls & Mugs| 283, Cups & Accessories|5027, Food Wrap & Containers|…”.  That output represents a confidence score and the identifiers and names of some classes in our catalogue.  If we get a confidence score above a certain threshold, we will then generate a Solr query like this:

/indexname/select/?qt=handlerName&q=red+cup&insertFilterQueriesHere&version=2.2&rows=40&start=0&
facet.limit=250&defType=edismax&boost=map(classID,4087,4087,1.500,1.0)&
boost=map(classID,283,283,1.250,1.0)&boost=map(classID,5027,5027,1.125,1.0)

Boosting is a small opening into Solr, through which a big bucket of smarts from external systems can be poured.  Of course, this only works if you have an intelligently classified catalogue, or whatever you call your body of material.  Creating such a thing is a domain-specific black art if there ever was one.  That’s a barrier to entry, to be sure.  But if you can figure all that out (wink!), this technique can really help you.

Hat tip, by the way, to Aaron Beppu, whose Hadoop World talk last year, available here (Powerpoint) and here (video), got me thinking about boosting Solr searches.

We were having a problem with the quality of the searches that were executed when our customers typed three words into the search box on the site.  The best vanilla SOLR treatments we could find for these searches were either too narrow or too broad: they gave us either a small number of high-quality results (sometimes no results at all), or a very large number of medium-quality results that matched two out of three of the terms.  So we wanted to do the restrictive, high-quality search, and set a threshold for an acceptable minimum number of results.  Then, if the number of results from the first search did not cross the threshold, we would run the broader search for backfill. Why not just have the front-end client code issue queries in parallel?  A good question, with a simple answer: pagination.  It is always better to let SOLR or Elastic Search handle pagination for you if you can.  Otherwise your client code will have to work unnecessarily hard to interleave or sequence results.

We found that other people had been asking for this feature, and related features, in SOLR-1093, since 2009.  The comments indicated a good amount of interest in the community.  The original feature request was for a parallel executor of more than one query.  Grant Ingersoll quickly commented that ‘fallback queries’ should be handled as well, and Lance Norskog added that serial execution would also be desirable. It turns out we only need the serial executor for fallback queries, so that’s what Karthick from our search team implemented.  It’s been in production for us for a couple of months now, and it works very well.  Karthick checked with the SOLR mailing list and then posted the patch.  It’s an extension to the SearchHandler class called MultiSesarchHandler.

Can others benefit from this patch?  I think so, but perhaps not indefinitely.   Will it be accepted into SOLR?  In the end, I think it will not, because the core SOLR people are working on a more general solution in trunk, similar in design to the ScriptUpdateProcessor.  This other thing is a completely different approach based on scripted queries.  When that’s in general use, we’ll probably switch to it.  But it’s not in SOLR 3.6, which is the most up-to-date general release as of this writing, and it’s not going to make it into SOLR 4.0.  For now we have all our patches applied to the very safe 3.6 branch, and we’re preparing to re-apply everything to 4.0 when it gets out of beta.  It might be a while before the other approach is easily available.  In the mean time, if you want to do this, get out your patch kits and enjoy!