Sphinx: Search Outside the Box

For many years, the only easy way to add decent search capability to a MySQL-backed web site was to use its full-text index support. It was fast, efficient, and reasonably configurable. But as more and more sites began to deal with larger datasets and moved from MyISAM to InnoDB, they found that it was harder to support their search needs. InnoDB does not provide full-text indexing, so that often meant keeping around a set of slaves that still ran MyISAM for the sole purpose of handling full-text search requests.

That‚Äôs exactly what we were doing at Craigslist before I began looking for a better way. My search for alternatives wasn‚Äôt motivated so much by the need to maintain the MyISAM slaves‚Äìthat part was automated. The real problem is that we kept running into an invisible ‚Äúglass ceiling‚Äù in performance. Low concurrency on the full-text searches eventually meant throwing far more hardware at the problem than was really necessary. Response times would go through the roof, CPU was not being overly taxed, disks were mostly idle, and there was still free memory…