lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <>
Subject Re: Simple Faceted Searching out of the box
Date Fri, 22 Sep 2006 15:58:41 GMT
On 9/22/06, Tim Archambault <> wrote:
> I have a couple of questions from some online newspaper folks who are
> interested in Solr and are trying to understand how and why it came to be. I
> think inherent in these questions is the underlying theme I hear all the
> time and that is "Solr is not a content management system. It's a search
> engine."
> What I really wonder about CNet is how they manage their content and how
> Solr fits into their overall architecture -- is it an add-on? a
> purpose-built hammer to handle a specific problem they were having? was it
> something they "wanted" ... or instead something they needed to do, despite
> preferring something else?

Putting on my CNET hat for a little history:

We had a search server... a very thin layer built around a proprietary
search engine, used in a ton of places, for search-box type
functionality and direct generation of dynamic content.

That search engine was being discontinued by the vendor, so a
replacement was needed.  RFPs were put out, and all the commercial
alternatives were examined, but licensing costs  for the number of
servers we were talking about was exorbitant.

So we decided to build our own...

The replacement: ATOMICS- a MySQL/Apache hybrid.
It works well for many of the search collections we have that don't
need much in the way of full-text search (MySQL does have full-text
capabilities, but nothing like Lucene).

Backup plan: something based on Lucene.
SOLAR really started out as a pure backup plan... just in case ATOMICS
had problems in some areas.  I had joined CNET a week earlier, and the
task of building "something lucene-based" was luckily handed to me as
I didn't have any other responsibilities yet.  Pretty much no
requirements except for the preference of something that spoke
HTTP/XML that could be put behind a load-balancer and scaled.

ATOMICS was pretty much done by the time I started on SOLAR, and was
rapidly deployed across CNET.  SOLAR had a tough time gaining traction
until someone crossed a problem that ATOMICS couldn't easily handle:
faceted browsing.  There was finally something concrete to aim for,
and filter caching, docsets, autowarming, custom query handlers, etc,
were rapidly added to allow the ability to write custom plugins that
could acutally do the faceting logic.

The result:

It soulds like Hoss might go into some more details in his ApacheCon session:

> Another question asked of me was "Will Solr ever connect with datasources
> directly?"

As far as where Solr fits into our architecture, it's a back-end
component in the generation of dynamic content... sort of the same
place that a database would occupy.

I don't know much about content generation in CNET, and specific
content manangement syustems, but a lot of it ends up in databases.
An "indexer" piece normally pulls stuff from one or more databases,
and puts them into a solr master, which is replicated out to solr
searchers (or slaves) that the app-servers generating dynamic content
hit through a load-balancer.

There is a diagram of that from my ApacheCon presentation:

As far as connecting to datasources directly... I think that being
able to pull content from a database is a good idea, and It's on the
todo list.  What specific other data sources did you have in mind?


View raw message