lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <>
Subject Re: Will Solr fit our needs?
Date Wed, 17 Mar 2010 23:37:51 GMT
Another option is the ExternalFileField:

This lets you store the current prices for all items in a separate
file. You can only use it in a function query, that is. But it does
allow you to maintain one Solr index, which is very very worthwhile.

On Wed, Mar 17, 2010 at 4:19 AM, Geert-Jan Brits <> wrote:
> If you dont' plan on filtering/ sorting and/or faceting on fast-changing
> fields it would be better to store them outside of solr/lucene in my
> opinion.
> If you must: for indexing-performance reasons you will probably end up with
> maintaining seperate indices (1 for slow-changing/static fields and 1 for
> fast-changing-fields) .
> You frequently commit the fast-changing -index to incorporate the changes
> in current_price. Afterwards you have 2 options I believe:
> 1. use parallelreader to query the seperate indices directly. Afaik, this is
> not (completely) integrated in Solr... I wouldn't recommend it.
> 2. after you commit the fast-changing-index, merge with the static-index.
> You're left with 1 fresh index, which you can push to your slave-servers.
> (all this in regular interverals)
> Disadvatages:
> - In any way, you must be very careful with maintaining multiple parallel
> indexes with the purpose of treating them as one. For instance document
> inserts must be done exactly in the same order, otherwise the indices go
> 'out-of-sync' and are unusable.
> - higher maintenance
> - there is always a time-window in which the current_price values are stale.
> If that's within reqs that's ok.
> The other path, which I recommend, would be to store the current_price
> outside of solr (like you're currently doing) but instead of using a
> relational db, try looking into persistent key-value stores. Many of them
> exist and a lot of progress has been made in the last couple of years. For
> simple key-lookups (what you need as far as I can tell) they really blow
> every relational db out of the water (considering the same hardware of
> course)
> We're currently using Tokyo Cabinet with the server-frontend Tokyo Tyrant
> and seeing almost a 5x increased in lookup performance compared to our
> previous kv-store memcachedDB which is based on BerkelyDB. Memcachedb was
> already several times faster than our mysql-setup (although not optimally
> tuned) .
> to sum things up: use the best tools for what they were meant to do.
> - index/search --> solr/ lucene without a doubt.
> - kv-lookup --> consensus is still forming, and a lot of players (with a lot
> of different types of functionality) but if all you need is simple
> key-value-lookup, I would go for Tokyo Cabinet (TC) / Tyrant at the moment.
>  Please note that TC and competitors aren't just some code/ hobby projects
> but are usually born out of a real need at huge websites / social networks
> such as TC which is born from mixi  (big social network in Japan) . So at
> least you're in good company..
> for kv-stores I would suggest to begin your research at:
> (beginning
> 2009)
> (half
> 2009)
> and get a feel of the kv-playing field.
> Hope this (pretty long) post helps,
> Geert-Jan
> 2010/3/17 Krzysztof Grodzicki <>
>> Hi Mortiz,
>> You can take a look on the project ZOIE -
>> I think it's that what are you looking
>> for.
>> br
>> Krzysztof
>> On Wed, Mar 17, 2010 at 9:49 AM, Moritz Mädler <>
>> wrote:
>> > Hi List,
>> >
>> > we are running a marketplace which has about a comparable functionality
>> like ebay (auctions, fixed-price items etc).
>> > The items are placed on the market by users who want to sell their goods.
>> >
>> > Currently we are using Sphinx as an indexing engine, but, as Sphinx
>> returns only document ids we have to make a
>> > database-query to fetch the data to display. This massively decreases
>> performance as we have to do two requests to
>> > display data.
>> >
>> > I heard that Solr is able to return a complete dataset and we hope a
>> switch to Solr can boost perfomance.
>> > A critical question is left and i was not able to find a solution for it
>> in the docs: Is it possible to update attributes directly in the
>> > index?
>> > An example for better illustration:
>> > We have an index which holds all the auctions (containing auctionid,
>> auction title) with its current prices(field: current_price). When a user
>> places a new bid,
>> > is it possible to update the attribute 'current_price' directly in the
>> index so that we can fetch the current_price from Solr and not from the
>> database?
>> >
>> > I hope you understood my problem. It would be kind if someone can point
>> me to the right direction.
>> >
>> > Thanks alot!
>> >
>> > Moritz

Lance Norskog

View raw message