lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Szűcs Roland <szucs.rol...@bookandwalk.hu>
Subject Re: very slow frequent updates
Date Wed, 24 Feb 2016 18:09:32 GMT
Thanks again Jeff. I will check the documentation of join queries becasue I
never used it before.

Regards

Roland

2016-02-24 19:07 GMT+01:00 Jeff Wartes <jwartes@whitepages.com>:

>
> I suspect your problem is the intersection of “very large document” and
> “high rate of change”. Either of those alone would be fine.
>
> You’re correct, if the thing you need to search or sort by is the thing
> with a high change rate, you probably aren’t going to be able to peel those
> things out of your index.
>
> Perhaps you could work something out with join queries? So you have two
> kinds of documents - book content and book price - and your high-frequency
> change is limited to documents with very little data.
>
>
>
>
>
> On 2/24/16, 4:01 AM, "roland.szucs@booknwalk.com on behalf of Szűcs
> Roland" <roland.szucs@booknwalk.com on behalf of
> szucs.roland@bookandwalk.hu> wrote:
>
> >I have checked it already in the ref. guide. It is stated that you can not
> >search in external fields:
> >
> https://cwiki.apache.org/confluence/display/solr/Working+with+External+Files+and+Processes
> >
> >Really I am very curios that my problem is not a usual one or the case is
> >that SOLR mainly focuses on search and not a kind of end-to-end support.
> >How this approach works with 1 million documents with frequently changing
> >prices?
> >
> >Thanks your time,
> >
> >Roland
> >
> >2016-02-24 12:39 GMT+01:00 Stefan Matheis <matheis.stefan@gmail.com>:
> >
> >> Depending of what features you do actually need, might be worth a look
> >> on "External File Fields" Roland?
> >>
> >> -Stefan
> >>
> >> On Wed, Feb 24, 2016 at 12:24 PM, Szűcs Roland
> >> <szucs.roland@bookandwalk.hu> wrote:
> >> > Thanks Jeff your help,
> >> >
> >> > Can it work in production environment? Imagine when my customer
> initiate
> >> a
> >> > query having 1 000 docs in the result set. I can not use the
> pagination
> >> of
> >> > SOLR as the field which is the basis of the sort is not included in
> the
> >> > schema for example the price. The customer wants the list in
> descending
> >> > order of the price.
> >> >
> >> > So I have to get all the 1000 docids from solr and find the metadata
> of
> >> > them in a sql database or in cache in best case. This is the way you
> >> > suggested? Is it not too slow?
> >> >
> >> > Regards,
> >> > Roland
> >> >
> >> > 2016-02-23 19:29 GMT+01:00 Jeff Wartes <jwartes@whitepages.com>:
> >> >
> >> >>
> >> >> My suggestion would be to split your problem domain. Use Solr
> >> exclusively
> >> >> for search - index the id and only those fields you need to search
> on.
> >> Then
> >> >> use some other data store for retrieval. Get the id’s from the solr
> >> >> results, and look them up in the data store to get the rest of your
> >> fields.
> >> >> This allows you to keep your solr docs as small as possible, and you
> >> only
> >> >> need to update them when a *searchable* field changes.
> >> >>
> >> >> Every “update" in solr is a delete/insert. Even the "atomic update”
> >> >> feature is just a shortcut for that. It requires stored fields
> because
> >> the
> >> >> data from the stored fields gets copied into the new insert.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 2/22/16, 12:21 PM, "Roland Szűcs" <roland.szucs@booknwalk.com>
> >> wrote:
> >> >>
> >> >> >Hi folks,
> >> >> >
> >> >> >We use SOLR 5.2.1. We have ebooks stored in SOLR. The majority
of
> the
> >> >> >fields do not change at all like content, author, publisher....
Only
> >> the
> >> >> >price field changes frequently.
> >> >> >
> >> >> >We let the customers to make full text search so we indexed the
> content
> >> >> >filed. Due to the frequency of the price updates we use the atomic
> >> update
> >> >> >feature. As a requirement of the atomic updates we have to store
all
> >> the
> >> >> >fields even the content field which is 1MB/document and we did
not
> >> want to
> >> >> >store it just index it.
> >> >> >
> >> >> >As we wanted to update 100 documents with atomic update it took
> about 3
> >> >> >minutes. Taking into account that our metadata /document is 1 Kb
and
> >> our
> >> >> >content field / document is 1MB we use 1000 more memory to
> accelerate
> >> the
> >> >> >update process.
> >> >> >
> >> >> >I am almost 100% sure that we make something wrong.
> >> >> >
> >> >> >What is the best practice of the frequent updates when 99% part
of a
> >> given
> >> >> >document is constant forever?
> >> >> >
> >> >> >Thank in advance
> >> >> >
> >> >> >--
> >> >> ><https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
> Roland
> >> >> Szűcs
> >> >> ><https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
> Connect
> >> >> with
> >> >> >me on Linkedin <
> >> >> https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
> >> >> ><https://bookandwalk.hu/>
> >> >> >CEO Phone: +36 1 210 81 13
> >> >> >Bookandwalk.hu <https://bokandwalk.hu/>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > <https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu> Szűcs
> >> Roland
> >> > <https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
> >> Ismerkedjünk
> >> > meg a Linkedin <
> >> https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
> >> > -en <https://bookandwalk.hu/>
> >> > Ügyvezető Telefon: +36 1 210 81 13
> >> > Bookandwalk.hu <https://bokandwalk.hu/>
> >>
> >
> >
> >
> >--
> ><https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu> Szűcs
> Roland
> ><https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
> Ismerkedjünk
> >meg a Linkedin <
> https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
> >-en <https://bookandwalk.hu/>
> >Ügyvezető Telefon: +36 1 210 81 13
> >Bookandwalk.hu <https://bokandwalk.hu/>
>



-- 
<https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu> Szűcs Roland
<https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu> Ismerkedjünk
meg a Linkedin <https://www.linkedin.com/pub/roland-sz%C5%B1cs/28/226/24/hu>
-en <https://bookandwalk.hu/>
Ügyvezető Telefon: +36 1 210 81 13
Bookandwalk.hu <https://bokandwalk.hu/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message