lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamal Kishore Aggarwal <kkroyal....@gmail.com>
Subject Re: Solr Stream vs Export Request Handlers
Date Thu, 18 Oct 2018 06:20:19 GMT
Hi,

Thanks again Joel for your reply. I have noted your suggestions.

I observed one more thing while using solrj to fetch the data using /stream
with export and direct /export. The solr QTime is almost same, however
elapsed time(total time) to fetch response in streaming with export is
better than direct /export (Streaming export taking 30% less than /export).

Is this also expected ?

Regards
Kamal Kishore



On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein <joelsolr@gmail.com> wrote:

> Yes this is correct. But keep in mind Streaming Expression has a wide range
> of features that have nothing at all to do with the export handler. In
> general with Streaming Expressions you want to find the functions that get
> the job done using the least amount of work. The /export handler is often
> not the best choice. You'll want to read through the various streaming
> expressions to see if they might be more efficient for your use case.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
> kkroyal.192@gmail.com> wrote:
>
> > Hi,
> >
> > After I performed the test on my data, I found out that direct /export
> and
> > streaming expression with export, both are giving almost same response
> > time. This was also pointed out by *Jan Høydahl* in his reply.
> >
> > Also, the documentation says export feature uses stream sorting technique
> > and streaming expression also uses steam technique. So, are they
> internally
> > works in same fashion. Please confirm.
> >
> > Regards
> > Kamal Kishore
> >
> >
> >
> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> > kkroyal.192@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Thanks Jan & Joel.
> > >
> > > Though I will evaluate the performance over my data, but based on your
> > > experience, which one of the two is better in performance ?. Please
> > suggest
> > >
> > > Yeah, I know export does not get the data from all shards, but we can
> > > write code to aggregate the data from all shards. But only if export is
> > > better over stream.
> > >
> > > Thanks
> > > Kamal Kishore
> > >
> > >
> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <joelsolr@gmail.com>
> > > wrote:
> > >
> > >> The export handler does not do distributed search. So if you have a
> > >> multi-shard collection you may have to use Streaming Expressions to
> get
> > >> exports from all shards.
> > >>
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >>
> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <jan.asf@cominvent.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Yes, you can choose which to use, it should give  you about same
> > result.
> > >> > If you already work with the Solr search API it would be the easiest
> > for
> > >> > you to consume /export as you don't need to learn the new syntax and
> > >> parse
> > >> > the Tuple response. However, if you need to do stuff with the docs
> as
> > >> you
> > >> > stream them from Solr, then streaming expressions lets you enrich
> the
> > >> docs,
> > >> > modify, join etc on the fly.
> > >> >
> > >> > PS: When the /export docs says it uses a streaming tecnique, it does
> > NOT
> > >> > mean that it has uses the solr feature streaming expressions :)
> > >> >
> > >> > --
> > >> > Jan Høydahl, search solution architect
> > >> > Cominvent AS - www.cominvent.com
> > >> >
> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> > >> > kkroyal.192@gmail.com>:
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > I have a requirement to fetch all data from a collection. One
way
> is
> > >> to
> > >> > use
> > >> > > streaming expression and other way is to use export.
> > >> > >
> > >> > > Streaming expression documentation says *streaming functions
are
> > >> designed
> > >> > > to work with entire result sets rather then the top N results
like
> > >> normal
> > >> > > search. This is supported by the /export handler.*
> > >> > >
> > >> > > Also, Export handler documentation says *this feature uses a
> stream
> > >> > sorting
> > >> > > technique that begins to send records within milliseconds and
> > >> continues
> > >> > to
> > >> > > stream results until the entire result set has been sorted and
> > >> exported.*
> > >> > >
> > >> > > These two statements concludes to me that for fetching entire
> > results
> > >> > > streaming expressions uses export handler and export handler
uses
> > >> stream,
> > >> > > so, whether I use streaming expression or export handler, they
are
> > >> > > internally same and would have same performance. I am correct
over
> > >> here
> > >> > to
> > >> > > say so ?
> > >> > >
> > >> > >
> > >> > > Ref Links:
> > >> > >
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> > >> > >
> > >> > >
> > >> > > Regards
> > >> > > Kamal Kishore
> > >> >
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message