lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Kan <solrexp...@gmail.com>
Subject Re: Measuring SOLR performance
Date Mon, 02 Sep 2013 19:12:21 GMT
Hi Erick,

Agree, this is perfectly fine to mix them in solr. But my question is about
solrjmeter input query format. Just couldn't find a suitable example on the
solrjmeter's github.

Dmitry



On Mon, Sep 2, 2013 at 5:40 PM, Erick Erickson <erickerickson@gmail.com>wrote:

> filter and facet queries can be freely intermixed, it's not a problem.
> What problem are you seeing when you try this?
>
> Best,
> Erick
>
>
> On Mon, Sep 2, 2013 at 7:46 AM, Dmitry Kan <solrexpert@gmail.com> wrote:
>
> > Hi Roman,
> >
> > What's the format for running the facet+filter queries?
> >
> > Would something like this work:
> >
> > field:foo  >=50  fq=other_field:bar facet=true
> facet.field=facet_field_name
> >
> >
> > Thanks,
> > Dmitry
> >
> >
> >
> > On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <solrexpert@gmail.com>
> wrote:
> >
> > > Hi Roman,
> > >
> > > With adminPath="/admin" or adminPath="/admin/cores", no. Interestingly
> > > enough, though, I can access
> > > http://localhost:8983/solr/statements/admin/system
> > >
> > > But I can access http://localhost:8983/solr/admin/cores, only when
> with
> > > adminPath="/admin/cores" (which suggests that this is the right value
> to
> > be
> > > used for cores), and not with adminPath="/admin".
> > >
> > > Bottom line, these core configuration is not self-evident.
> > >
> > > Dmitry
> > >
> > >
> > >
> > >
> > > On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <roman.chyla@gmail.com
> > >wrote:
> > >
> > >> Hi Dmitry,
> > >> So it seems solrjmeter should not assume the adminPath - and perhaps
> > needs
> > >> to be passed as an argument. When you set the adminPath, are you able
> to
> > >> access localhost:8983/solr/statements/admin/cores ?
> > >>
> > >> roman
> > >>
> > >>
> > >> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <solrexpert@gmail.com>
> > wrote:
> > >>
> > >> > Hi Roman,
> > >> >
> > >> > I have noticed a difference with different solr.xml config contents.
> > It
> > >> is
> > >> > probably legit, but thought to let you know (tests run on fresh
> > >> checkout as
> > >> > of today).
> > >> >
> > >> > As mentioned before, I have two cores configured in solr.xml. If the
> > >> file
> > >> > is:
> > >> >
> > >> > [code]
> > >> > <solr persistent="false">
> > >> >
> > >> >   <!--
> > >> >   adminPath: RequestHandler path to manage cores.
> > >> >     If 'null' (or absent), cores will not be manageable via request
> > >> handler
> > >> >   -->
> > >> >   <cores adminPath="/admin/cores" host="${host:}"
> > >> > hostPort="${jetty.port:8983}" hostContext="${hostContext:solr}">
> > >> >     <core name="metadata" instanceDir="metadata" />
> > >> >     <core name="statements" instanceDir="statements" />
> > >> >   </cores>
> > >> > </solr>
> > >> > [/code]
> > >> >
> > >> > then the instruction:
> > >> >
> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a --durationInSecs
> > 60
> > >> -R
> > >> > cms -t /solr/statements -e statements -U 100
> > >> >
> > >> > works just fine. If however the solr.xml has adminPath set to
> "/admin"
> > >> > solrjmeter produces an error:
> > >> >
> > >> > [error]
> > >> > **ERROR**
> > >> >   File "solrjmeter.py", line 1386, in <module>
> > >> >     main(sys.argv)
> > >> >   File "solrjmeter.py", line 1278, in main
> > >> >     check_prerequisities(options)
> > >> >   File "solrjmeter.py", line 375, in check_prerequisities
> > >> >     error('Cannot find admin pages: %s, please report a bug' %
> apath)
> > >> >   File "solrjmeter.py", line 66, in error
> > >> >     traceback.print_stack()
> > >> > Cannot find admin pages: http://localhost:8983/solr/admin, please
> > >> report a
> > >> > bug
> > >> > [/error]
> > >> >
> > >> > With both solr.xml configs the following url returns just fine:
> > >> >
> > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > >> >
> > >> > Regards,
> > >> >
> > >> > Dmitry
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <solrexpert@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi Roman,
> > >> > >
> > >> > > This looks much better, thanks! The ordinary non-comarison mode
> > works.
> > >> > > I'll post here, if there are other findings.
> > >> > >
> > >> > > Thanks for quick turnarounds,
> > >> > >
> > >> > > Dmitry
> > >> > >
> > >> > >
> > >> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <
> roman.chyla@gmail.com
> > >> > >wrote:
> > >> > >
> > >> > >> Hi Dmitry, oh yes, late night fixes... :) The latest commit
> should
> > >> make
> > >> > it
> > >> > >> work for you.
> > >> > >> Thanks!
> > >> > >>
> > >> > >> roman
> > >> > >>
> > >> > >>
> > >> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <
> solrexpert@gmail.com>
> > >> > wrote:
> > >> > >>
> > >> > >> > Hi Roman,
> > >> > >> >
> > >> > >> > Something bad happened in fresh checkout:
> > >> > >> >
> > >> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > >> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > >> --durationInSecs
> > >> > 60
> > >> > >> -R
> > >> > >> > cms -t /solr/statements -e statements -U 100
> > >> > >> >
> > >> > >> > Traceback (most recent call last):
> > >> > >> >   File "solrjmeter.py", line 1392, in <module>
> > >> > >> >     main(sys.argv)
> > >> > >> >   File "solrjmeter.py", line 1347, in main
> > >> > >> >     save_into_file('before-test.json',
> > >> simplejson.dumps(before_test))
> > >> > >> >   File
> "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
> > >> line
> > >> > >> 286,
> > >> > >> > in dumps
> > >> > >> >     return _default_encoder.encode(obj)
> > >> > >> >   File
> "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > >> line
> > >> > >> 226,
> > >> > >> > in encode
> > >> > >> >     chunks = self.iterencode(o, _one_shot=True)
> > >> > >> >   File
> "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > >> line
> > >> > >> 296,
> > >> > >> > in iterencode
> > >> > >> >     return _iterencode(o, 0)
> > >> > >> >   File
> "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > >> line
> > >> > >> 202,
> > >> > >> > in default
> > >> > >> >     raise TypeError(repr(o) + " is not JSON serializable")
> > >> > >> > TypeError: <__main__.ForgivingValue object at 0x7fc6d4040fd0>
> is
> > >> not
> > >> > >> JSON
> > >> > >> > serializable
> > >> > >> >
> > >> > >> >
> > >> > >> > Regards,
> > >> > >> >
> > >> > >> > D.
> > >> > >> >
> > >> > >> >
> > >> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
> > >> roman.chyla@gmail.com>
> > >> > >> > wrote:
> > >> > >> >
> > >> > >> > > Hi Dmitry,
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
> > >> solrexpert@gmail.com>
> > >> > >> > wrote:
> > >> > >> > >
> > >> > >> > > > Hi Roman,
> > >> > >> > > >
> > >> > >> > > > Good point. I managed to run the command with
-C and double
> > >> > quotes:
> > >> > >> > > >
> > >> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour
-x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > > >
> > >> > >> > > > As a result got several files (html, css,
js, csv) in the
> > >> running
> > >> > >> > > directory
> > >> > >> > > > (any way to specify where the output should
be stored in
> this
> > >> > case?)
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > > i know it is confusing, i plan to change it - but
later, now
> it
> > >> is
> > >> > too
> > >> > >> > busy
> > >> > >> > > here...
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > When I look onto the comparison dashboard,
I see this:
> > >> > >> > > >
> > >> > >> > > > http://pbrd.co/17IRI0b
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > > two things: the tests probably took more than one
hour to
> > >> finish, so
> > >> > >> they
> > >> > >> > > are not aligned - try generating the comparison
with '-c
> >  14400'
> > >> >  (ie.
> > >> > >> > > 4x3600 secs)
> > >> > >> > >
> > >> > >> > > the other thing: if you have only two datapoints,
the dygraph
> > >> will
> > >> > not
> > >> > >> > show
> > >> > >> > > anything - there must be more datapoints/measurements
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > One more thing: all the previous tests were
run with
> > softCommit
> > >> > >> > disabled.
> > >> > >> > > > After enabling it, the tests started to fail:
> > >> > >> > > >
> > >> > >> > > > $ python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx
-q
> > >> > >> > > > ./queries/demo/demo.queries -s localhost -p
8983 -a
> > >> > >> --durationInSecs 60
> > >> > >> > > -R
> > >> > >> > > > g1 -t /solr/statements -e statements -U 100
> > >> > >> > > > $ cd g1
> > >> > >> > > > Reading results of the previous test
> > >> > >> > > > $ cd 2013.08.12.16.32.48
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > >> > >> > > > $ mkdir 2013.08.12.16.33.02
> > >> > >> > > > $ cd 2013.08.12.16.33.02
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > >> > >> > > > Traceback (most recent call last):
> > >> > >> > > >   File "solrjmeter.py", line 1427, in <module>
> > >> > >> > > >     main(sys.argv)
> > >> > >> > > >   File "solrjmeter.py", line 1381, in main
> > >> > >> > > >     before_test = harvest_details_about_montysolr(options)
> > >> > >> > > >   File "solrjmeter.py", line 562, in
> > >> > harvest_details_about_montysolr
> > >> > >> > > >     indexLstModified =
> > >> > >> > cores_data['status'][cn]['index']['lastModified'],
> > >> > >> > > > KeyError: 'lastModified'
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > > Thanks for letting me know, that info is probably
not
> available
> > >> in
> > >> > >> this
> > >> > >> > > situation - i've cooked st quick to fix it, please
try the
> > latest
> > >> > >> commit
> > >> > >> > > (hope it doesn't do more harm, i should get some
sleep ..;))
> > >> > >> > >
> > >> > >> > > roman
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > In case it matters:  Python 2.7.3, ubuntu,
solr 4.3.1.
> > >> > >> > > >
> > >> > >> > > > Thanks,
> > >> > >> > > >
> > >> > >> > > > Dmitry
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla
<
> > >> > roman.chyla@gmail.com>
> > >> > >> > > wrote:
> > >> > >> > > >
> > >> > >> > > > > Hi Dmitry,
> > >> > >> > > > > The command seems good. Are you sure
your shell is not
> > doing
> > >> > >> > something
> > >> > >> > > > > funny with the params? You could try:
> > >> > >> > > > >
> > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour
-x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > -a
> > >> > >> > > > >
> > >> > >> > > > > where g1 and foo are results of the individual
runs, ie.
> > >> > something
> > >> > >> > that
> > >> > >> > > > was
> > >> > >> > > > > started and saved with '-R g1' and '-R
foo' respectively
> > >> > >> > > > >
> > >> > >> > > > > so, for example, i have these comparisons
inside
> > >> > >> > > > > '/var/lib/montysolr/different-java-settings/solrjmeter',
> so
> > >> I am
> > >> > >> > > > generating
> > >> > >> > > > > the comparison by:
> > >> > >> > > > >
> > >> > >> > > > > export
> > >> > >> > > > >
> > >> > >>
> > SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
> > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour
-x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > -a
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > roman
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry
Kan <
> > >> > solrexpert@gmail.com
> > >> > >> >
> > >> > >> > > > wrote:
> > >> > >> > > > >
> > >> > >> > > > > > Hi Roman,
> > >> > >> > > > > >
> > >> > >> > > > > > One more question. I tried to compare
different runs
> (g1
> > vs
> > >> > cms)
> > >> > >> > > using
> > >> > >> > > > > the
> > >> > >> > > > > > command below, but get an error.
Should I attach some
> > other
> > >> > >> > param(s)?
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > python solrjmeter.py -C g1,foo -c
hour -x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > > > > > **ERROR**
> > >> > >> > > > > >   File "solrjmeter.py", line 1427,
in <module>
> > >> > >> > > > > >     main(sys.argv)
> > >> > >> > > > > >   File "solrjmeter.py", line 1303,
in main
> > >> > >> > > > > >     check_options(options, args)
> > >> > >> > > > > >   File "solrjmeter.py", line 185,
in check_options
> > >> > >> > > > > >     error("The folder '%s' does
not exist" % rf)
> > >> > >> > > > > >   File "solrjmeter.py", line 66,
in error
> > >> > >> > > > > >     traceback.print_stack()
> > >> > >> > > > > > The folder '0' does not exist
> > >> > >> > > > > >
> > >> > >> > > > > > Dmitry
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM,
Dmitry Kan <
> > >> > >> solrexpert@gmail.com>
> > >> > >> > > > wrote:
> > >> > >> > > > > >
> > >> > >> > > > > > > Hi Roman,
> > >> > >> > > > > > >
> > >> > >> > > > > > > Finally, this has worked! Thanks
for quick support.
> > >> > >> > > > > > >
> > >> > >> > > > > > > The graphs look awesome. At
least on the index sample
> > :)
> > >> It
> > >> > is
> > >> > >> > > quite
> > >> > >> > > > > easy
> > >> > >> > > > > > > to setup and run + possible
to run directly on the
> > shard
> > >> > >> server
> > >> > >> > in
> > >> > >> > > > > > > background mode.
> > >> > >> > > > > > >
> > >> > >> > > > > > > my test run was:
> > >> > >> > > > > > >
> > >> > >> > > > > > > python solrjmeter.py -a -x
./jmx/SolrQueryTest.jmx -q
> > >> > >> > > > > > > ./queries/demo/demo.queries
-s localhost -p 8983 -a
> > >> > >> > > --durationInSecs
> > >> > >> > > > 60
> > >> > >> > > > > > -R
> > >> > >> > > > > > > foo -t /solr/statements -e
statements
> > >> > >> > > > > > >
> > >> > >> > > > > > > Thanks!
> > >> > >> > > > > > >
> > >> > >> > > > > > > Dmitry
> > >> > >> > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54
AM, Roman Chyla <
> > >> > >> > roman.chyla@gmail.com
> > >> > >> > > >
> > >> > >> > > > > > wrote:
> > >> > >> > > > > > >
> > >> > >> > > > > > >> Hi Dmitry,
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> I've modified the solrjmeter
to retrieve data from
> > under
> > >> > the
> > >> > >> > core
> > >> > >> > > > (the
> > >> > >> > > > > > -t
> > >> > >> > > > > > >> parameter) and the rest
from the /solr/admin - I
> could
> > >> test
> > >> > >> it
> > >> > >> > > only
> > >> > >> > > > > > >> against
> > >> > >> > > > > > >> 4.0, but it is there the
same as 4.3 - it seems...so
> > you
> > >> > can
> > >> > >> try
> > >> > >> > > the
> > >> > >> > > > > > fresh
> > >> > >> > > > > > >> checkout
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> my test was: python solrjmeter.py
-a -x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > -t
> > >> > >> > > > > > >> /solr/collection1 -R foo
-q ./queries/demo/* -p 9002
> > -s
> > >> > >> adsate
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> Thanks!
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> roman
> > >> > >> > > > > > >>
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> On Tue, Aug 6, 2013 at
9:46 AM, Dmitry Kan <
> > >> > >> > solrexpert@gmail.com>
> > >> > >> > > > > > wrote:
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> > Hi,
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > Thanks for the clarification,
Shawn!
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > So with this in mind,
the following work:
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> >
> > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > >> > >> > > > > > >> >
> > >> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > not copying their
output to save space.
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > Roman:
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > is this something
that should be set via -t
> > parameter
> > >> as
> > >> > >> well?
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > Dmitry
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > On Tue, Aug 6, 2013
at 4:34 PM, Shawn Heisey <
> > >> > >> > solr@elyograg.org
> > >> > >> > > >
> > >> > >> > > > > > wrote:
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > > On 8/6/2013 6:17
AM, Dmitry Kan wrote:
> > >> > >> > > > > > >> > > > Of three
URLs you asked for, only the 3rd one
> > gave
> > >> > >> > response:
> > >> > >> > > > > > >> > > <snip>
> > >> > >> > > > > > >> > > > The rest
report 404.
> > >> > >> > > > > > >> > > >
> > >> > >> > > > > > >> > > > On Mon,
Aug 5, 2013 at 8:38 PM, Roman Chyla <
> > >> > >> > > > > > roman.chyla@gmail.com>
> > >> > >> > > > > > >> > > wrote:
> > >> > >> > > > > > >> > > >
> > >> > >> > > > > > >> > > >> Hi Dmitry,
> > >> > >> > > > > > >> > > >> So I
think the admin pages are different on
> > your
> > >> > >> version
> > >> > >> > of
> > >> > >> > > > > solr,
> > >> > >> > > > > > >> what
> > >> > >> > > > > > >> > > do
> > >> > >> > > > > > >> > > >> you
see when you request... ?
> > >> > >> > > > > > >> > > >>
> > >> > >> > > > > > >> > > >>
> > http://localhost:8983/solr/admin/system?wt=json
> > >> > >> > > > > > >> > > >>
> > http://localhost:8983/solr/admin/mbeans?wt=json
> > >> > >> > > > > > >> > > >>
> http://localhost:8983/solr/admin/cores?wt=json
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > > Unless you have
a valid defaultCoreName set in
> > your
> > >> > >> > > (old-style)
> > >> > >> > > > > > >> > > solr.xml, the
first two URLs won't work, as
> you've
> > >> > >> > discovered.
> > >> > >> > > > > > >>  Without
> > >> > >> > > > > > >> > > that valid defaultCoreName
(or if you wanted
> info
> > >> from
> > >> > a
> > >> > >> > > > different
> > >> > >> > > > > > >> > > core), you'd
need to add a core name to the URL
> > for
> > >> > them
> > >> > >> to
> > >> > >> > > > work.
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > > The third one,
which works for you, is a global
> > >> handler
> > >> > >> for
> > >> > >> > > > > > >> manipulating
> > >> > >> > > > > > >> > > cores, so naturally
it doesn't need a core name
> to
> > >> > >> function.
> > >> > >> > > >  The
> > >> > >> > > > > > URL
> > >> > >> > > > > > >> > > path for this
handler is defined by solr.xml.
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > > Thanks,
> > >> > >> > > > > > >> > > Shawn
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >>
> > >> > >> > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > >
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message