Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of dmitry.kan@gmail.com
 designates 209.85.160.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANGii8e9F6Ow_11BbJ+3x8xsJPKCkCMvWgDqUgR12YES92UJ+Q@mail.gmail.com>
References: 
 <CAOnbiDZZKicrR5KU1dS1sqG8C_6QD1=C2JHUaeXWdAVRYtBkvw@mail.gmail.com>
	<CAOnbiDZBXJyiFiNQGxf607i+o1x2GzOsAhdcVC2jZgRTtg1WTw@mail.gmail.com>
	<CANGii8e9F6Ow_11BbJ+3x8xsJPKCkCMvWgDqUgR12YES92UJ+Q@mail.gmail.com>
Date: Thu, 12 Apr 2012 22:09:56 +0300
Message-ID: 
 <CAOnbiDaz7kvkpSXzsM6cATC6nujQ-jeLROqzvN_gAP+BWyWZHQ@mail.gmail.com>
Subject: Re: solr 3.4 with nTiers >= 2: usage of ids param causes
 NullPointerException (NPE)
From: Dmitry Kan <dmitry.kan@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=20cf305b0ab00197fa04bd801a27

--20cf305b0ab00197fa04bd801a27
Content-Type: text/plain; charset=UTF-8

Mikhail,

Thanks for sharing your thoughts. Yes I have tried checking for NULL and
the entire chain of queries between tiers seems to work. But I suspect,
that some docs will be missing. In principle, unless there is an
OutOfMemory or a shard down, the doc ids should be retrieving valid
documents. So this is just a design, as Yonik pointed out.

I would be willing to contribute a patch, it is just an issue of
understanding what exactly should be fixed in the architecture, and I
suspect it isn't a small change..

Dmitry

On Thu, Apr 12, 2012 at 9:22 PM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:

> Dmitry,
>
> The last NPE in HighlightingComponent is just a sad coding issue.
> few rows later we can see that developer expected to have some docs not
> found
> // remove nulls in case not all docs were able to be retrieved
>      rb.rsp.add("highlighting", SolrPluginUtils.removeNulls(new
> SimpleOrderedMap(arr)));
> But as you already know he forgot to check if(sdoc!=null){.
> Is there anything that stopping you from contributing the patch, beside of
> the lack of time, of course?
>
> about the core issue I can't get into it and, particularly, how the using
> disjunction query in place of IDS can help you. Could you please provide
> more detailed info like stacktraces, etc. Btw, have you checked trunk for
> your case?
>
> On Thu, Apr 12, 2012 at 7:08 PM, Dmitry Kan <dmitry.kan@gmail.com> wrote:
>
> > Can anyone help me out with this? Is this too complicated / unclear? I
> > could share more detail if needed.
> >
> > On Wed, Apr 11, 2012 at 3:16 PM, Dmitry Kan <dmitry.kan@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > Hopefully this question is not too complex to handle, but I'm currently
> > > stuck with it.
> > >
> > > We have a system with nTiers, that is:
> > >
> > > Solr front base ---> Solr front --> shards
> > >
> > > Inside QueryComponent there is a method
> > createRetrieveDocs(ResponseBuilder
> > > rb) which collects doc ids of each shard and sends them in different
> > > queries using the ids parameter:
> > >
> > > [code]
> > > sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
> > > [/code]
> > >
> > > This actually produces NPE (same as in
> > > https://issues.apache.org/jira/browse/SOLR-1477) in the first tier,
> > > because Solr front (on the second tier) fails to process such a query.
> I
> > > have tried to fix this by using a unique field with a value of ids ORed
> > > (the following code substitutes the code above):
> > >
> > > [code]
> > >       StringBuffer idsORed = new StringBuffer();
> > >       for (Iterator<String> iterator = ids.iterator();
> > iterator.hasNext();
> > > ) {
> > >         String next = iterator.next();
> > >
> > >         if (iterator.hasNext()) {
> > >           idsORed.append(next).append(" OR ");
> > >         } else {
> > >           idsORed.append(next);
> > >         }
> > >       }
> > >
> > >       sreq.params.add(rb.req.getSchema().getUniqueKeyField().getName(),
> > > idsORed.toString());
> > > [/code]
> > >
> > > This works perfectly if for rows=n there is n or less hits from a
> > > distributed query. However, if there are more than 2*n hits, the
> querying
> > > fails with an NPE in a completely different component, which is
> > > HighlightComponent (highlights are requested in the same query with
> > >
> >
> hl=true&hl.fragsize=50000&hl.requireFieldMatch=true&hl.fl=targetTextField):
> > >
> > > SEVERE: java.lang.NullPointerException
> > >         at
> > >
> >
> org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:161)
> > >         at
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> > >         at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> > >         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
> > >         at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
> > >         at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
> > >         at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > >         at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > >         at
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > >         at
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > >         at
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > >         at
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > >         at
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > >         at
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> > >         at
> > >
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > >         at
> > >
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> > >         at
> > > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > >         at java.lang.Thread.run(Thread.java:619)
> > >
> > > It sounds like the ids of documents somehow get shuffled and the
> > > instruction (only a hypothesis)
> > >
> > > [code]
> > > ShardDoc sdoc = rb.resultIds.get(id);
> > > [/code]
> > >
> > > returns sdoc=null, which causes the next line of code to fail with an
> > NPE:
> > >
> > > [code]
> > > int idx = sdoc.positionInResponse;
> > > [/code]
> > >
> > > Am I missing anything? Can something be done for solving this issue?
> > >
> > > Thanks.
> > >
> > > --
> > > Regards,
> > >
> > > Dmitry Kan
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> gedel@yandex.ru
>
> <http://www.griddynamics.com>
>  <mkhludnev@griddynamics.com>
>


-- 
Regards,

Dmitry Kan

--20cf305b0ab00197fa04bd801a27--