lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phong Dais <phong.gd...@gmail.com>
Subject Re: SolrException: Unavailable Service
Date Wed, 13 Apr 2011 10:37:53 GMT
Erick,

I was under the misconception that a solr "transaction" is ACID.
>From what you said, I guess solr "transactions" are not Isolated.

Thanks,
Phong

On Tue, Apr 12, 2011 at 2:54 PM, Erick Erickson <erickerickson@gmail.com>wrote:

> See below:
>
> On Tue, Apr 12, 2011 at 2:21 PM, Phong Dais <phong.gdais@gmail.com> wrote:
>
> > Erick,
> >
> > My setup is not quite the way you described.  I have multiple threads
> > indexing simultaneously, but I only have 1 thread doing the commit after
> > all
> > indexing threads finished.  I have multiple instances of this running
> each
> > in their own java vm.  I'm ok with throwing out all the docs indexed so
> far
> > if the commit fail.
> >
> >
> But this is really the same thing. On the back end, Solr is piping them all
> into
> a common index and that is where the autocommit happens.
>
> The fact that it's happening in separate JVMs doesn't alter the concept,
> you
> should
> let autocommit handle things. The problem here is knowing what hasn't
> indexed.
>
>
> > I did not know that the recommended procedure is to use auto commit.  I
> > will
> > explore this avenue.  I was not aware of the master slave setup neither.
> >
> > The first thing that comes to mind is how do I know which docs did not
> get
> > committed if the auto commit ever fails?  What is the recommended
> procedure
> > for handling failure?  Any failed docs will need to be index at some
> point
> > in the future.
> >
>
> Assuming that you have a <uniqueKey> defined, you can look at the logs to
> see failures.
> Then you can choose to re-index all the documents that have changed around
> that
> time (backing up as far as you need to to be safe) . The key here is that
> you can re-index and the old copy (if any) will be replaced by the
> re-indexed
> copy.
>
> There's nothing really built into Solr that does this for you, you really
> have to build this
> part yourself.
>
> Best
> Erick
>
>
> >
> > Thanks for the valuable inputs.
> >
> > Phong
> >
> >
> > On Tue, Apr 12, 2011 at 9:09 AM, Erick Erickson <erickerickson@gmail.com
> > >wrote:
> >
> > > Sorry, fat fingers. Sent that last e-mail inadvertently.
> > >
> > > Anyway, if I have this correct, I'd recommend going to
> > > autocommit and NOT committing from the clients. That's
> > > usually the recommended procedure.
> > >
> > > This is especially true if you have a master/slave setup,
> > > because each commit from each client will trigger
> > > (potentially) a replication.
> > >
> > > Best
> > > Erick
> > >
> > > On Tue, Apr 12, 2011 at 9:07 AM, Erick Erickson <
> erickerickson@gmail.com
> > > >wrote:
> > >
> > > > If your commit from the client fails, you don't really know the
> > > > state of your index anyway. All the threads you have sending
> > > > documents to Solr are adding them to a single internal buffer.
> > > > Committing flushes that buffer.
> > > >
> > > > So if thread 1 gets an error on commit, it will presumably
> > > > have some documents from thread 2 in the commit. But
> > > > thread 2 won't necessarily see the results. So I don't think
> > > > your statement about needing to know if a commit fails
> > > > is really
> > > >
> > > >
> > > > On Tue, Apr 12, 2011 at 8:50 AM, Phong Dais <phong.gdais@gmail.com>
> > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I did not want to hijack this thread (
> > > >>
> http://www.mail-archive.com/solr-user@lucene.apache.org/msg34181.html
> > )
> > > >> but I am experiencing the same exact problem mentioned here.
> > > >>
> > > >> To sum up the issue, I am getting intermittent "Unavailable Service"
> > > >> exception during indexing commit phase.
> > > >> I know that I am calling commit "very often" but I do not see any
> way
> > > >> around
> > > >> this.  This is my situation, I am
> > > >> indexing a huge amount of documents using multiple instance of SolrJ
> > > >> client
> > > >> running on multiple servers.  There is no way
> > > >> for me control when "commit" is called from these clients, so two
> > > >> different
> > > >> clients can call commit "at the same time".
> > > >> I am not sure if I can/should use auto/timed commit because I need
> to
> > > know
> > > >> if a commit failed so I can rollback the batch that failed.
> > > >>
> > > >> What kind of options do I have?
> > > >> Should I try to catch the exception and keep trying to "recommit"
> > until
> > > it
> > > >> goes through?  I can see some potential of problems with this
> > approach.
> > > >> Do I need to write a request broker to queue up all these commit and
> > > send
> > > >> them to solr one by one in a "timely" manner?
> > > >>
> > > >> Just wanted to know if anyone has a solution for this problem before
> I
> > > >> dive
> > > >> off the deep end.
> > > >>
> > > >> Thanks,
> > > >> Phong
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message