lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Johnson <jej2...@gmail.com>
Subject Re: Solr 4.2 Cloud Replication Replica has higher version than Master?
Date Wed, 03 Apr 2013 18:24:33 GMT
Where is this information stored in ZK?  I don't see it in the cluster
state (or perhaps I don't understand it ;) ).

Perhaps something with my process is broken.  What I do when I start from
scratch is the following

ZkCLI -cmd upconfig ...
ZkCLI -cmd linkconfig ....

but I don't ever explicitly create the collection.  What should the steps
from scratch be?  I am moving from an unreleased snapshot of 4.0 so I never
did that previously either so perhaps I did create the collection in one of
my steps to get this working but have forgotten it along the way.


On Wed, Apr 3, 2013 at 2:16 PM, Mark Miller <markrmiller@gmail.com> wrote:

> Thanks for digging Jamie. In 4.2, hash ranges are assigned up front when a
> collection is created - each shard gets a range, which is stored in
> zookeeper. You should not be able to end up with the same id on different
> shards - something very odd going on.
>
> Hopefully I'll have some time to try and help you reproduce. Ideally we
> can capture it in a test case.
>
> - Mark
>
> On Apr 3, 2013, at 1:13 PM, Jamie Johnson <jej2003@gmail.com> wrote:
>
> > no, my thought was wrong, it appears that even with the parameter set I
> am
> > seeing this behavior.  I've been able to duplicate it on 4.2.0 by
> indexing
> > 100,000 documents on 10 threads (10,000 each) when I get to 400,000 or
> so.
> > I will try this on 4.2.1. to see if I see the same behavior
> >
> >
> > On Wed, Apr 3, 2013 at 12:37 PM, Jamie Johnson <jej2003@gmail.com>
> wrote:
> >
> >> Since I don't have that many items in my index I exported all of the
> keys
> >> for each shard and wrote a simple java program that checks for
> duplicates.
> >> I found some duplicate keys on different shards, a grep of the files for
> >> the keys found does indicate that they made it to the wrong places.  If
> you
> >> notice documents with the same ID are on shard 3 and shard 5.  Is it
> >> possible that the hash is being calculated taking into account only the
> >> "live" nodes?  I know that we don't specify the numShards param @
> startup
> >> so could this be what is happening?
> >>
> >> grep -c "7cd1a717-3d94-4f5d-bcb1-9d8a95ca78de" *
> >> shard1-core1:0
> >> shard1-core2:0
> >> shard2-core1:0
> >> shard2-core2:0
> >> shard3-core1:1
> >> shard3-core2:1
> >> shard4-core1:0
> >> shard4-core2:0
> >> shard5-core1:1
> >> shard5-core2:1
> >> shard6-core1:0
> >> shard6-core2:0
> >>
> >>
> >> On Wed, Apr 3, 2013 at 10:42 AM, Jamie Johnson <jej2003@gmail.com>
> wrote:
> >>
> >>> Something interesting that I'm noticing as well, I just indexed 300,000
> >>> items, and some how 300,020 ended up in the index.  I thought perhaps I
> >>> messed something up so I started the indexing again and indexed another
> >>> 400,000 and I see 400,064 docs.  Is there a good way to find possibile
> >>> duplicates?  I had tried to facet on key (our id field) but that didn't
> >>> give me anything with more than a count of 1.
> >>>
> >>>
> >>> On Wed, Apr 3, 2013 at 9:22 AM, Jamie Johnson <jej2003@gmail.com>
> wrote:
> >>>
> >>>> Ok, so clearing the transaction log allowed things to go again.  I am
> >>>> going to clear the index and try to replicate the problem on 4.2.0
> and then
> >>>> I'll try on 4.2.1
> >>>>
> >>>>
> >>>> On Wed, Apr 3, 2013 at 8:21 AM, Mark Miller <markrmiller@gmail.com
> >wrote:
> >>>>
> >>>>> No, not that I know if, which is why I say we need to get to the
> bottom
> >>>>> of it.
> >>>>>
> >>>>> - Mark
> >>>>>
> >>>>> On Apr 2, 2013, at 10:18 PM, Jamie Johnson <jej2003@gmail.com>
> wrote:
> >>>>>
> >>>>>> Mark
> >>>>>> It's there a particular jira issue that you think may address
this?
> I
> >>>>> read
> >>>>>> through it quickly but didn't see one that jumped out
> >>>>>> On Apr 2, 2013 10:07 PM, "Jamie Johnson" <jej2003@gmail.com>
wrote:
> >>>>>>
> >>>>>>> I brought the bad one down and back up and it did nothing.
 I can
> >>>>> clear
> >>>>>>> the index and try4.2.1. I will save off the logs and see
if there
> is
> >>>>>>> anything else odd
> >>>>>>> On Apr 2, 2013 9:13 PM, "Mark Miller" <markrmiller@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>>> It would appear it's a bug given what you have said.
> >>>>>>>>
> >>>>>>>> Any other exceptions would be useful. Might be best
to start
> >>>>> tracking in
> >>>>>>>> a JIRA issue as well.
> >>>>>>>>
> >>>>>>>> To fix, I'd bring the behind node down and back again.
> >>>>>>>>
> >>>>>>>> Unfortunately, I'm pressed for time, but we really need
to get to
> >>>>> the
> >>>>>>>> bottom of this and fix it, or determine if it's fixed
in 4.2.1
> >>>>> (spreading
> >>>>>>>> to mirrors now).
> >>>>>>>>
> >>>>>>>> - Mark
> >>>>>>>>
> >>>>>>>> On Apr 2, 2013, at 7:21 PM, Jamie Johnson <jej2003@gmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Sorry I didn't ask the obvious question.  Is there
anything else
> >>>>> that I
> >>>>>>>>> should be looking for here and is this a bug?  I'd
be happy to
> >>>>> troll
> >>>>>>>>> through the logs further if more information is
needed, just let
> me
> >>>>>>>> know.
> >>>>>>>>>
> >>>>>>>>> Also what is the most appropriate mechanism to fix
this.  Is it
> >>>>>>>> required to
> >>>>>>>>> kill the index that is out of sync and let solr
resync things?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Apr 2, 2013 at 5:45 PM, Jamie Johnson <jej2003@gmail.com
> >
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> sorry for spamming here....
> >>>>>>>>>>
> >>>>>>>>>> shard5-core2 is the instance we're having issues
with...
> >>>>>>>>>>
> >>>>>>>>>> Apr 2, 2013 7:27:14 PM org.apache.solr.common.SolrException
log
> >>>>>>>>>> SEVERE: shard update error StdNode:
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> http://10.38.33.17:7577/solr/dsc-shard5-core2/:org.apache.solr.common.SolrException
> >>>>>>>> :
> >>>>>>>>>> Server at http://10.38.33.17:7577/solr/dsc-shard5-core2returned
> >>>>> non
> >>>>>>>> ok
> >>>>>>>>>> status:503, message:Service Unavailable
> >>>>>>>>>>      at
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
> >>>>>>>>>>      at
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
> >>>>>>>>>>      at
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
> >>>>>>>>>>      at
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
> >>>>>>>>>>      at
> >>>>>>>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>>>>>>>      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>      at
> >>>>>>>>>>
> >>>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> >>>>>>>>>>      at
> >>>>>>>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>>>>>>>      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>      at
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>>>>>>>      at
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>>>>>>>      at java.lang.Thread.run(Thread.java:662)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Apr 2, 2013 at 5:43 PM, Jamie Johnson
<
> jej2003@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> here is another one that looks interesting
> >>>>>>>>>>>
> >>>>>>>>>>> Apr 2, 2013 7:27:14 PM org.apache.solr.common.SolrException
log
> >>>>>>>>>>> SEVERE: org.apache.solr.common.SolrException:
ClusterState says
> >>>>> we are
> >>>>>>>>>>> the leader, but locally we don't think so
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistributedUpdateProcessor.java:293)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:228)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:339)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> >>>>>>>>>>>      at
> >>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:1797)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:637)
> >>>>>>>>>>>      at
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Apr 2, 2013 at 5:41 PM, Jamie Johnson
<
> jej2003@gmail.com
> >>>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Looking at the master it looks like
at some point there were
> >>>>> shards
> >>>>>>>> that
> >>>>>>>>>>>> went down.  I am seeing things like
what is below.
> >>>>>>>>>>>>
> >>>>>>>>>>>> NFO: A cluster state change: WatchedEvent
state:SyncConnected
> >>>>>>>>>>>> type:NodeChildrenChanged path:/live_nodes,
has occurred -
> >>>>>>>> updating... (live
> >>>>>>>>>>>> nodes size: 12)
> >>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>> org.apache.solr.common.cloud.ZkStateReader$3
> >>>>>>>>>>>> process
> >>>>>>>>>>>> INFO: Updating live nodes... (9)
> >>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>> runLeaderProcess
> >>>>>>>>>>>> INFO: Running the leader process.
> >>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>> shouldIBeLeader
> >>>>>>>>>>>> INFO: Checking if I should try and be
the leader.
> >>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>> shouldIBeLeader
> >>>>>>>>>>>> INFO: My last published State was Active,
it's okay to be the
> >>>>> leader.
> >>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>> runLeaderProcess
> >>>>>>>>>>>> INFO: I may be the new leader - try
and sync
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Apr 2, 2013 at 5:09 PM, Mark
Miller <
> >>>>> markrmiller@gmail.com
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I don't think the versions you are
thinking of apply here.
> >>>>> Peersync
> >>>>>>>>>>>>> does not look at that - it looks
at version numbers for
> >>>>> updates in
> >>>>>>>> the
> >>>>>>>>>>>>> transaction log - it compares the
last 100 of them on leader
> >>>>> and
> >>>>>>>> replica.
> >>>>>>>>>>>>> What it's saying is that the replica
seems to have versions
> >>>>> that
> >>>>>>>> the leader
> >>>>>>>>>>>>> does not. Have you scanned the logs
for any interesting
> >>>>> exceptions?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Did the leader change during the
heavy indexing? Did any zk
> >>>>> session
> >>>>>>>>>>>>> timeouts occur?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Apr 2, 2013, at 4:52 PM, Jamie
Johnson <jej2003@gmail.com
> >
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I am currently looking at moving
our Solr cluster to 4.2 and
> >>>>>>>> noticed a
> >>>>>>>>>>>>>> strange issue while testing
today.  Specifically the replica
> >>>>> has a
> >>>>>>>>>>>>> higher
> >>>>>>>>>>>>>> version than the master which
is causing the index to not
> >>>>>>>> replicate.
> >>>>>>>>>>>>>> Because of this the replica
has fewer documents than the
> >>>>> master.
> >>>>>>>> What
> >>>>>>>>>>>>>> could cause this and how can
I resolve it short of taking
> >>>>> down the
> >>>>>>>>>>>>> index
> >>>>>>>>>>>>>> and scping the right version
in?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> MASTER:
> >>>>>>>>>>>>>> Last Modified:about an hour
ago
> >>>>>>>>>>>>>> Num Docs:164880
> >>>>>>>>>>>>>> Max Doc:164880
> >>>>>>>>>>>>>> Deleted Docs:0
> >>>>>>>>>>>>>> Version:2387
> >>>>>>>>>>>>>> Segment Count:23
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> REPLICA:
> >>>>>>>>>>>>>> Last Modified: about an hour
ago
> >>>>>>>>>>>>>> Num Docs:164773
> >>>>>>>>>>>>>> Max Doc:164773
> >>>>>>>>>>>>>> Deleted Docs:0
> >>>>>>>>>>>>>> Version:3001
> >>>>>>>>>>>>>> Segment Count:30
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> in the replicas log it says
this:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> INFO: Creating new http client,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>>>
> config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM org.apache.solr.update.PeerSync
sync
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2
> >>>>>>>>>>>>>> url=http://10.38.33.17:7577/solrSTART
replicas=[
> >>>>>>>>>>>>>> http://10.38.33.16:7575/solr/dsc-shard5-core1/]
> nUpdates=100
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM org.apache.solr.update.PeerSync
> >>>>>>>> handleVersions
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2
url=
> >>>>>>>>>>>>> http://10.38.33.17:7577/solr
> >>>>>>>>>>>>>> Received 100 versions from
> >>>>> 10.38.33.16:7575/solr/dsc-shard5-core1/
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM org.apache.solr.update.PeerSync
> >>>>>>>> handleVersions
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2
url=
> >>>>>>>>>>>>> http://10.38.33.17:7577/solr  Our
> >>>>>>>>>>>>>> versions are newer. ourLowThreshold=1431233788792274944
> >>>>>>>>>>>>>> otherHigh=1431233789440294912
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM org.apache.solr.update.PeerSync
sync
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2
> >>>>>>>>>>>>>> url=http://10.38.33.17:7577/solrDONE.
sync succeeded
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> which again seems to point that
it thinks it has a newer
> >>>>> version of
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> index so it aborts.  This happened
while having 10 threads
> >>>>> indexing
> >>>>>>>>>>>>> 10,000
> >>>>>>>>>>>>>> items writing to a 6 shard (1
replica each) cluster.  Any
> >>>>> thoughts
> >>>>>>>> on
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>> or what I should look for would
be appreciated.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message