lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bjarke Buur Mortensen <morten...@eluence.com>
Subject Re: Recipe for moving to solr cloud without reindexing
Date Wed, 08 Aug 2018 13:34:28 GMT
Erick,

thanks, that is of course something I left out of the original question.
Our Solr is 7.1, so that should not present a problem (crossing fingers).

However, on my dev box I'm trying out the steps, and here I have some
segments created with version 6 of Solr.

After having copied data from my non-cloud solr into my
single-shard-single-replica collection and verified that Solr Cloud works
with this collection, I then submit the splitshard command

http://172.17.0.4:8984
/solr/admin/collections?action=SPLITSHARD&collection=procurement&shard=shard1

However, this gives me the error:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://172.17.0.4:8984/solr:
java.lang.IllegalArgumentException: Cannot merge a segment that has been
created with major version 6 into this index which has been created by
major version 7"}

I have tried running both optimize and IndexUpgrader on the index before
shard splitting, but the same error still occurs.

Any ideas as to why this happens?

Below is an output from running IndexUpgrader, which I cannot decipher.
It both states that "All segments upgraded to version 7.1.0" and ''all
running merges have aborted" ¯\_(ツ)_/¯

Thanks a lot,
Bjarke


======================
java -cp
/opt/solr/server/solr-webapp/webapp/WEB-INF/lib/lucene-backward-codecs-7.1.0.jar:/opt/solr/server/solr-webapp/webapp/WEB-INF/lib/lucene-core-7.1.0.jar
org.apache.lucene.index.IndexUpgrader -delete-prior-commits -verbose
/var/solr/cloud/procurement_shard1_replica_n1/data/index
IFD 0 [2018-08-08T13:00:18.244Z; main]: init: current segments file is
"segments_4vs";
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@721e0f4f
IFD 0 [2018-08-08T13:00:18.266Z; main]: init: load commit "segments_4vs"
IFD 0 [2018-08-08T13:00:18.270Z; main]: now checkpoint
"_bhg(7.1.0):C108396" [1 segments ; isCommit = false]
IFD 0 [2018-08-08T13:00:18.270Z; main]: 0 msec to checkpoint
IW 0 [2018-08-08T13:00:18.270Z; main]: init: create=false
IW 0 [2018-08-08T13:00:18.273Z; main]:
dir=MMapDirectory@/var/solr/cloud/procurement_shard1_replica_n1/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2
index=_bhg(7.1.0):C108396
version=7.1.0
analyzer=null
ramBufferSizeMB=16.0
maxBufferedDocs=-1
mergedSegmentWarmer=null
delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
commit=null
openMode=CREATE_OR_APPEND
similarity=org.apache.lucene.search.similarities.BM25Similarity
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1,
maxMergeCount=-1, ioThrottle=true
codec=Lucene70
infoStream=org.apache.lucene.util.PrintStreamInfoStream
mergePolicy=UpgradeIndexMergePolicy([TieredMergePolicy: maxMergeAtOnce=10,
maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0,
forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0,
maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1)
indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@5ba23b66
readerPooling=true
perThreadHardLimitMB=1945
useCompoundFile=true
commitOnClose=true
indexSort=null
writer=org.apache.lucene.index.IndexWriter@2ff4f00f

IW 0 [2018-08-08T13:00:18.273Z; main]: MMapDirectory.UNMAP_SUPPORTED=true
IndexUpgrader 0 [2018-08-08T13:00:18.274Z; main]: Upgrading all pre-7.1.0
segments of index directory
'MMapDirectory@/var/solr/cloud/procurement_shard1_replica_n1/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2' to
version 7.1.0...
IW 0 [2018-08-08T13:00:18.274Z; main]: forceMerge: index now
_bhg(7.1.0):C108396
IW 0 [2018-08-08T13:00:18.274Z; main]: now flush at forceMerge
IW 0 [2018-08-08T13:00:18.274Z; main]:   start flush: applyAllDeletes=true
IW 0 [2018-08-08T13:00:18.274Z; main]:   index before flush
_bhg(7.1.0):C108396
DW 0 [2018-08-08T13:00:18.274Z; main]: startFullFlush
DW 0 [2018-08-08T13:00:18.275Z; main]: main finishFullFlush success=true
IW 0 [2018-08-08T13:00:18.275Z; main]: now apply all deletes for all
segments buffered updates bytesUsed=0 reader pool bytesUsed=0
BD 0 [2018-08-08T13:00:18.275Z; main]: waitApply: no deletes to apply
UPGMP 0 [2018-08-08T13:00:18.276Z; main]: findForcedMerges:
segmentsToUpgrade={}
MS 0 [2018-08-08T13:00:18.282Z; main]: initDynamicDefaults spins=true
maxThreadCount=1 maxMergeCount=6
MS 0 [2018-08-08T13:00:18.282Z; main]: now merge
MS 0 [2018-08-08T13:00:18.282Z; main]:   index: _bhg(7.1.0):C108396
MS 0 [2018-08-08T13:00:18.282Z; main]:   no more merges pending; now return
IndexUpgrader 0 [2018-08-08T13:00:18.282Z; main]: All segments upgraded to
version 7.1.0
IndexUpgrader 0 [2018-08-08T13:00:18.283Z; main]: Enforcing commit to
rewrite all index metadata...
IW 0 [2018-08-08T13:00:18.283Z; main]: commit: start
IW 0 [2018-08-08T13:00:18.283Z; main]: commit: enter lock
IW 0 [2018-08-08T13:00:18.283Z; main]: commit: now prepare
IW 0 [2018-08-08T13:00:18.283Z; main]: prepareCommit: flush
IW 0 [2018-08-08T13:00:18.283Z; main]:   index before flush
_bhg(7.1.0):C108396
DW 0 [2018-08-08T13:00:18.283Z; main]: startFullFlush
IW 0 [2018-08-08T13:00:18.283Z; main]: now apply all deletes for all
segments buffered updates bytesUsed=0 reader pool bytesUsed=0
BD 0 [2018-08-08T13:00:18.283Z; main]: waitApply: no deletes to apply
DW 0 [2018-08-08T13:00:18.284Z; main]: main finishFullFlush success=true
IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit(): start
IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit
index=_bhg(7.1.0):C108396 changeCount=2
IW 0 [2018-08-08T13:00:18.293Z; main]: startCommit: wrote pending segments
file "pending_segments_4vt"
IW 0 [2018-08-08T13:00:18.295Z; main]: done all syncs:
[_bhg_Lucene50_0.tip, _bhg.fdx, _bhg.fnm, _bhg.nvm, _bhg.fdt, _bhg.si,
_bhg_Lucene50_0.pos, _bhg.nvd, _bhg_Lucene50_0.doc, _bhg_Lucene50_0.tim]
IW 0 [2018-08-08T13:00:18.295Z; main]: commit: pendingCommit != null
IW 0 [2018-08-08T13:00:18.298Z; main]: commit: done writing segments file
"segments_4vt"
IFD 0 [2018-08-08T13:00:18.298Z; main]: now checkpoint
"_bhg(7.1.0):C108396" [1 segments ; isCommit = true]
IFD 0 [2018-08-08T13:00:18.298Z; main]: deleteCommits: now decRef commit
"segments_4vs"
IFD 0 [2018-08-08T13:00:18.298Z; main]: delete [segments_4vs]
IFD 0 [2018-08-08T13:00:18.299Z; main]: 0 msec to checkpoint
IW 0 [2018-08-08T13:00:18.319Z; main]: commit: took 16.0 msec
IW 0 [2018-08-08T13:00:18.319Z; main]: commit: done
IndexUpgrader 0 [2018-08-08T13:00:18.319Z; main]: Committed upgraded
metadata to index.
IW 0 [2018-08-08T13:00:18.319Z; main]: now flush at close
IW 0 [2018-08-08T13:00:18.319Z; main]:   start flush: applyAllDeletes=true
IW 0 [2018-08-08T13:00:18.319Z; main]:   index before flush
_bhg(7.1.0):C108396
DW 0 [2018-08-08T13:00:18.319Z; main]: startFullFlush
DW 0 [2018-08-08T13:00:18.320Z; main]: main finishFullFlush success=true
IW 0 [2018-08-08T13:00:18.320Z; main]: now apply all deletes for all
segments buffered updates bytesUsed=0 reader pool bytesUsed=0
BD 0 [2018-08-08T13:00:18.320Z; main]: waitApply: no deletes to apply
MS 0 [2018-08-08T13:00:18.320Z; main]: updateMergeThreads ioThrottle=true
targetMBPerSec=10240.0 MB/sec
MS 0 [2018-08-08T13:00:18.320Z; main]: now merge
MS 0 [2018-08-08T13:00:18.321Z; main]:   index: _bhg(7.1.0):C108396
MS 0 [2018-08-08T13:00:18.321Z; main]:   no more merges pending; now return
IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges
IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges done
IW 0 [2018-08-08T13:00:18.321Z; main]: commit: start
IW 0 [2018-08-08T13:00:18.321Z; main]: commit: enter lock
IW 0 [2018-08-08T13:00:18.321Z; main]: commit: now prepare
IW 0 [2018-08-08T13:00:18.321Z; main]: prepareCommit: flush
IW 0 [2018-08-08T13:00:18.321Z; main]:   index before flush
_bhg(7.1.0):C108396
DW 0 [2018-08-08T13:00:18.321Z; main]: startFullFlush
IW 0 [2018-08-08T13:00:18.321Z; main]: now apply all deletes for all
segments buffered updates bytesUsed=0 reader pool bytesUsed=0
BD 0 [2018-08-08T13:00:18.322Z; main]: waitApply: no deletes to apply
DW 0 [2018-08-08T13:00:18.322Z; main]: main finishFullFlush success=true
IW 0 [2018-08-08T13:00:18.322Z; main]: startCommit(): start
IW 0 [2018-08-08T13:00:18.322Z; main]:   skip startCommit(): no changes
pending
IW 0 [2018-08-08T13:00:18.322Z; main]: commit: pendingCommit == null; skip
IW 0 [2018-08-08T13:00:18.322Z; main]: commit: took 0.9 msec
IW 0 [2018-08-08T13:00:18.322Z; main]: commit: done
IW 0 [2018-08-08T13:00:18.322Z; main]: rollback
IW 0 [2018-08-08T13:00:18.322Z; main]: all running merges have aborted
IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: done finish merges
DW 0 [2018-08-08T13:00:18.323Z; main]: abort
DW 0 [2018-08-08T13:00:18.323Z; main]: done abort success=true
IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: infos=_bhg(7.1.0):C108396
IFD 0 [2018-08-08T13:00:18.323Z; main]: now checkpoint
"_bhg(7.1.0):C108396" [1 segments ; isCommit = false]
IFD 0 [2018-08-08T13:00:18.323Z; main]: 0 msec to checkpoint

2018-08-07 16:38 GMT+02:00 Erick Erickson <erickerickson@gmail.com>:

> Bjarke:
>
> One thing, what version of Solr are you moving _from_ and _to_?
> Solr/Lucene only guarantee one major backward revision so you can copy
> an index created with Solr 6 to another Solr 6 or Solr 7, but you
> couldn't copy an index created with Solr 5 to Solr 7...
>
> Also note that shard splitting is a very expensive operation, so be
> patient....
>
> Best,
> Erick
>
> On Tue, Aug 7, 2018 at 6:17 AM, Rahul Singh
> <rahul.xavier.singh@gmail.com> wrote:
> > Bjarke,
> >
> > I am imagining that at some point you may need to shard that data if it
> grows. Or do you imagine this data to remain stagnant?
> >
> > Generally you want to add solrcloud to do two things : 1. Increase
> availability with replicas 2. Increase available data via shards 3.
> Increase fault tolerance with leader and replicas being spread around the
> cluster.
> >
> > You would be bypassing general High availability / distributed computing
> processes by trying to not reindex.
> >
> > Rahul
> > On Aug 7, 2018, 7:06 AM -0400, Bjarke Buur Mortensen <
> mortensen@eluence.com>, wrote:
> >> Hi List,
> >>
> >> is there a cookbook recipe for moving an existing solr core to a solr
> cloud
> >> collection.
> >>
> >> We currently have a single machine with a large core (~150gb), and we
> would
> >> like to move to solr cloud.
> >>
> >> I haven't been able to find anything that reuses an existing index, so
> any
> >> pointers much appreciated.
> >>
> >> Thanks,
> >> Bjarke
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message