lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Solr Map Reduce Indexer Tool GoLive to SolrCloud with index on local file system
Date Thu, 03 Jul 2014 01:50:06 GMT
How would the MapReduceIndexerTool (MRIT for short)
find the local disk to write from HDFS to for each shard?
All it has is the information in the Solr configs, which are
usually relative paths on the local Solr machines, relative
to SOLR_HOME. Which could be different on each node
(that would be screwy, but possible).

Permissions would also be a royal pain to get right....

You _can_ forego the --go-live option and copy from
the HDFS nodes to your local drive and then execute
the "mergeIndexes" command, see:
https://cwiki.apache.org/confluence/display/solr/Merging+Indexes
Note that there is the MergeIndexTool, but there's also
the Core Admin command.

The sub-indexes are in a partition in HDFS and numbered
sequentially.

Best,
Erick

On Wed, Jul 2, 2014 at 3:23 PM, Tom Chen <tomchen1000@gmail.com> wrote:
> Hi,
>
>
> When we run Solr Map Reduce Indexer Tool (
> https://github.com/markrmiller/solr-map-reduce-example), it generates
> indexes on HDFS
>
> The last stage is Go Live to merge the generated index to live SolrCloud
> index.
>
> If the live SolrCloud write index to local file system (rather than HDFS),
> the Go Live gives such error like this:
>
> 2014-07-02 13:41:01,518 INFO org.apache.solr.hadoop.GoLive: Live merge
> hdfs://
> bdvs086.test.com:9000/tmp/0000088-140618120223665-oozie-oozi-W/results/part-00000
> into http://bdvs087.test.com:8983/solr
> 2014-07-02 13:41:01,796 ERROR org.apache.solr.hadoop.GoLive: Error sending
> live merge command
> java.util.concurrent.ExecutionException:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> directory '/opt/testdir/solr/node/hdfs:/
> bdvs086.test.com:9000/tmp/0000088-140618120223665-oozie-oozi-W/results/part-00001/data/index'
> does not exist
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:233)
> at java.util.concurrent.FutureTask.get(FutureTask.java:94)
> at org.apache.solr.hadoop.GoLive.goLive(GoLive.java:126)
> at
> org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:867)
> at
> org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:609)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.solr.hadoop.MapReduceIndexerTool.main(MapReduceIndexerTool.java:596)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
> at java.lang.reflect.Method.invoke(Method.java:611)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:434)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(AccessController.java:310)
> at javax.security.auth.Subject.doAs(Subject.java:573)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> directory '/opt/testdir/solr/node/hdfs:/
> bdvs086.test.com:9000/tmp/0000088-140618120223665-oozie-oozi-W/results/part-00001/data/index'
> does not exist
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
> at
> org.apache.solr.client.solrj.request.CoreAdminRequest.process(CoreAdminRequest.java:493)
> at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:100)
> at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:89)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:452)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> at java.lang.Thread.run(Thread.java:738)
>
> Any way to setup SolrCloud to write index to local file system, while
> allowing the Solr MapReduceIndexerTool's GoLive to merge index generated on
> HDFS to the SolrCloud?
>
> Thanks,
> Tom

Mime
View raw message