lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Chen <tomchen1...@gmail.com>
Subject Solr Map Reduce Indexer Tool GoLive to SolrCloud with index on local file system
Date Wed, 02 Jul 2014 22:23:29 GMT
Hi,


When we run Solr Map Reduce Indexer Tool (
https://github.com/markrmiller/solr-map-reduce-example), it generates
indexes on HDFS

The last stage is Go Live to merge the generated index to live SolrCloud
index.

If the live SolrCloud write index to local file system (rather than HDFS),
the Go Live gives such error like this:

2014-07-02 13:41:01,518 INFO org.apache.solr.hadoop.GoLive: Live merge
hdfs://
bdvs086.test.com:9000/tmp/0000088-140618120223665-oozie-oozi-W/results/part-00000
into http://bdvs087.test.com:8983/solr
2014-07-02 13:41:01,796 ERROR org.apache.solr.hadoop.GoLive: Error sending
live merge command
java.util.concurrent.ExecutionException:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
directory '/opt/testdir/solr/node/hdfs:/
bdvs086.test.com:9000/tmp/0000088-140618120223665-oozie-oozi-W/results/part-00001/data/index'
does not exist
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:233)
at java.util.concurrent.FutureTask.get(FutureTask.java:94)
at org.apache.solr.hadoop.GoLive.goLive(GoLive.java:126)
at
org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:867)
at
org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:609)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.solr.hadoop.MapReduceIndexerTool.main(MapReduceIndexerTool.java:596)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:434)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
directory '/opt/testdir/solr/node/hdfs:/
bdvs086.test.com:9000/tmp/0000088-140618120223665-oozie-oozi-W/results/part-00001/data/index'
does not exist
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at
org.apache.solr.client.solrj.request.CoreAdminRequest.process(CoreAdminRequest.java:493)
at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:100)
at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:89)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:452)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)

Any way to setup SolrCloud to write index to local file system, while
allowing the Solr MapReduceIndexerTool's GoLive to merge index generated on
HDFS to the SolrCloud?

Thanks,
Tom

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message