lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alessio crisantemi <alessio.crisant...@gmail.com>
Subject Re: nutch in solr
Date Sun, 05 Feb 2012 17:32:01 GMT
if I look the solr and nuth libs I found:
apache-solr-solrj-1.4.1.jar on Solr
and
solr-solrj-3.4.0.jar

this are the only jar files with a word 'solrj'....
taht's the problem?!

2012/2/5 Geek Gamer <geek4377@gmail.com>

> looks like solrj version in nutch classpath is different that the solr
> version on server,
> can you  post the versions for both nutch and solr?
>
>
> On Sun, Feb 5, 2012 at 10:24 PM, alessio crisantemi
> <alessio.crisantemi@gmail.com> wrote:
> > no, all run on port 8983.
> > ..
> >
> > 2012/2/5 Matthew Parker <mparker@apogeeintegration.com>
> >
> >> Doesn't tomcat run on port 8080, and not port 8983? Or did you change
> the
> >> tomcat's default port to 8983?
> >> On Feb 5, 2012 5:17 AM, "alessio crisantemi" <
> alessio.crisantemi@gmail.com
> >> >
> >> wrote:
> >>
> >> > Hi All,
> >> > I have some problems with integration of Nutch in Solr and Tomcat.
> >> >
> >> > I follo Nutch tutorial for integration and now, I can crawl a website:
> >> all
> >> > works right.
> >> > But It I try the solr integration, I can't indexing on Solr.
> >> >
> >> > follow the nutch output after the command:
> >> > bin/nutch crawl urls -solr http://127.0.0.1:8983/solr/ -depth 3
> -topN 5
> >> >
> >> > I read "java.lang.RuntimeException: Invalid version (expected 2, but
> 1)
> >> or
> >> > the data in not in 'javabin' format"
> >> > MAY BE THERE IS A PROBLEM BETWEEN NUTCH 1.4 VERSION AND SOLR 1.4.1?
> MAY
> >> BE
> >> > IT REQUIRE A 3.X SOLR VERSION?
> >> >
> >> > thanks,
> >> > a.
> >> >
> >> > crawl started in: crawl-20120203151719
> >> > rootUrlDir = urls
> >> > threads = 10
> >> > depth = 3
> >> > solrUrl=http://127.0.0.1:8983/solr/
> >> > topN = 5
> >> > Injector: starting at 2012-02-03 15:17:20
> >> > Injector: crawlDb: crawl-20120203151719/crawldb
> >> > Injector: urlDir: urls
> >> > Injector: Converting injected urls to crawl db entries.
> >> > Injector: Merging injected urls into crawl db.
> >> > Injector: finished at 2012-02-03 15:17:31, elapsed: 00:00:10
> >> > Generator: starting at 2012-02-03 15:17:31
> >> > Generator: Selecting best-scoring urls due for fetch.
> >> > Generator: filtering: true
> >> > Generator: normalizing: true
> >> > Generator: topN: 5
> >> > Generator: jobtracker is 'local', generating exactly one partition.
> >> > Generator: Partitioning selected urls for politeness.
> >> > Generator: segment: crawl-20120203151719/segments/20120203151735
> >> > Generator: finished at 2012-02-03 15:17:39, elapsed: 00:00:07
> >> > Fetcher: Your 'http.agent.name' value should be listed first in
> >> > 'http.robots.agents' property.
> >> > Fetcher: starting at 2012-02-03 15:17:39
> >> > Fetcher: segment: crawl-20120203151719/segments/20120203151735
> >> > Using queue mode : byHost
> >> > Fetcher: threads: 10
> >> > Fetcher: time-out divisor: 2
> >> > QueueFeeder finished: total 1 records + hit by time limit :0
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > fetching http://www.gioconews.it/
> >> > Using queue mode : byHost
> >> > -finishing thread FetcherThread, activeThreads=3
> >> > -finishing thread FetcherThread, activeThreads=2
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > Fetcher: throughput threshold: -1
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > Fetcher: throughput threshold retries: 5
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > fetch of http://www.gioconews.it/ failed with:
> >> > java.net.UnknownHostException: www.gioconews.it
> >> > -finishing thread FetcherThread, activeThreads=0
> >> > -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=0
> >> > Fetcher: finished at 2012-02-03 15:17:44, elapsed: 00:00:05
> >> > ParseSegment: starting at 2012-02-03 15:17:44
> >> > ParseSegment: segment: crawl-20120203151719/segments/20120203151735
> >> > ParseSegment: finished at 2012-02-03 15:17:48, elapsed: 00:00:04
> >> > CrawlDb update: starting at 2012-02-03 15:17:48
> >> > CrawlDb update: db: crawl-20120203151719/crawldb
> >> > CrawlDb update: segments:
> [crawl-20120203151719/segments/20120203151735]
> >> > CrawlDb update: additions allowed: true
> >> > CrawlDb update: URL normalizing: true
> >> > CrawlDb update: URL filtering: true
> >> > CrawlDb update: 404 purging: false
> >> > CrawlDb update: Merging segment data into db.
> >> > CrawlDb update: finished at 2012-02-03 15:17:53, elapsed: 00:00:05
> >> > Generator: starting at 2012-02-03 15:17:53
> >> > Generator: Selecting best-scoring urls due for fetch.
> >> > Generator: filtering: true
> >> > Generator: normalizing: true
> >> > Generator: topN: 5
> >> > Generator: jobtracker is 'local', generating exactly one partition.
> >> > Generator: 0 records selected for fetching, exiting ...
> >> > Stopping at depth=1 - no more URLs to fetch.
> >> > LinkDb: starting at 2012-02-03 15:17:57
> >> > LinkDb: linkdb: crawl-20120203151719/linkdb
> >> > LinkDb: URL normalize: true
> >> > LinkDb: URL filter: true
> >> > LinkDb: adding segment:
> >> >
> >> >
> >>
> file:/C:/temp/apache-nutch-1.4-bin/runtime/local/crawl-20120203151719/segments/20120203151735
> >> > LinkDb: finished at 2012-02-03 15:18:01, elapsed: 00:00:04
> >> > SolrIndexer: starting at 2012-02-03 15:18:01
> >> > java.lang.RuntimeException: Invalid version (expected 2, but 1) or the
> >> data
> >> > in not in 'javabin' format
> >> > SolrDeleteDuplicates: starting at 2012-02-03 15:18:09
> >> > SolrDeleteDuplicates: Solr url: http://127.0.0.1:8983/solr/
> >> > Exception in thread "main" java.io.IOException:
> >> > org.apache.solr.client.solrj.SolrServerException: Error executing
> query
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getSplits(SolrDeleteDuplicates.java:200)
> >> >        at
> >> > org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
> >> >        at
> >> >
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
> >> >        at
> >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> >> >        at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
> >> >        at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> >> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >> >        at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> >> > Caused by: org.apache.solr.client.solrj.SolrServerException: Error
> >> > executing query
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
> >> >        at
> >> > org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getSplits(SolrDeleteDuplicates.java:198)
> >> >        ... 9 more
> >> > Caused by: java.lang.RuntimeException: Invalid version (expected 2,
> but
> >> 1)
> >> > or the data in not in 'javabin' format
> >> >        at
> >> >
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:472)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
> >> >        ... 11 more
> >> > Alessio@PC-Alessio/cygdrive/c/temp/apache-nutch-1.4-bin/runtime/local
> >> > $ bin/nutch crawl urls -solr http://127.0.0.1:8983/solr/ -depth 3
> -topN
> >> 5
> >> > crawl started in: crawl-20120203162510
> >> > rootUrlDir = urls
> >> > threads = 10
> >> > depth = 3
> >> > solrUrl=http://127.0.0.1:8983/solr/
> >> > topN = 5
> >> > Injector: starting at 2012-02-03 16:25:11
> >> > Injector: crawlDb: crawl-20120203162510/crawldb
> >> > Injector: urlDir: urls
> >> > Injector: Converting injected urls to crawl db entries.
> >> > Injector: Merging injected urls into crawl db.
> >> > Injector: finished at 2012-02-03 16:25:20, elapsed: 00:00:09
> >> > Generator: starting at 2012-02-03 16:25:20
> >> > Generator: Selecting best-scoring urls due for fetch.
> >> > Generator: filtering: true
> >> > Generator: normalizing: true
> >> > Generator: topN: 5
> >> > Generator: jobtracker is 'local', generating exactly one partition.
> >> > Generator: Partitioning selected urls for politeness.
> >> > Generator: segment: crawl-20120203162510/segments/20120203162525
> >> > Generator: finished at 2012-02-03 16:25:28, elapsed: 00:00:08
> >> > Fetcher: Your 'http.agent.name' value should be listed first in
> >> > 'http.robots.agents' property.
> >> > Fetcher: starting at 2012-02-03 16:25:28
> >> > Fetcher: segment: crawl-20120203162510/segments/20120203162525
> >> > Using queue mode : byHost
> >> > Fetcher: threads: 10
> >> > Fetcher: time-out divisor: 2
> >> > QueueFeeder finished: total 1 records + hit by time limit :0
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > fetching http://www.gioconews.it/
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Using queue mode : byHost
> >> > Fetcher: throughput threshold: -1
> >> > Fetcher: throughput threshold retries: 5
> >> > -finishing thread FetcherThread, activeThreads=2
> >> > -finishing thread FetcherThread, activeThreads=3
> >> > -finishing thread FetcherThread, activeThreads=6
> >> > -finishing thread FetcherThread, activeThreads=5
> >> > -finishing thread FetcherThread, activeThreads=5
> >> > -finishing thread FetcherThread, activeThreads=4
> >> > -finishing thread FetcherThread, activeThreads=3
> >> > -finishing thread FetcherThread, activeThreads=2
> >> > -finishing thread FetcherThread, activeThreads=1
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> >> > fetch of http://www.gioconews.it/ failed with:
> >> > java.net.UnknownHostException: www.gioconews.it
> >> > -finishing thread FetcherThread, activeThreads=0
> >> > -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
> >> > -activeThreads=0
> >> > Fetcher: finished at 2012-02-03 16:25:47, elapsed: 00:00:18
> >> > ParseSegment: starting at 2012-02-03 16:25:47
> >> > ParseSegment: segment: crawl-20120203162510/segments/20120203162525
> >> > ParseSegment: finished at 2012-02-03 16:25:51, elapsed: 00:00:04
> >> > CrawlDb update: starting at 2012-02-03 16:25:52
> >> > CrawlDb update: db: crawl-20120203162510/crawldb
> >> > CrawlDb update: segments:
> [crawl-20120203162510/segments/20120203162525]
> >> > CrawlDb update: additions allowed: true
> >> > CrawlDb update: URL normalizing: true
> >> > CrawlDb update: URL filtering: true
> >> > CrawlDb update: 404 purging: false
> >> > CrawlDb update: Merging segment data into db.
> >> > CrawlDb update: finished at 2012-02-03 16:25:57, elapsed: 00:00:05
> >> > Generator: starting at 2012-02-03 16:25:58
> >> > Generator: Selecting best-scoring urls due for fetch.
> >> > Generator: filtering: true
> >> > Generator: normalizing: true
> >> > Generator: topN: 5
> >> > Generator: jobtracker is 'local', generating exactly one partition.
> >> > Generator: 0 records selected for fetching, exiting ...
> >> > Stopping at depth=1 - no more URLs to fetch.
> >> > LinkDb: starting at 2012-02-03 16:26:01
> >> > LinkDb: linkdb: crawl-20120203162510/linkdb
> >> > LinkDb: URL normalize: true
> >> > LinkDb: URL filter: true
> >> > LinkDb: adding segment:
> >> >
> >> >
> >>
> file:/C:/temp/apache-nutch-1.4-bin/runtime/local/crawl-20120203162510/segments/20120203162525
> >> > LinkDb: finished at 2012-02-03 16:26:05, elapsed: 00:00:04
> >> > SolrIndexer: starting at 2012-02-03 16:26:06
> >> > java.lang.RuntimeException: Invalid version (expected 2, but 1) or the
> >> data
> >> > in not in 'javabin' format
> >> > SolrDeleteDuplicates: starting at 2012-02-03 16:26:13
> >> > SolrDeleteDuplicates: Solr url: http://127.0.0.1:8983/solr/
> >> > Exception in thread "main" java.io.IOException:
> >> > org.apache.solr.client.solrj.SolrServerException: Error executing
> query
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getSplits(SolrDeleteDuplicates.java:200)
> >> >        at
> >> > org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
> >> >        at
> >> >
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
> >> >        at
> >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> >> >        at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
> >> >        at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> >> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >> >        at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> >> > Caused by: org.apache.solr.client.solrj.SolrServerException: Error
> >> > executing query
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
> >> >        at
> >> > org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
> >> >        at
> >> >
> >> >
> >>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getSplits(SolrDeleteDuplicates.java:198)
> >> >        ... 9 more
> >> > Caused by: java.lang.RuntimeException: Invalid version (expected 2,
> but
> >> 1)
> >> > or the data in not in 'javabin' format
> >> >        at
> >> >
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:472)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
> >> >        at
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
> >> >        ... 11 more
> >> >
> >>
> >> ------------------------------
> >> This e-mail and any files transmitted with it may be proprietary.
>  Please
> >> note that any views or opinions presented in this e-mail are solely
> those
> >> of the author and do not necessarily represent those of Apogee
> Integration.
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message