nutch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Pappert ...@fwpsystems.com>
Subject Re: Solr Indexing Problem
Date Fri, 02 Dec 2011 17:46:47 GMT
Hello,

do I have to point the "segment parameter" of
the readseg command to the "timestamp directory"?

hadoop dfs -ls crawl/segments/20111130041413/
Found 7 items
-rw-r--r--   2 fwp supergroup          0 2011-11-30 15:55 /user/fwp/crawl/segments/20111130041413/_SUCCESS
drwxr-xr-x   - fwp supergroup          0 2011-11-30 15:55 /user/fwp/crawl/segments/20111130041413/content
drwxr-xr-x   - fwp supergroup          0 2011-11-30 15:55 /user/fwp/crawl/segments/20111130041413/crawl_fetch
drwxr-xr-x   - fwp supergroup          0 2011-11-30 15:55 /user/fwp/crawl/segments/20111130041413/crawl_generate
drwxr-xr-x   - fwp supergroup          0 2011-11-30 15:55 /user/fwp/crawl/segments/20111130041413/crawl_parse
drwxr-xr-x   - fwp supergroup          0 2011-11-30 15:55 /user/fwp/crawl/segments/20111130041413/parse_data
drwxr-xr-x   - fwp supergroup          0 2011-11-30 15:55 /user/fwp/crawl/segments/20111130041413/parse_text


like this?

nutch readseg -list crawl/segments/20111130041413/

I got still the same exception for every segment.

Exception in thread "main" java.io.EOFException
	at java.io.DataInputStream.readFully(DataInputStream.java:180)
	at java.io.DataInputStream.readFully(DataInputStream.java:152)
	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
	at org.apache.hadoop.mapred.SequenceFileOutputFormat.getReaders(SequenceFileOutputFormat.java:93)
	at org.apache.nutch.segment.SegmentReader.getStats(SegmentReader.java:455)
	at org.apache.nutch.segment.SegmentReader.list(SegmentReader.java:433)
	at org.apache.nutch.segment.SegmentReader.main(SegmentReader.java:579)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

But the solrindex command works for all the segments, except one. 

I'm not sure about the usage of the segment reader.

greetz,
Rafael.

-
On 1/Dec/ 2011, at 11:02 , Markus Jelsma wrote:

> Seems corrupt or empty. Use segment reader to check out the segment.
> 
> On Wednesday 30 November 2011 23:31:53 Rafael Pappert wrote:
>> Hello List,
>> 
>> I try to index my parsed into solr with the solrindex command,
>> but on one segment I got the following exception:
>> 
>> java.io.EOFException
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>> 	at
>> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecor
>> dReader.java:43) at
>> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceF
>> ileInputFormat.java:63) at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:1
>> 97) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at
>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:396)
>> 	at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>> java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253)
>> 
>> java.io.EOFException
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>> 	at
>> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecor
>> dReader.java:43) at
>> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceF
>> ileInputFormat.java:63) at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:1
>> 97) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at
>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:396)
>> 	at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>> java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253)
>> 
>> java.io.EOFException
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>> 	at
>> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecor
>> dReader.java:43) at
>> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceF
>> ileInputFormat.java:63) at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:1
>> 97) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at
>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:396)
>> 	at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>> java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253)
>> 
>> java.io.EOFException
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> 	at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>> 	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>> 	at
>> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecor
>> dReader.java:43) at
>> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceF
>> ileInputFormat.java:63) at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:1
>> 97) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at
>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:396)
>> 	at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>> java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253)
>> 
>> Whats wrong with this segment? All of the other commands worked without any
>> problem.
>> 
>> Best regards,
>> Rafael.
> 
> -- 
> Markus Jelsma - CTO - Openindex


Mime
View raw message