Re: State of nutchbase

2009-12-08T01:11:50Z
Hey everyone,

So I restarted nutchbase efforts with adding an abstraction to the hbase
api. The idea is to use an intermediate nutch api (which then talks with
hbase) instead of communicating with hbase directly. This allows us a) to
not be completely tied down to hbase, making a move to another db in the
future easier b) perhaps to immediately support multiple databases with easy
data migration between them.

What I have is very very (VERY) early and extremely alpha but I am quite
happy with overall idea so I am sharing it for suggestions and reviews.
Again, instead of using hbase directly, nutch will use a nice java bean with
getters and setters. Nutch will then figure out what to read/write into
hbase.

I decided to use avro because it has a very clean design. Here is a  very
basic WebTableRow class:
{"namespace": "org.apache.nutch.storage",
 "protocol": "Web",

 "types": [
     {"name": "WebTableRow", "type": "record",
      "fields": [
          {"name": "rowKey", "type": "string"},
          {"name": "fetchTime", "type": "long"},
          {"name": "title", "type": "string"},
          {"name": "text", "type": "string"},
          {"name": "status", "type": "int"}
      ]
     }
 ]
}

(ignore "protocol". I haven't yet figured out how to compile schemas without
protocols)

I have copied and modified avro's SpecificCompiler to generate a java class.
It is mostly the same class as avro's SpecificCompiler however the variables
are all private and are accessed through getters and setters. Here is a
portion of the file:

public class WebTableRow extends NutchTableRow< Utf8> implements
SpecificRecord {
  @RowKey // these are used for reflection
  private Utf8 rowKey;
  @RowField
  private long fetchTime;
  @RowField
  private Utf8 title;
  @RowField
  private Utf8 text;
  @RowField
  private int status;
  public Utf8 getRowKey() { .... }
  public void setRowKey(Utf8 value) {....}
  public long getFetchTime() { .... }
  public void setFetchTime(long value) { .... }
  .....

Note that NutchTableRow extends SpecificRecordBase so this is a proper avro
record. In the future, once hadoop MR supports avro as a serialization
format NutchTableRow-s can easily be output through maps and reduces which
is a nice bonus.

We need to force the usage of setters instead of direct access to variables.
Because one of the nice things about hbase is that you only update the
columns that you changed. However to know which fields are updated (and
thus, map them to hbase columns), we must keep track of what changed.
Currently, NutchTableRow keeps a BitSet for all fields and all setter
functions update this BitSet so we know exactly what changed.

There is also a new interface called NutchSerializer that defines readRow
and writeRow methods(it also needs scans, delete rows etc.. but that's for
later). Currently HbaseSerializer implements NutchSerializer and reads and
writes WebTableRow-s. HbaseSerializer currently works via reflection. It
should be easy to add code generation to our SpecificCompiler so that we can
also output a WebTableRowHbaseSerializer along with WebTableRow instead of
using reflection.

What I have currently can read and write primitive types + strings into and
from hbase. You can check it out from github.com/dogacan/nutchbase (branch
master, package o.a.n.storage). Again, I would like to note that the code is
very very alpha and is not in a good shape but it should be a good starting
point if you are interested.

Once hbase support is solid, I intend to add support for other databases
(bdb, cassandra and sql come to mind). If I got everything right, then
moving data from one database to another is an incredibly trivial task. So,
you can start with, say, bdb then switch over to hbase once your data gets
large.

Oh I forgot... HbaseSerializer reads a hbase-mapping.xml file that describes
the mapping between fields and hbase columns:


     
    
  
    
    
    
    
  

Sorry for the long and rambling email. Feel free to ask if anything is
unclear (and I assume it must be, given my incoherent description :)
-- 
Doğacan Güney



Re: State of nutchbase
2009-12-06T20:55:42Z

Alban Mouton wrote:
> Hello,
> 
> I have looked a little into nutch code and mailing lists. I think the 
> nutchbase branch (http://issues.apache.org/jira/browse/NUTCH-650) is 
> very interesting, with a good potential to improve code clarity and 
> flexibility (I find data structure quite obscure in current version). 
> The issue is untouched since last august, so my question is : can 
> nutchbase really be part of nutch 1.1 ? 

Definitely no. Release 1.1 will be an update to 1.0, with no major 
design changes. However, we intend to integrate the nutchbase branch 
with trunk at some point - but since this would be a major change it 
would come under 2.0 branch or so ...


> Is there still much work to do 
> or is it almost ready ? Is it a worthy issue for an interested developer 
> with a (still !) limited knowledge of the project ?

Please contact Dogacan, who is leading the work on this branch. AFAIK 
he's going to update the design soon.

> 
> So far I have only tried to run nutchbase in eclipse by applying the 
> tutorial (http://wiki.apache.org/nutch/RunNutchInEclipse1.0) but I run 
> in errors when building, mostly from Parser and tests. I may start by 
> cleaning this up.

See above - please coordinate with Dogacan to avoid duplication of effort.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com






[jira] Commented: (NUTCH-770) Timebomb for Fetcher
2009-12-05T16:51:21Z


    [ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786443#action_12786443
] 

MilleBii commented on NUTCH-770:
--------------------------------

Tried it succesfully on a windows platform.

It does not work on a Ubuntu, pseudo-distributed hadoop configuration with mappers running
in parallel ????



> Timebomb for Fetcher
> --------------------
>
>                 Key: NUTCH-770
>                 URL: https://issues.apache.org/jira/browse/NUTCH-770
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch
>
>
> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is
not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes
is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder
skips all remaining entries then all active queues are purged. This allows to keep the Fetch
step under comtrol and works well in combination with NUTCH-769

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Issue Comment Edited: (NUTCH-770) Timebomb for Fetcher
2009-12-05T16:51:21Z


    [ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786443#action_12786443
] 

MilleBii edited comment on NUTCH-770 at 12/5/09 4:50 PM:
---------------------------------------------------------

Tried it succesfully on a windows platform.

It does not work on a Ubuntu, pseudo-distributed hadoop configuration with two mappers running
in parallel ????



      was (Author: millebii):
    Tried it succesfully on a windows platform.

It does not work on a Ubuntu, pseudo-distributed hadoop configuration with mappers running
in parallel ????


  
> Timebomb for Fetcher
> --------------------
>
>                 Key: NUTCH-770
>                 URL: https://issues.apache.org/jira/browse/NUTCH-770
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch
>
>
> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is
not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes
is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder
skips all remaining entries then all active queues are purged. This allows to keep the Fetch
step under comtrol and works well in combination with NUTCH-769

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






State of nutchbase
2009-12-05T14:56:49Z

Hello,

I have looked a little into nutch code and mailing lists. I think the
nutchbase branch (http://issues.apache.org/jira/browse/NUTCH-650) is very
interesting, with a good potential to improve code clarity and flexibility
(I find data structure quite obscure in current version). The issue is
untouched since last august, so my question is : can nutchbase really be
part of nutch 1.1 ? Is there still much work to do or is it almost ready ?
Is it a worthy issue for an interested developer with a (still !) limited
knowledge of the project ?

So far I have only tried to run nutchbase in eclipse by applying the
tutorial (http://wiki.apache.org/nutch/RunNutchInEclipse1.0) but I run in
errors when building, mostly from Parser and tests. I may start by cleaning
this up.

Eclipse build errors:

Description    Resource    Path    Location    Type
FetcherOutputFormat cannot be resolved to a type
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 362    Java Problem
Generator.GENERATE_MAX_PER_HOST_BY_IP cannot be resolved
TestGenerator.java    /nutchbase/src/test/org/apache/nutch/crawl    line
202    Java Problem
ParseImpl cannot be resolved to a type    ArcSegmentCreator.java
/nutchbase/src/java/org/apache/nutch/tools/arc    line 229    Java Problem
ParseImpl cannot be resolved to a type    BasicFields.java
/nutchbase/src/java/org/apache/nutch/indexer/field    line 335    Java
Problem
ParseImpl cannot be resolved to a type    ExtParser.java
/nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext    line
138    Java Problem
ParseImpl cannot be resolved to a type    MSBaseParser.java
/nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms    line
108    Java Problem
ParseImpl cannot be resolved to a type    OOParser.java
/nutchbase/src/plugin/parse-oo/src/java/org/apache/nutch/parse/oo    line
103    Java Problem
ParseImpl cannot be resolved to a type    PdfParser.java
/nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf    line
155    Java Problem
ParseImpl cannot be resolved to a type    RSSParser.java
/nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss    line
187    Java Problem
ParseImpl cannot be resolved to a type    SWFParser.java
/nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf    line
113    Java Problem
ParseImpl cannot be resolved to a type    TestIndexingFilters.java
/nutchbase/src/test/org/apache/nutch/indexer    line 45    Java Problem
ParseImpl cannot be resolved to a type    TestMoreIndexingFilter.java
/nutchbase/src/plugin/index-more/src/test/org/apache/nutch/indexer/more
line 61    Java Problem
ParseImpl cannot be resolved to a type    TextParser.java
/nutchbase/src/plugin/parse-text/src/java/org/apache/nutch/parse/text
line 55    Java Problem
ParseImpl cannot be resolved to a type    ZipParser.java
/nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip    line
105    Java Problem
ParseResult cannot be resolved    ExtParser.java
/nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext    line
137    Java Problem
ParseResult cannot be resolved    MSBaseParser.java
/nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms    line
107    Java Problem
ParseResult cannot be resolved    OOParser.java
/nutchbase/src/plugin/parse-oo/src/java/org/apache/nutch/parse/oo    line
103    Java Problem
ParseResult cannot be resolved    PdfParser.java
/nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf    line
155    Java Problem
ParseResult cannot be resolved    RSSParser.java
/nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss    line
187    Java Problem
ParseResult cannot be resolved    SWFParser.java
/nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf    line
113    Java Problem
ParseResult cannot be resolved    TextParser.java
/nutchbase/src/plugin/parse-text/src/java/org/apache/nutch/parse/text
line 55    Java Problem
ParseResult cannot be resolved    ZipParser.java
/nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip    line
105    Java Problem
ParseResult cannot be resolved to a type    ArcSegmentCreator.java
/nutchbase/src/java/org/apache/nutch/tools/arc    line 159    Java Problem
ParseResult cannot be resolved to a type    CCParseFilter.java
/nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch
line 267    Java Problem
ParseResult cannot be resolved to a type    CCParseFilter.java
/nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch
line 267    Java Problem
ParseResult cannot be resolved to a type    ExtParser.java
/nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext    line
69    Java Problem
ParseResult cannot be resolved to a type    FeedParser.java
/nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed    line
106    Java Problem
ParseResult cannot be resolved to a type    FeedParser.java
/nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed    line
108    Java Problem
ParseResult cannot be resolved to a type    FeedParser.java
/nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed    line
108    Java Problem
ParseResult cannot be resolved to a type    FeedParser.java
/nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed    line
211    Java Problem
ParseResult cannot be resolved to a type    FeedParser.java
/nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed    line
221    Java Problem
ParseResult cannot be resolved to a type    HTMLLanguageParser.java
/nutchbase/src/plugin/languageidentifier/src/java/org/apache/nutch/analysis/lang
line 90    Java Problem
ParseResult cannot be resolved to a type    HTMLLanguageParser.java
/nutchbase/src/plugin/languageidentifier/src/java/org/apache/nutch/analysis/lang
line 90    Java Problem
ParseResult cannot be resolved to a type    MSBaseParser.java
/nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms    line
64    Java Problem
ParseResult cannot be resolved to a type    MSExcelParser.java
/nutchbase/src/plugin/parse-msexcel/src/java/org/apache/nutch/parse/msexcel
line 40    Java Problem
ParseResult cannot be resolved to a type    MSPowerPointParser.java
/nutchbase/src/plugin/parse-mspowerpoint/src/java/org/apache/nutch/parse/mspowerpoint
line 44    Java Problem
ParseResult cannot be resolved to a type    MSWordParser.java
/nutchbase/src/plugin/parse-msword/src/java/org/apache/nutch/parse/msword
line 43    Java Problem
ParseResult cannot be resolved to a type    OOParser.java
/nutchbase/src/plugin/parse-oo/src/java/org/apache/nutch/parse/oo    line
63    Java Problem
ParseResult cannot be resolved to a type    PdfParser.java
/nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf    line
69    Java Problem
ParseResult cannot be resolved to a type    RSSParser.java
/nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss    line
80    Java Problem
ParseResult cannot be resolved to a type    RelTagParser.java
/nutchbase/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag
line 68    Java Problem
ParseResult cannot be resolved to a type    RelTagParser.java
/nutchbase/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag
line 68    Java Problem
ParseResult cannot be resolved to a type    SWFParser.java
/nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf    line
64    Java Problem
ParseResult cannot be resolved to a type    SWFParser.java
/nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf    line
125    Java Problem
ParseResult cannot be resolved to a type    TestFeedParser.java
/nutchbase/src/plugin/feed/src/test/org/apache/nutch/parse/feed    line
94    Java Problem
ParseResult cannot be resolved to a type    TextParser.java
/nutchbase/src/plugin/parse-text/src/java/org/apache/nutch/parse/text
line 41    Java Problem
ParseResult cannot be resolved to a type    ZipParser.java
/nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip    line
55    Java Problem
The constructor Fetcher(Configuration) is undefined    TestFetcher.java
/nutchbase/src/test/org/apache/nutch/fetcher    line 100    Java Problem
The constructor Fetcher(Configuration) is undefined    TestFetcher.java
/nutchbase/src/test/org/apache/nutch/fetcher    line 177    Java Problem
The constructor Generator(Configuration) is undefined    TestFetcher.java
/nutchbase/src/test/org/apache/nutch/fetcher    line 94    Java Problem
The constructor Generator(Configuration) is undefined
TestGenerator.java    /nutchbase/src/test/org/apache/nutch/crawl    line
312    Java Problem
The constructor Injector(Configuration) is undefined    TestFetcher.java
/nutchbase/src/test/org/apache/nutch/fetcher    line 90    Java Problem
The constructor Injector(Configuration) is undefined    TestInjector.java
/nutchbase/src/test/org/apache/nutch/crawl    line 70    Java Problem
The constructor NutchWritable(ParseImpl) is undefined
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 229    Java Problem
The import org.apache.nutch.fetcher.FetcherOutputFormat cannot be
resolved    ArcSegmentCreator.java
/nutchbase/src/java/org/apache/nutch/tools/arc    line 44    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 50    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
BasicFields.java    /nutchbase/src/java/org/apache/nutch/indexer/field
line 61    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
ExtParser.java
/nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext    line
26    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
MSBaseParser.java
/nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms    line
39    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
PdfParser.java
/nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf    line
41    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
RSSParser.java
/nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss    line
41    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
TestExtParser.java
/nutchbase/src/plugin/parse-ext/src/test/org/apache/nutch/parse/ext    line
26    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
TestIndexingFilters.java    /nutchbase/src/test/org/apache/nutch/indexer
line 26    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
TestMSWordParser.java
/nutchbase/src/plugin/parse-msword/src/test/org/apache/nutch/parse/msword
line 26    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
TestMoreIndexingFilter.java
/nutchbase/src/plugin/index-more/src/test/org/apache/nutch/indexer/more
line 29    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
TestZipParser.java
/nutchbase/src/plugin/parse-zip/src/test/org/apache/nutch/parse/zip    line
26    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
ZipParser.java
/nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip    line
33    Java Problem
The import org.apache.nutch.parse.ParseImpl cannot be resolved
ZipTextExtractor.java
/nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip    line
41    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 51    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
ExtParser.java
/nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext    line
21    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
FeedParser.java
/nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed    line
43    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
HTMLLanguageParser.java
/nutchbase/src/plugin/languageidentifier/src/java/org/apache/nutch/analysis/lang
line 33    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
MSBaseParser.java
/nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms    line
40    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
MSExcelParser.java
/nutchbase/src/plugin/parse-msexcel/src/java/org/apache/nutch/parse/msexcel
line 20    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
MSPowerPointParser.java
/nutchbase/src/plugin/parse-mspowerpoint/src/java/org/apache/nutch/parse/mspowerpoint
line 20    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
MSWordParser.java
/nutchbase/src/plugin/parse-msword/src/java/org/apache/nutch/parse/msword
line 21    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
PdfParser.java
/nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf    line
37    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
RSSParser.java
/nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss    line
36    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
RelTagParser.java
/nutchbase/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag
line 38    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
TestFeedParser.java
/nutchbase/src/plugin/feed/src/test/org/apache/nutch/parse/feed    line
32    Java Problem
The import org.apache.nutch.parse.ParseResult cannot be resolved
ZipParser.java
/nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip    line
34    Java Problem
The method calculate(WebTableRow, Parse) in the type Signature is not
applicable for the arguments (Content, Parse)    ArcSegmentCreator.java
/nutchbase/src/java/org/apache/nutch/tools/arc    line 187    Java Problem
The method calculate(WebTableRow, Parse) in the type Signature is not
applicable for the arguments (Content, Parse)    ArcSegmentCreator.java
/nutchbase/src/java/org/apache/nutch/tools/arc    line 208    Java Problem
The method fetch(String, int, boolean) from the type Fetcher is not
visible    TestFetcher.java
/nutchbase/src/test/org/apache/nutch/fetcher    line 178    Java Problem
The method fetch(String, int, boolean) in the type Fetcher is not applicable
for the arguments (Path, int, boolean)    TestFetcher.java
/nutchbase/src/test/org/apache/nutch/fetcher    line 101    Java Problem
The method generate(String, long, long, boolean) in the type Generator is
not applicable for the arguments (Path, Path, int, int, long, boolean,
boolean)    TestGenerator.java
/nutchbase/src/test/org/apache/nutch/crawl    line 313    Java Problem
The method generate(String, long, long, boolean) in the type Generator is
not applicable for the arguments (Path, Path, int, long, long, boolean,
boolean)    TestFetcher.java
/nutchbase/src/test/org/apache/nutch/fetcher    line 95    Java Problem
The method getData() is undefined for the type Parse
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 200    Java Problem
The method getData() is undefined for the type Parse
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 211    Java Problem
The method getData() is undefined for the type Parse
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 213    Java Problem
The method getData() is undefined for the type Parse
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 216    Java Problem
The method getData() is undefined for the type Parse
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 230    Java Problem
The method getData() is undefined for the type Parse
ArcSegmentCreator.java    /nutchbase/src/java/org/apache/nutch/tools/arc
line 244    Java Problem
The method getData() is undefined for the type Parse    BasicFields.java
/nutchbase/src/java/org/apache/nutch/indexer/field    line 386    Java
Problem
The method getData() is undefined for the type Parse    BasicFields.java
/nutchbase/src/java/org/apache/nutch/indexer/field    line 395    Java
Problem
The method getData() is undefined for the type Parse
CCIndexingFilter.java
/nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch
line 55    Java Problem
The method getData() is undefined for the type Parse
CCParseFilter.java
/nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch
line 280    Java Problem
The method getData() is undefined for the type Parse
CCParseFilter.java
/nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch
line 286    Java Problem
The method getData() is undefined for the type Parse
CCParseFilter.java
/nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch
line 291    Java Problem
The method getData() is undefined for the type Parse
FeedIndexingFilter.java
/nutchbase/src/plugin/feed/src/java/org/apache/nutch/indexer/feed    line
76    Java Problem





[jira] Updated: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-05T14:25:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche updated NUTCH-767:
--------------------------------

    Attachment: NUTCH-767-part3.patch

the problems with the test comes from the fact that tika's detection of the mimetypes based
on content returns "text/plain"  when no mimetype can be identified, e.g. in our case because
we have an empty byte array as content.

Tika's MimeTypes used to have a default value which was used in MimeUtil to determine when
to use the type guessed by Tika but it has been removed since. The best course of action is
probably to take into account Tika's guess only if it is not  "text/plain" or "application/octet-stream",
which is what this patch implements.

The expected mime types in the test class are set to their original values (pre patch v2)
apart from the one which used Tika's default Mime Type.  

J.

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767-part2.patch, NUTCH-767-part3.patch, NUTCH-767.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Reopened: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-05T14:19:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche reopened NUTCH-767:
---------------------------------


the problem with the test class has been investigated. am reopening the issue so that we can
mark it as definitely fixed 

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767-part2.patch, NUTCH-767.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






Hudson build is back to normal: Nutch-trunk #1002
2009-12-05T04:54:23Z

See 







[jira] Commented: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-05T04:54:20Z


    [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786339#action_12786339
] 

Hudson commented on NUTCH-767:
------------------------------

Integrated in Nutch-trunk #1002 (See [http://hudson.zones.apache.org/hudson/job/Nutch-trunk/1002/])
     Fix a failing test - still needs more work.


> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767-part2.patch, NUTCH-767.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Closed: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-04T11:00:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrzej Bialecki  closed NUTCH-767.
-----------------------------------

    Resolution: Fixed

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767-part2.patch, NUTCH-767.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Updated: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-04T11:00:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrzej Bialecki  updated NUTCH-767:
------------------------------------

    Remaining Estimate: 0h
     Original Estimate: 0h

I applied the patch, and I'm closing this issue - we will track the test failures when we
upgrade to Tika 0.6, which is imminent.

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767-part2.patch, NUTCH-767.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






Build failed in Hudson: Nutch-trunk #1001
2009-12-04T04:03:44Z

See 

------------------------------------------
[...truncated 4696 lines...]
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 
    [mkdir] Created dir: 
     [copy] Copying 6 files to 

init:
    [mkdir] Created dir: 

init-plugin:

deps-jar:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

jar:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

compile-test:

compile:
     [echo] Compiling plugin: urlfilter-regex
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

jar:

deps-test:

deploy:

copy-generated-lib:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlfilter-suffix
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlfilter-validator
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-basic
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-pass
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 
    [mkdir] Created dir: 
     [copy] Copying 4 files to 

init:
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-regex
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

init:

init-plugin:

compile:

jar:
      [jar] Warning: skipping jar archive 
because no files were included.

deps-test:

deploy:

copy-generated-lib:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

compile:

job:
      [jar] Building jar: 

compile-core-test:
    [javac] Compiling 43 source files to 
    [javac] :102:
cannot find symbol
    [javac] symbol  : variable DEFAULT
    [javac] location: class org.apache.tika.mime.MimeTypes
    [javac]     assertEquals(MimeTypes.DEFAULT, c.getContentType());
    [javac]                           ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error

BUILD FAILED
:229: Compile
failed; see the compiler error output for details.

Total time: 1 minute 6 seconds
Archiving artifacts
Publishing Javadoc
Recording test results






[jira] Updated: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-03T10:42:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche updated NUTCH-767:
--------------------------------

    Attachment: NUTCH-767-part2.patch

Fixes compilation issues for test class src/test/org/apache/nutch/protocol/TestContent.java
and temporarily make sure that the expected mime-type values correspond to what is returned
by Tika. I will investigate why Tika does not return txt/html when the HTML document has no
text

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767-part2.patch, NUTCH-767.patch
>
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






Build failed in Hudson: Nutch-trunk #1000
2009-12-03T04:44:40Z

See 

------------------------------------------
[...truncated 4696 lines...]
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 
    [mkdir] Created dir: 
     [copy] Copying 6 files to 

init:
    [mkdir] Created dir: 

init-plugin:

deps-jar:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

jar:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

compile-test:

compile:
     [echo] Compiling plugin: urlfilter-regex
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

jar:

deps-test:

deploy:

copy-generated-lib:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlfilter-suffix
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlfilter-validator
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-basic
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-pass
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 
    [mkdir] Created dir: 
     [copy] Copying 4 files to 

init:
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-regex
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

init:

init-plugin:

compile:

jar:
      [jar] Warning: skipping jar archive 
because no files were included.

deps-test:

deploy:

copy-generated-lib:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

compile:

job:
      [jar] Building jar: 

compile-core-test:
    [javac] Compiling 43 source files to 
    [javac] :102:
cannot find symbol
    [javac] symbol  : variable DEFAULT
    [javac] location: class org.apache.tika.mime.MimeTypes
    [javac]     assertEquals(MimeTypes.DEFAULT, c.getContentType());
    [javac]                           ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error

BUILD FAILED
:229: Compile
failed; see the compiler error output for details.

Total time: 47 seconds
Archiving artifacts
Publishing Javadoc
Recording test results






[jira] Updated: (NUTCH-774) Retry interval in crawl date is set to 0
2009-12-02T12:12:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Reinhard Schwab updated NUTCH-774:
----------------------------------

    Attachment: NUTCH-774.patch

fixes also a minor typo in AbstractFetchSchedule.java

> Retry interval in crawl date is set to 0
> ----------------------------------------
>
>                 Key: NUTCH-774
>                 URL: https://issues.apache.org/jira/browse/NUTCH-774
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Reinhard Schwab
>             Fix For: 1.1
>
>         Attachments: NUTCH-774.patch
>
>
> When i fetch and parse a feed with the feed plugin,
> http://www.wachauclimbing.net/home/impressum-disclaimer/feed/
> another crawl date is generated
> http://www.wachauclimbing.net/home/impressum-disclaimer/comment-page-1/
> after fetching a second round
> the dump in the crawl db still shows a retry interval with value 0.
> http://www.wachauclimbing.net/home/impressum-disclaimer/comment-page-1/ Version: 7
> Status: 2 (db_fetched)
> Fetch time: Wed Dec 02 12:48:22 CET 2009
> Modified time: Thu Jan 01 01:00:00 CET 1970
> Retries since fetch: 0
> Retry interval: 0 seconds (0 days)
> Score: 1.0833334
> Signature: db9ab2193924cd2d0b53113a500ca604
> Metadata: _pst_: success(1), lastModified=0
> a check should be done in DefaultFetchSchedule (or AbstractFetchSchedule) in the
> method 
> setFetchSchedule

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Reopened: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-02T12:12:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrzej Bialecki  reopened NUTCH-767:
-------------------------------------


> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767.patch
>
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Commented: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-02T12:12:20Z


    [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784790#action_12784790
] 

Andrzej Bialecki  commented on NUTCH-767:
-----------------------------------------

Reopening this issue, because TestContent is failing now - after fixing a trivial compilation
problem, now the problem seems to be that the type for empty content is auto-detected as "text/plain"
and this value overrides the hint from the Content-Type header.

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767.patch
>
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Updated: (NUTCH-774) Retry interval in crawl date is set to 0
2009-12-02T12:12:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Reinhard Schwab updated NUTCH-774:
----------------------------------

    Patch Info: [Patch Available]

> Retry interval in crawl date is set to 0
> ----------------------------------------
>
>                 Key: NUTCH-774
>                 URL: https://issues.apache.org/jira/browse/NUTCH-774
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Reinhard Schwab
>             Fix For: 1.1
>
>         Attachments: NUTCH-774.patch
>
>
> When i fetch and parse a feed with the feed plugin,
> http://www.wachauclimbing.net/home/impressum-disclaimer/feed/
> another crawl date is generated
> http://www.wachauclimbing.net/home/impressum-disclaimer/comment-page-1/
> after fetching a second round
> the dump in the crawl db still shows a retry interval with value 0.
> http://www.wachauclimbing.net/home/impressum-disclaimer/comment-page-1/ Version: 7
> Status: 2 (db_fetched)
> Fetch time: Wed Dec 02 12:48:22 CET 2009
> Modified time: Thu Jan 01 01:00:00 CET 1970
> Retries since fetch: 0
> Retry interval: 0 seconds (0 days)
> Score: 1.0833334
> Signature: db9ab2193924cd2d0b53113a500ca604
> Metadata: _pst_: success(1), lastModified=0
> a check should be done in DefaultFetchSchedule (or AbstractFetchSchedule) in the
> method 
> setFetchSchedule

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Created: (NUTCH-774) Retry interval in crawl date is set to 0
2009-12-02T12:06:20Z

Retry interval in crawl date is set to 0
----------------------------------------

                 Key: NUTCH-774
                 URL: https://issues.apache.org/jira/browse/NUTCH-774
             Project: Nutch
          Issue Type: Bug
          Components: fetcher
    Affects Versions: 1.0.0
            Reporter: Reinhard Schwab
             Fix For: 1.1


When i fetch and parse a feed with the feed plugin,
http://www.wachauclimbing.net/home/impressum-disclaimer/feed/
another crawl date is generated
http://www.wachauclimbing.net/home/impressum-disclaimer/comment-page-1/

after fetching a second round
the dump in the crawl db still shows a retry interval with value 0.

http://www.wachauclimbing.net/home/impressum-disclaimer/comment-page-1/ Version: 7
Status: 2 (db_fetched)
Fetch time: Wed Dec 02 12:48:22 CET 2009
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 0 seconds (0 days)
Score: 1.0833334
Signature: db9ab2193924cd2d0b53113a500ca604
Metadata: _pst_: success(1), lastModified=0

a check should be done in DefaultFetchSchedule (or AbstractFetchSchedule) in the
method 
setFetchSchedule



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Commented: (NUTCH-666) Analysis plugins for multiple language and new Language Identifier Tool
2009-12-02T07:22:20Z


    [ https://issues.apache.org/jira/browse/NUTCH-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784680#action_12784680
] 

Raja Santosh Panda commented on NUTCH-666:
------------------------------------------

Hi,

I am looking forward to use only the language identifier (language-identifier.jar) plugin
for identification of chinese, japanese and korean languages.  

Can someone help me in this regard ?

Is this already implemented ? If yes, how can i take the dev version and use it ?

Can i use the language identifier of version 1.0 and train it (create N-Gram profiles) to
identify the above 3 languages ??

Any help is highly appreciated.

Regards
Raja

> Analysis plugins for multiple language and new Language Identifier Tool
> -----------------------------------------------------------------------
>
>                 Key: NUTCH-666
>                 URL: https://issues.apache.org/jira/browse/NUTCH-666
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.1
>         Environment: All
>            Reporter: Dennis Kubes
>            Assignee: Dennis Kubes
>             Fix For: 1.1
>
>         Attachments: NUTCH-666-1-20081126.patch
>
>
> Add analysis plugins for czech, greek, japanese, chinese, korean, dutch, russian, and
thai.  Also includes a new Language Identifier tool that used the new indexing framework in
NUTCH-646.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






Build failed in Hudson: Nutch-trunk #999
2009-12-02T04:04:16Z

See 

------------------------------------------
[...truncated 4696 lines...]
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 
    [mkdir] Created dir: 
     [copy] Copying 6 files to 

init:
    [mkdir] Created dir: 

init-plugin:

deps-jar:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

jar:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

compile-test:

compile:
     [echo] Compiling plugin: urlfilter-regex
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

init:

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: lib-regex-filter

jar:

deps-test:

deploy:

copy-generated-lib:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlfilter-suffix
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlfilter-validator
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-basic
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

init:
    [mkdir] Created dir: 
    [mkdir] Created dir: 
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-pass
    [javac] Compiling 1 source file to 

jar:
      [jar] Building jar: 

deps-test:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 
    [mkdir] Created dir: 
     [copy] Copying 4 files to 

init:
    [mkdir] Created dir: 

init-plugin:

deps-jar:

compile:
     [echo] Compiling plugin: urlnormalizer-regex
    [javac] Compiling 1 source file to 
    [javac] Note: 
uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

jar:
      [jar] Building jar: 

deps-test:

init:

init-plugin:

compile:

jar:
      [jar] Warning: skipping jar archive 
because no files were included.

deps-test:

deploy:

copy-generated-lib:

deploy:
    [mkdir] Created dir: 
     [copy] Copying 1 file to 

copy-generated-lib:
     [copy] Copying 1 file to 

compile:

job:
      [jar] Building jar: 

compile-core-test:
    [javac] Compiling 43 source files to 
    [javac] :102:
cannot find symbol
    [javac] symbol  : variable DEFAULT
    [javac] location: class org.apache.tika.mime.MimeTypes
    [javac]     assertEquals(MimeTypes.DEFAULT, c.getContentType());
    [javac]                           ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error

BUILD FAILED
:229: Compile
failed; see the compiler error output for details.

Total time: 1 minute 5 seconds
Archiving artifacts
Publishing Javadoc
Recording test results






[jira] Commented: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-01T19:03:21Z


    [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784337#action_12784337
] 

Andrzej Bialecki  commented on NUTCH-767:
-----------------------------------------

Fixed in rev. 885869. Thank you!

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767.patch
>
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Closed: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-01T19:03:21Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrzej Bialecki  closed NUTCH-767.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 1.1
         Assignee: Andrzej Bialecki   (was: Chris A. Mattmann)

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: NUTCH-767.patch
>
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Updated: (NUTCH-767) Update Tika to v0.5  for the MimeType detection
2009-12-01T16:16:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche updated NUTCH-767:
--------------------------------

    Description: 
The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is now
split in several jars, we need to place the tika-core.jar in the main nutch lib.


  was:
The version 5 of TIka requires a few changes to the MimeType implementation. Tika is now split
in several jars, we need to place the tika-core.jar in the main nutch lib.


        Summary: Update Tika to v0.5  for the MimeType detection  (was: Update Tika to v5.0
 for the MimeType detection)

> Update Tika to v0.5  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Chris A. Mattmann
>         Attachments: NUTCH-767.patch
>
>
> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Updated: (NUTCH-767) Update Tika to v5.0  for the MimeType detection
2009-12-01T16:12:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche updated NUTCH-767:
--------------------------------

    Description: 
The version 5 of TIka requires a few changes to the MimeType implementation. Tika is now split
in several jars, we need to place the tika-core.jar in the main nutch lib.


  was:
The latest version of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.


        Summary: Update Tika to v5.0  for the MimeType detection  (was: Update version of
Tika for the MimeType detection)

> Update Tika to v5.0  for the MimeType detection
> -----------------------------------------------
>
>                 Key: NUTCH-767
>                 URL: https://issues.apache.org/jira/browse/NUTCH-767
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Chris A. Mattmann
>         Attachments: NUTCH-767.patch
>
>
> The version 5 of TIka requires a few changes to the MimeType implementation. Tika is
now split in several jars, we need to place the tika-core.jar in the main nutch lib.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Closed: (NUTCH-769) Fetcher to skip queues for URLS getting repeated exceptions
2009-12-01T15:16:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrzej Bialecki  closed NUTCH-769.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 1.1
         Assignee: Andrzej Bialecki 

> Fetcher to skip queues for URLS getting repeated exceptions  
> -------------------------------------------------------------
>
>                 Key: NUTCH-769
>                 URL: https://issues.apache.org/jira/browse/NUTCH-769
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: NUTCH-769-2.patch, NUTCH-769.patch
>
>
> As discussed on the mailing list (see http://www.mail-archive.com/nutch-user@lucene.apache.org/msg15360.html)
this patch allows to clear URLs queues in the Fetcher when more than a set number of exceptions
have been encountered in a row. This can speed up the fetching substantially in cases where
target hosts are not responsive (as a TimeoutException would be thrown) and limits cases where
a whole Fetch step is slowed down because of a few queues.
> by default the parameter fetcher.max.exceptions.per.queue has a value of -1 and is deactivated.
 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Commented: (NUTCH-769) Fetcher to skip queues for URLS getting repeated exceptions
2009-12-01T15:16:20Z


    [ https://issues.apache.org/jira/browse/NUTCH-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784260#action_12784260
] 

Andrzej Bialecki  commented on NUTCH-769:
-----------------------------------------

I had to apply this patch by hand, due to NUTCH-770. I also added conf/nutch-default.xml documentation.
This was committed in rev. 885785 - thanks!

> Fetcher to skip queues for URLS getting repeated exceptions  
> -------------------------------------------------------------
>
>                 Key: NUTCH-769
>                 URL: https://issues.apache.org/jira/browse/NUTCH-769
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: NUTCH-769-2.patch, NUTCH-769.patch
>
>
> As discussed on the mailing list (see http://www.mail-archive.com/nutch-user@lucene.apache.org/msg15360.html)
this patch allows to clear URLs queues in the Fetcher when more than a set number of exceptions
have been encountered in a row. This can speed up the fetching substantially in cases where
target hosts are not responsive (as a TimeoutException would be thrown) and limits cases where
a whole Fetch step is slowed down because of a few queues.
> by default the parameter fetcher.max.exceptions.per.queue has a value of -1 and is deactivated.
 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Closed: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20
2009-12-01T14:59:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dennis Kubes closed NUTCH-768.
------------------------------

    Resolution: Fixed

Weird.  The hsqldb License file was the same checksum as that pulled from hadoop.  It must
have had the windows EOL in hadoop distribution as well.  I changed it anyways.  Everything
committed with revision 885778.

> Upgrade Nutch 1.0 to use Hadoop 0.20
> ------------------------------------
>
>                 Key: NUTCH-768
>                 URL: https://issues.apache.org/jira/browse/NUTCH-768
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.1
>         Environment: All
>            Reporter: Dennis Kubes
>            Assignee: Dennis Kubes
>             Fix For: 1.1
>
>         Attachments: NUTCH-768-1-20091125.patch
>
>
> Upgrade Nutch 1.0 to use the Hadoop 0.20 release.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Commented: (NUTCH-770) Timebomb for Fetcher
2009-12-01T14:51:20Z


    [ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784250#action_12784250
] 

Andrzej Bialecki  commented on NUTCH-770:
-----------------------------------------

Fixed in rev. 885776. Thank you!

> Timebomb for Fetcher
> --------------------
>
>                 Key: NUTCH-770
>                 URL: https://issues.apache.org/jira/browse/NUTCH-770
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch
>
>
> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is
not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes
is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder
skips all remaining entries then all active queues are purged. This allows to keep the Fetch
step under comtrol and works well in combination with NUTCH-769

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Closed: (NUTCH-770) Timebomb for Fetcher
2009-12-01T14:51:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrzej Bialecki  closed NUTCH-770.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 1.1
         Assignee: Andrzej Bialecki 

> Timebomb for Fetcher
> --------------------
>
>                 Key: NUTCH-770
>                 URL: https://issues.apache.org/jira/browse/NUTCH-770
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>
>         Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch
>
>
> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is
not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes
is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder
skips all remaining entries then all active queues are purged. This allows to keep the Fetch
step under comtrol and works well in combination with NUTCH-769

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Commented: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20
2009-12-01T12:46:20Z


    [ https://issues.apache.org/jira/browse/NUTCH-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784206#action_12784206
] 

Andrzej Bialecki  commented on NUTCH-768:
-----------------------------------------

+1.

Minor nit: file lib/hsqldb-1.8.0.10.LICENSE.txt uses Windows EOL style, this should be probably
corrected before commit.

> Upgrade Nutch 1.0 to use Hadoop 0.20
> ------------------------------------
>
>                 Key: NUTCH-768
>                 URL: https://issues.apache.org/jira/browse/NUTCH-768
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.1
>         Environment: All
>            Reporter: Dennis Kubes
>            Assignee: Dennis Kubes
>             Fix For: 1.1
>
>         Attachments: NUTCH-768-1-20091125.patch
>
>
> Upgrade Nutch 1.0 to use the Hadoop 0.20 release.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Commented: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20
2009-12-01T05:28:20Z


    [ https://issues.apache.org/jira/browse/NUTCH-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784066#action_12784066
] 

Dennis Kubes commented on NUTCH-768:
------------------------------------

If no objections I will commit this tomorrow sometime?

> Upgrade Nutch 1.0 to use Hadoop 0.20
> ------------------------------------
>
>                 Key: NUTCH-768
>                 URL: https://issues.apache.org/jira/browse/NUTCH-768
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.1
>         Environment: All
>            Reporter: Dennis Kubes
>            Assignee: Dennis Kubes
>             Fix For: 1.1
>
>         Attachments: NUTCH-768-1-20091125.patch
>
>
> Upgrade Nutch 1.0 to use the Hadoop 0.20 release.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






Build failed in Hudson: Nutch-trunk #998
2009-12-01T04:02:44Z

See 

------------------------------------------
A timer trigger started this job
Building remotely on lucene.zones.apache.org (Solaris 10)
Checking out http://svn.apache.org/repos/asf/lucene/nutch/trunk
ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/nutch/trunk
org.tmatesoft.svn.core.SVNException: svn: timed out waiting for server
svn: OPTIONS request failed on '/repos/asf/lucene/nutch/trunk'
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:103)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:87)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:616)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:273)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:261)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:516)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:98)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1001)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getLatestRevision(DAVRepository.java:178)
	at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:482)
	at org.tmatesoft.svn.core.wc.SVNBasicClient.getLocations(SVNBasicClient.java:851)
	at org.tmatesoft.svn.core.wc.SVNBasicClient.createRepository(SVNBasicClient.java:534)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:893)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:791)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:617)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:543)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2052)
	at hudson.remoting.UserRequest.perform(UserRequest.java:69)
	at hudson.remoting.UserRequest.perform(UserRequest.java:23)
	at hudson.remoting.Request$2.run(Request.java:200)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:519)
	at org.tmatesoft.svn.core.internal.util.SVNSocketConnection.run(SVNSocketConnection.java:57)
	... 1 more
Archiving artifacts
Publishing Javadoc
Recording test results






Re: wrong wiki front page
2009-11-30T16:22:00Z

Thanks. There is already a JIRA issue :
https://issues.apache.org/jira/browse/INFRA-2251 but it is pending since
last month !

2009/11/30 Andrzej Bialecki 

> Alban Mouton wrote:
>
>> No reaction ? Isn't the Wiki admin on this mailing list ? I don't see any
>> link on the Wiki to contact the admin.
>>
>> The french frontpage is still the generic MoinMoin wiki home page and that
>> can make a bad impression to newcomers !
>>
>
> We have little control over the MoinMoin config (AFAIK it's configured for
> multiple projects), and what you noticed is probably a fallout of the recent
> wiki upgrade - please create a JIRA issue here:
> https://issues.apache.org/jira/browse/INFRA (don't forget to mention the
> project name).
>
>
> --
> Best regards,
> Andrzej Bialecki     <><
>  ___. ___ ___ ___ _ _   __________________________________
> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
> ___|||__||  \|  ||  |  Embedded Unix, System Integration
> http://www.sigram.com  Contact: info at sigram dot com
>
>





Re: wrong wiki front page
2009-11-30T15:57:16Z

Alban Mouton wrote:
> No reaction ? Isn't the Wiki admin on this mailing list ? I don't see 
> any link on the Wiki to contact the admin.
> 
> The french frontpage is still the generic MoinMoin wiki home page and 
> that can make a bad impression to newcomers !

We have little control over the MoinMoin config (AFAIK it's configured 
for multiple projects), and what you noticed is probably a fallout of 
the recent wiki upgrade - please create a JIRA issue here: 
https://issues.apache.org/jira/browse/INFRA (don't forget to mention the 
project name).


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com






Re: wrong wiki front page
2009-11-30T15:36:42Z

No reaction ? Isn't the Wiki admin on this mailing list ? I don't see any
link on the Wiki to contact the admin.

The french frontpage is still the generic MoinMoin wiki home page and that
can make a bad impression to newcomers !

2009/11/26 Alban Mouton 

> Issue and solutions described here :http://wiki.apache.org/httpd/HelpOnConfiguration#Default_front_page
>
> 2009/11/24 Alban Mouton 
>
>> Hello everybody,
>>
>>
>> I don't know if it is a known issue, but it's been like that since at
>> least a couple of days so I figured I should tell someone. The root url for
>> the nutch wiki http://wiki.apache.org/nutch/ doesn't redirect to
>> http://wiki.apache.org/nutch/FrontPage ! It's annoying because that's the
>> url given by google and the nutch website. It might be a language detection
>> problem because I see this ugly and not very helpful page :
>> http://wiki.apache.org/nutch/PageD%27Accueil (page d'accueil = home page
>> in french).
>>
>> Not much of a contribution for my first message here, but I hope to do
>> more soon.
>>
>> Alban Mouton
>>
>
>





[jira] Updated: (NUTCH-770) Timebomb for Fetcher
2009-11-30T14:31:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche updated NUTCH-770:
--------------------------------

    Attachment: NUTCH-770-v3.patch

the v2 applied the Lucene code formatting to the whole java file which caused far too many
changes, the v3 does the same as the v2 (add param and description to nutch default + change
timebomb to timelimit) but applies the code formatting only to the relevant portions of code

> Timebomb for Fetcher
> --------------------
>
>                 Key: NUTCH-770
>                 URL: https://issues.apache.org/jira/browse/NUTCH-770
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>         Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch
>
>
> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is
not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes
is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder
skips all remaining entries then all active queues are purged. This allows to keep the Fetch
step under comtrol and works well in combination with NUTCH-769

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Updated: (NUTCH-770) Timebomb for Fetcher
2009-11-30T13:39:20Z


     [ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche updated NUTCH-770:
--------------------------------

    Attachment: NUTCH-770-v2.patch

* renamed timebomb into timelimit
* added parameter and its description in nutch-default.xml
* applied Lucene codestyle from http://wiki.apache.org/lucene-java/HowToContribute?action=AttachFile&do=view&target=Eclipse-Lucene-Codestyle.xml

> Timebomb for Fetcher
> --------------------
>
>                 Key: NUTCH-770
>                 URL: https://issues.apache.org/jira/browse/NUTCH-770
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>         Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770.patch
>
>
> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is
not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes
is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder
skips all remaining entries then all active queues are purged. This allows to keep the Fetch
step under comtrol and works well in combination with NUTCH-769

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.






[jira] Commented: (NUTCH-770) Timebomb for Fetcher
2009-11-30T12:59:22Z


    [ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783638#action_12783638
] 

Andrzej Bialecki  commented on NUTCH-770:
-----------------------------------------

bq.   "time limit" is definitely better than timebomb (but not as amusing). 

:) let's got for "informative" and "less confusing" now ... Could you please also add the
nutch-default.xml property and its documentation.

Re: FetchQueues - ok, you have a point here.

Re: code style - yes.

> Timebomb for Fetcher
> --------------------
>
>                 Key: NUTCH-770
>                 URL: https://issues.apache.org/jira/browse/NUTCH-770
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>         Attachments: log-770, NUTCH-770.patch
>
>
> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is
not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes
is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder
skips all remaining entries then all active queues are purged. This allows to keep the Fetch
step under comtrol and works well in combination with NUTCH-769

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
nutch-dev@lucene.apache.org Archives

java.net.URL synchronization

Re: State of nutchbase

Re: State of nutchbase

[jira] Commented: (NUTCH-770) Timebomb for Fetcher

[jira] Issue Comment Edited: (NUTCH-770) Timebomb for Fetcher

State of nutchbase

[jira] Updated: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Reopened: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

Hudson build is back to normal: Nutch-trunk #1002

[jira] Commented: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Closed: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Updated: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

Build failed in Hudson: Nutch-trunk #1001

[jira] Updated: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

Build failed in Hudson: Nutch-trunk #1000

[jira] Updated: (NUTCH-774) Retry interval in crawl date is set to 0

[jira] Reopened: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Commented: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Updated: (NUTCH-774) Retry interval in crawl date is set to 0

[jira] Created: (NUTCH-774) Retry interval in crawl date is set to 0

[jira] Commented: (NUTCH-666) Analysis plugins for multiple language and new Language Identifier Tool

Build failed in Hudson: Nutch-trunk #999

[jira] Commented: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Closed: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Updated: (NUTCH-767) Update Tika to v0.5 for the MimeType detection

[jira] Updated: (NUTCH-767) Update Tika to v5.0 for the MimeType detection

[jira] Closed: (NUTCH-769) Fetcher to skip queues for URLS getting repeated exceptions

[jira] Commented: (NUTCH-769) Fetcher to skip queues for URLS getting repeated exceptions

[jira] Closed: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20

[jira] Commented: (NUTCH-770) Timebomb for Fetcher

[jira] Closed: (NUTCH-770) Timebomb for Fetcher

[jira] Commented: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20

[jira] Commented: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20

Build failed in Hudson: Nutch-trunk #998

Re: wrong wiki front page

Re: wrong wiki front page

Re: wrong wiki front page

[jira] Updated: (NUTCH-770) Timebomb for Fetcher

[jira] Updated: (NUTCH-770) Timebomb for Fetcher

[jira] Commented: (NUTCH-770) Timebomb for Fetcher
