user@nutch.apache.org Archives

Re: Nutch 2.1: extension point ParseFilter: doc is null

2013-05-24T10:00:55Z

Dear Lewis,

Thanks for the hints. Yes I am using currently neko parser and did some
debugging. Seems, that issue is still open for nutch-2.1:
https://issues.apache.org/jira/browse/NUTCH-1253
(See stacktrace further below)
I had however no success to change to neko 1.9.6.2 (since it is already
in maven repo) and did not try out yet the solution suggested in the
issue (neko 1.9.12).

With tagsoup, reltag plugin works smoothly.

Best regards,
Martin





java.util.concurrent.ExecutionException: java.lang.AbstractMethodError:
org.cyberneko.html.HTMLScanner.getCharacterOffset()I
        at java.util.concurrent.FutureTask
$Sync.innerGet(FutureTask.java:262)
        at java.util.concurrent.FutureTask.get(FutureTask.java:119)
        at
org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:148)
        at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:129)
        at org.apache.nutch.parse.ParseUtil.process(ParseUtil.java:176)
        at org.apache.nutch.parse.ParserJob
$ParserMapper.map(ParserJob.java:129)
        at org.apache.nutch.parse.ParserJob
$ParserMapper.map(ParserJob.java:78)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.LocalJobRunner
$Job.run(LocalJobRunner.java:212)
Caused by: java.lang.AbstractMethodError:
org.cyberneko.html.HTMLScanner.getCharacterOffset()I
        at org.apache.xerces.xni.parser.XMLParseException.(Unknown
Source)
        at org.cyberneko.html.HTMLConfiguration
$ErrorReporter.createException(HTMLConfiguration.java:673)
        at org.cyberneko.html.HTMLConfiguration
$ErrorReporter.reportError(HTMLConfiguration.java:662)
        at org.cyberneko.html.HTMLScanner
$ContentScanner.scanAttribute(HTMLScanner.java:2468)
        at org.cyberneko.html.HTMLScanner
$ContentScanner.scanAttribute(HTMLScanner.java:2424)
        at org.cyberneko.html.HTMLScanner
$ContentScanner.scanStartElement(HTMLScanner.java:2328)
        at org.cyberneko.html.HTMLScanner
$ContentScanner.scan(HTMLScanner.java:1881)
        at
org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:809)
        at
org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:478)
        at
org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:431)
        at
org.cyberneko.html.parsers.DOMFragmentParser.parse(DOMFragmentParser.java:164)
        at
org.apache.nutch.parse.html.HtmlParser.parseNeko(HtmlParser.java:275)
        at
org.apache.nutch.parse.html.HtmlParser.parse(HtmlParser.java:238)
        at
org.apache.nutch.parse.html.HtmlParser.getParse(HtmlParser.java:173)
        at
org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:36)
        at
org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:23)
        at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)









AbstractMethodError
On Thu, 2013-05-23 at 21:16 -0700, Lewis John Mcgibbney wrote:
> Hi Martin,
> I am struggling to understand how the DocumentFragment (populated either by
> private methods parseTagSoup or parseNeko depending on your config in
> nutch-site.xml) is null!
> What you don't mention is some problem you are having?
> I can't DEBUG the code tonight but I am interested to see what is up here.
> Lewis
> 
> On Thursday, May 23, 2013, Martin Aesch  wrote:
> > Dear nutchers,
> >
> > I extended the ParseFilter extension point
> >
> > public Parse filter(String url, WebPage page, Parse parse,
> >     HTMLMetaTags metaTags, DocumentFragment doc) {
> >
> > From what I understand, plugin parse-html should populate the
> > DocumentFragment doc.
> >
> > Unfortunately, doc is always null. I tried this with my own plugin, as
> > well as with the nutch-shipped plugin microformats-reltag, which extends
> > the same point.
> >
> > Both plugins are working, and they are called. I attached my debugger,
> > and both for my own plugin as well as for the reltag-plugin, doc is
> > always null.
> >
> > I checked parse-plugins.xml, yes, parse-html is called and my mime-types
> > are those which call parse-html
> > (extension-id="org.apache.nutch.parse.html.HtmlParser").
> >
> > What am I missing?
> >
> > Thanks,
> > Martin
> >
> >
>

Re: Nutch 2.1: extension point ParseFilter: doc is null

2013-05-24T04:16:10Z

Hi Martin,
I am struggling to understand how the DocumentFragment (populated either by
private methods parseTagSoup or parseNeko depending on your config in
nutch-site.xml) is null!
What you don't mention is some problem you are having?
I can't DEBUG the code tonight but I am interested to see what is up here.
Lewis

On Thursday, May 23, 2013, Martin Aesch  wrote:
> Dear nutchers,
>
> I extended the ParseFilter extension point
>
> public Parse filter(String url, WebPage page, Parse parse,
>     HTMLMetaTags metaTags, DocumentFragment doc) {
>
> From what I understand, plugin parse-html should populate the
> DocumentFragment doc.
>
> Unfortunately, doc is always null. I tried this with my own plugin, as
> well as with the nutch-shipped plugin microformats-reltag, which extends
> the same point.
>
> Both plugins are working, and they are called. I attached my debugger,
> and both for my own plugin as well as for the reltag-plugin, doc is
> always null.
>
> I checked parse-plugins.xml, yes, parse-html is called and my mime-types
> are those which call parse-html
> (extension-id="org.apache.nutch.parse.html.HtmlParser").
>
> What am I missing?
>
> Thanks,
> Martin
>
>

-- 
*Lewis*

Re: Explanation of RegexURLFIlterTestBase benchmark's

2013-05-24T01:51:14Z

Hi Kirby,

On Thu, May 23, 2013 at 6:36 PM, Kirby Bohling wrote:

>
> Not that I think you need them in particular, but it seems like Nutch could
> be doing plenty of benchmarking, and micro benchmarking in particular.
>

I agree with this. It is not my goal to attack this head on but (I think)
it is useful for us to know more about the different components of Nutch
and how they operate, micro benchmarking would certainly be a way of making
this realistic.
This being said, I am quite keen on the idea of third party libraries (such
as bk.brics automaton [0]) being tested in thier own environment, by their
own development team. In this case, some comparative *results* (of an older
bk.brics library) can be seen here [1].
Anyone is free to infer from this what they wish, but it gives a bit of an
idea about the gains which can be achieved.
If regex p is something which you (I mean this collectively to refer to
anyone) think is a bottle neck for your Nutch deployment. Try out the
automaton plugin and hopefully things get better for you. AFAIK we use the
most up-to-date library available here so things should work well.

Thanks for the post Kirby.

[0] http://www.brics.dk/automaton/index.html
[1] http://tusker.org/regex/regex_benchmark.html

Re: Explanation of RegexURLFIlterTestBase benchmark's

2013-05-24T01:36:43Z

Re-reading my e-mail, I realize it might be read poorly.  Thanks for giving
me the benefit of the doubt.

There's a bunch of good material out on the web, and ultimately, the truth
is that micro benchmarks can always be misleading, and the only accurate
benchmark is real workload testing.  That said, micros benchmarks can be
accurate and useful in limit contexts.

There are several good resources if you want to do micro benchmarking w/
Java:
https://code.google.com/p/caliper/  (Full Disclosure: Written by my
employer.  I've never used it, but the theory/docs are sound)

Peter Lawrey has a good blog that touches on issues of performance and has
a couple of posts explicitly on mistakes he's made in extremely high
performance benchmarking:
http://mechanical-sympathy.blogspot.com/

Pretty decent explanations here:
http://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java

Not that I think you need them in particular, but it seems like Nutch could
be doing plenty of benchmarking, and micro benchmarking in particular.
 Knowing the pitfalls is valuable.  Lots of very smart people screw this up
regularly and make poorly founded decisions armed with faulty data.  Not
always sure I qualify as smart, but I've done it more than once.

Anyways, those reference have plenty of gory details for folks who are
interested in why things like this happen.

Kirby

On Thu, May 23, 2013 at 6:48 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> You know, this was my suspicion Kirby.
> Thanks for giving the heads up... automaton rocks.
> Lewis
>
>
> On Thu, May 23, 2013 at 5:06 PM, Kirby Bohling  >wrote:
>
> > Standard micro-benchmark issues with Java, run the 50 last and it'll run
> > faster.  JVM warmup, and JIT compilation, yadda, yadda, yadda.
> >
> >
> > On Thu, May 23, 2013 at 1:57 PM, Lewis John Mcgibbney <
> > lewis.mcgibbney@gmail.com> wrote:
> >
> > > Hi All,
> > > A really nice aspect of the regex (urlfilter-automaton and
> > urfilter-regex)
> > > plugin implementation's in Nutch is that there is a small but very
> useful
> > > RegexURLFilterBaseTest [0] which compares benchmarks for simple regex
> > > parsing.
> > > The results we get are as follows
> > >
> > > urls      automaton      regex
> > > 50        343ms           210ms
> > > 100      48ms             187ms
> > > 200      65ms             363ms
> > > 400      100ms           692ms
> > > 800      165ms           1385ms
> > >
> > > The problem I have here is understanding why the first (50) bench
> appears
> > > to be more expensive for both implementations?
> > > Additionally, why does this same bench cost much more for automaton?
> > >
> > > Anyone have a clue?
> > > Thanks
> > > Lewis
> > >
> > > [0]
> > >
> > >
> >
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java?view=markup
> > >
> > > --
> > > *Lewis*
> > >
> >
>
>
>
> --
> *Lewis*
>

Re: Explanation of RegexURLFIlterTestBase benchmark's

2013-05-24T00:58:40Z

Just ran the tests twice (to be clear: invoked bench() twice in same run)
to see the timings for regex-urlfilter:

(inputs) time
*(50) 231ms*
(100) 169ms
(200) 326ms
(400) 683ms
(800) 1420ms
*(50) 109ms*
(100) 188ms
(200) 319ms
(400) 714ms
(800) 1442ms

Kirby is right.


On Thu, May 23, 2013 at 5:48 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> You know, this was my suspicion Kirby.
> Thanks for giving the heads up... automaton rocks.
> Lewis
>
>
> On Thu, May 23, 2013 at 5:06 PM, Kirby Bohling  >wrote:
>
> > Standard micro-benchmark issues with Java, run the 50 last and it'll run
> > faster.  JVM warmup, and JIT compilation, yadda, yadda, yadda.
> >
> >
> > On Thu, May 23, 2013 at 1:57 PM, Lewis John Mcgibbney <
> > lewis.mcgibbney@gmail.com> wrote:
> >
> > > Hi All,
> > > A really nice aspect of the regex (urlfilter-automaton and
> > urfilter-regex)
> > > plugin implementation's in Nutch is that there is a small but very
> useful
> > > RegexURLFilterBaseTest [0] which compares benchmarks for simple regex
> > > parsing.
> > > The results we get are as follows
> > >
> > > urls      automaton      regex
> > > 50        343ms           210ms
> > > 100      48ms             187ms
> > > 200      65ms             363ms
> > > 400      100ms           692ms
> > > 800      165ms           1385ms
> > >
> > > The problem I have here is understanding why the first (50) bench
> appears
> > > to be more expensive for both implementations?
> > > Additionally, why does this same bench cost much more for automaton?
> > >
> > > Anyone have a clue?
> > > Thanks
> > > Lewis
> > >
> > > [0]
> > >
> > >
> >
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java?view=markup
> > >
> > > --
> > > *Lewis*
> > >
> >
>
>
>
> --
> *Lewis*
>

Re: Explanation of RegexURLFIlterTestBase benchmark's

2013-05-24T00:48:27Z

You know, this was my suspicion Kirby.
Thanks for giving the heads up... automaton rocks.
Lewis


On Thu, May 23, 2013 at 5:06 PM, Kirby Bohling wrote:

> Standard micro-benchmark issues with Java, run the 50 last and it'll run
> faster.  JVM warmup, and JIT compilation, yadda, yadda, yadda.
>
>
> On Thu, May 23, 2013 at 1:57 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi All,
> > A really nice aspect of the regex (urlfilter-automaton and
> urfilter-regex)
> > plugin implementation's in Nutch is that there is a small but very useful
> > RegexURLFilterBaseTest [0] which compares benchmarks for simple regex
> > parsing.
> > The results we get are as follows
> >
> > urls      automaton      regex
> > 50        343ms           210ms
> > 100      48ms             187ms
> > 200      65ms             363ms
> > 400      100ms           692ms
> > 800      165ms           1385ms
> >
> > The problem I have here is understanding why the first (50) bench appears
> > to be more expensive for both implementations?
> > Additionally, why does this same bench cost much more for automaton?
> >
> > Anyone have a clue?
> > Thanks
> > Lewis
> >
> > [0]
> >
> >
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java?view=markup
> >
> > --
> > *Lewis*
> >
>



-- 
*Lewis*

Re: Explanation of RegexURLFIlterTestBase benchmark's

2013-05-24T00:06:50Z

Standard micro-benchmark issues with Java, run the 50 last and it'll run
faster.  JVM warmup, and JIT compilation, yadda, yadda, yadda.


On Thu, May 23, 2013 at 1:57 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Hi All,
> A really nice aspect of the regex (urlfilter-automaton and urfilter-regex)
> plugin implementation's in Nutch is that there is a small but very useful
> RegexURLFilterBaseTest [0] which compares benchmarks for simple regex
> parsing.
> The results we get are as follows
>
> urls      automaton      regex
> 50        343ms           210ms
> 100      48ms             187ms
> 200      65ms             363ms
> 400      100ms           692ms
> 800      165ms           1385ms
>
> The problem I have here is understanding why the first (50) bench appears
> to be more expensive for both implementations?
> Additionally, why does this same bench cost much more for automaton?
>
> Anyone have a clue?
> Thanks
> Lewis
>
> [0]
>
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java?view=markup
>
> --
> *Lewis*
>

Nutch 2.1: extension point ParseFilter: doc is null

2013-05-23T21:28:40Z

Dear nutchers,

I extended the ParseFilter extension point

public Parse filter(String url, WebPage page, Parse parse,
    HTMLMetaTags metaTags, DocumentFragment doc) {

>From what I understand, plugin parse-html should populate the
DocumentFragment doc.

Unfortunately, doc is always null. I tried this with my own plugin, as
well as with the nutch-shipped plugin microformats-reltag, which extends
the same point.

Both plugins are working, and they are called. I attached my debugger,
and both for my own plugin as well as for the reltag-plugin, doc is
always null. 

I checked parse-plugins.xml, yes, parse-html is called and my mime-types
are those which call parse-html
(extension-id="org.apache.nutch.parse.html.HtmlParser").

What am I missing?

Thanks,
Martin

Re: Nutch 2.1 pdf parsing

2013-05-23T20:31:48Z

Hi Lewis,

thank you very much. I will try your solution.


2013/5/23 Lewis John Mcgibbney 

> Hi Adriana,
> If I were you I would switch your logging to DEBUG for the ParserJob
>
> - log4j.logger.org.apache.nutch.parse.ParserJob=INFO,cmdstdout
> + log4j.logger.org.apache.nutch.parse.ParserJob=DEBUG,cmdstdout
>
>
> recompile the code, then look closely at the parse chunk of the log to see
> what parser is being used, and if there are any particular issues flagged
> up @runtime.
>
>
> On Thu, May 23, 2013 at 8:14 AM, Adriana Farina
> wrote:
>
> > Hi,
> >
> > I'm using Nutch 2.1 in distributed mode on top of Hadoop 1.0.4, with
> HBase
> > 0.90.4 as database.
> >
> > I wrote a Java class from which I run the crawling cycle, the code that
> > implements the crawling cycle is the following:
> >
> >                   for (int i = 0; i < depth; i++) {
> > batchid = generator.generate((Long) args.get(Nutch.ARG_TOPN),
> > System.currentTimeMillis(), false, false);
> > fetcher.fetch(batchid, 1, false, -1);
> > parser.parse(batchid, false, true);
> > updater.run(new String[0]);
> >   }
> >
> > The problem is that I'm not able to parse the pdf files, inside HBase I
> got
> > no pdf content. The strange thing is that I got one row with the
> following
> > content: column=p:parsestat, timestamp=1369316742871,
> > value=\x04\x90\x03\x02\x96\x01org.apache.nutch.parse.ParseException:
> Unable
> > to successfully parse content\x00.
> >
> > It seems to me that I have configured all nutch property files correctly.
> > Can anybody help me?
> >
> > Thank you very much.
> >
> >
> > --
> > Adriana Farina
> >
>
>
>
> --
> *Lewis*
>



-- 
Adriana Farina

Re: error crawling

2013-05-23T20:16:15Z

I do not think that script works in nutch-2.x.
For example I see this
$bin/nutch generate $commonOptions $CRAWL_ID/crawldb $CRAWL_ID/segments -topN $sizeFetchlist
-numFetchers $numSlaves -noFilter

There are no crawldb or segments in nutch-2.x.

When you use crawlid in inject command it creates a crawlid_webpage table in hbase and when
you use generate, fetch and etc it queries webpage table which does not exist.

Alex.

 

 

-----Original Message-----
From: Christopher Gross 
To: user 
Sent: Wed, May 22, 2013 6:23 pm
Subject: Re: error crawling


I'm trying to crawl. I'm just running the script that I pulled from the
nutch site, so I assumed that it would be good to go, like the old
runbot.sh script. I could try removing that part, but I still get the error
farther down in the main body of the loop.

-- Christopher Gross
Sent from my nexus 7
On May 22, 2013 4:40 PM,  wrote:

> what are you trying to achieve? What is the reason running inject with a
> crawlIId?
>
>
>
>
>
>
> -----Original Message-----
> From: Christopher Gross 
> To: user 
> Sent: Wed, May 22, 2013 12:25 pm
> Subject: Re: error crawling
>
>
> Sure, I'll try.  I'm also confused about this -- I had it working at one
> point, and it stopped working after migrating to a new box (copied
> everything over but cleared out the HBase).
>
> My hadoop.log for today has:
> store.HBaseStore - Keyclass and nameclass match but mismatching table
> names  mappingfile schema is 'webpage' vs actual schema 'crawl_webpage' ,
> assuming they are the same.
>
> I have nothing in a config file for a "crawl_webpage".  I ran:
> grep crawl_webpage *
> and got nothing.
> Running:
> grep webpage *
> gets me hits on gora mapping files for accumulo, hbase, cassandra and sql,
> as well as the nutch-default.xml file.
> nutch-default.xml has a "storage.schema.webpage" which has a value of
> "webpage".
>
> Now, what I'm thinking is that my CRAWL_ID is set to crawl, and for
> whatever reason, that is the table that nutch is making is that CRAWL_ID +
> _ + "webpage".
>
> I tried making the gora mapping file use crawl_webpage but then I ended up
> with some crawl_crawl_webpage error messages, so I cleared out the HBase
> (again) and rolled back the file.
>
> Perhaps I'm running on an older one, can you point me in the right
> direction for getting that "crawl" script that replaces the 1.x "runbot.sh"
> script?
>
>
> -- Chris
>
>
> On Mon, May 20, 2013 at 1:55 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi Chris,
> >
> > On Mon, May 20, 2013 at 10:21 AM, Christopher Gross  > >wrote:
> >
> > > Lewis --
> > > Is the DEBUG something set in the conf/log4j.properties file?  I have
> the
> > > rootLogger set to INFO,DRFA and the threshold is ALL.  Everything else
> is
> > > INFO or WARN (no DEBUGs to be found.)
> > >
> > >
> > Well yes you can set it in the log4j.properties file, however if you are
> > working with anything older than 2.x HEAD then by default the logging is
> > hardcoded as INFO.
> > The DEBUG logging was implemented as of NUTCH-1496 and is now built into
> > 2.x HEAD. An example can be seen here
> >
> >
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/java/org/apache/nutch/parse/ParserJob.java?r1=1379438&r2=1408271
> >
> > BTW here is the HBase thread which I referred to before
> > http://www.mail-archive.com/user@nutch.apache.org/msg09245.html
> >
> >
> > > I'm still a bit lost on what I need to do for the gora-hbase portion.
>  My
> > > gora-hbase-mapping.xml is unchanged.  Also, from the nutch-default.xml
> > > file:
> > > 
> > >   storage.schema.webpage
> > >   webpage
> > >   This value holds the schema name used for Nutch web db.
> > >   Note that Nutch ignores the value in the gora mapping files, and uses
> > >   this as the webpage schema name.
> > >   
> > > 
> > >
> > > So that would lead me to believe that the gora file is just ignored.
> > > If I have the "crawlId" set to "crawlId" -- where do I need to tell
> nutch
> > > to look in the hbase for the "crawlId_webpage"?
> > >
> > > I am unsure as to what your problem is here Chris. Can you please try
> to
> > explain it in layman's terms for me so I understand what problem you are
> > facing?
> > Thanks
> > Lewis
> >
>
>
>

Explanation of RegexURLFIlterTestBase benchmark's

2013-05-23T19:57:04Z

Hi All,
A really nice aspect of the regex (urlfilter-automaton and urfilter-regex)
plugin implementation's in Nutch is that there is a small but very useful
RegexURLFilterBaseTest [0] which compares benchmarks for simple regex
parsing.
The results we get are as follows

urls      automaton      regex
50        343ms           210ms
100      48ms             187ms
200      65ms             363ms
400      100ms           692ms
800      165ms           1385ms

The problem I have here is understanding why the first (50) bench appears
to be more expensive for both implementations?
Additionally, why does this same bench cost much more for automaton?

Anyone have a clue?
Thanks
Lewis

[0]
http://svn.apache.org/viewvc/nutch/branches/2.x/src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java?view=markup

-- 
*Lewis*

Re: Nutch 2.1 pdf parsing

2013-05-23T18:09:38Z

Hi Adriana,
If I were you I would switch your logging to DEBUG for the ParserJob

- log4j.logger.org.apache.nutch.parse.ParserJob=INFO,cmdstdout
+ log4j.logger.org.apache.nutch.parse.ParserJob=DEBUG,cmdstdout


recompile the code, then look closely at the parse chunk of the log to see
what parser is being used, and if there are any particular issues flagged
up @runtime.


On Thu, May 23, 2013 at 8:14 AM, Adriana Farina
wrote:

> Hi,
>
> I'm using Nutch 2.1 in distributed mode on top of Hadoop 1.0.4, with HBase
> 0.90.4 as database.
>
> I wrote a Java class from which I run the crawling cycle, the code that
> implements the crawling cycle is the following:
>
>                   for (int i = 0; i < depth; i++) {
> batchid = generator.generate((Long) args.get(Nutch.ARG_TOPN),
> System.currentTimeMillis(), false, false);
> fetcher.fetch(batchid, 1, false, -1);
> parser.parse(batchid, false, true);
> updater.run(new String[0]);
>   }
>
> The problem is that I'm not able to parse the pdf files, inside HBase I got
> no pdf content. The strange thing is that I got one row with the following
> content: column=p:parsestat, timestamp=1369316742871,
> value=\x04\x90\x03\x02\x96\x01org.apache.nutch.parse.ParseException: Unable
> to successfully parse content\x00.
>
> It seems to me that I have configured all nutch property files correctly.
> Can anybody help me?
>
> Thank you very much.
>
>
> --
> Adriana Farina
>



-- 
*Lewis*

Nutch 2.1 pdf parsing

2013-05-23T15:14:38Z

Hi,

I'm using Nutch 2.1 in distributed mode on top of Hadoop 1.0.4, with HBase
0.90.4 as database.

I wrote a Java class from which I run the crawling cycle, the code that
implements the crawling cycle is the following:

                  for (int i = 0; i < depth; i++) {
batchid = generator.generate((Long) args.get(Nutch.ARG_TOPN),
System.currentTimeMillis(), false, false);
fetcher.fetch(batchid, 1, false, -1);
parser.parse(batchid, false, true);
updater.run(new String[0]);
  }

The problem is that I'm not able to parse the pdf files, inside HBase I got
no pdf content. The strange thing is that I got one row with the following
content: column=p:parsestat, timestamp=1369316742871,
value=\x04\x90\x03\x02\x96\x01org.apache.nutch.parse.ParseException: Unable
to successfully parse content\x00.

It seems to me that I have configured all nutch property files correctly.
Can anybody help me?

Thank you very much.


-- 
Adriana Farina

Re: Nutch 2.1 - Unauthorized

2013-05-23T09:56:40Z

Hi,

I think he is referring to this issue:

https://issues.apache.org/jira/browse/NUTCH-1575

BR,
Tobias

Am 22.05.2013 um 18:14 schrieb Lewis John Mcgibbney:

> Hi Feng,
> Where is the patch please?
> Thank you very much
> Lewis
> 
> On Wednesday, May 22, 2013, feng lu  wrote:
>> Hi Daniel
>> 
>> Now Nutch 2.x can not support solr authentication, I have already open an
>> issue and add a patch , you can patch this and try again.
>> 
>> Thanks
>> 
>> 
>> On Wed, May 22, 2013 at 7:34 PM, Daniel Hüsch > wrote:
>> 
>>> Hi,
>>> 
>>> we used Nutch 1.x with an authentication for tomcat.
>>> We used this:
>>> 
>>> 
>>> 
>>> >>       password="PASSWORD"
>>>       roles="solr_admin"/>
>>> 
>>> in /conf/tomcat-users.xml
>>> And in configuration of Nutch 1.x we used "solr.auth" for the connection
>>> between nutch and solr.
>>> But this function (solr.auth) is not available in Nutch 2.1.
>>> 
>>> So we got an error on hadoop.log at indexing:
>>> 
>>> 2013-05-22 12:50:26,581 ERROR solr.SolrIndexerJob - SolrIndexerJob:
>>> org.apache.solr.common.**SolrException: Unauthorized
>>> Unauthorized
>>> 
>>> 
>>> How can we indexing with an authentication and Nutch 2.1?
>>> 
>>> Thanks,
>>> 
>>> Daniel
>>> 
>> 
>> 
>> 
>> --
>> Don't Grow Old, Grow Up... :-)
>> 
> 
> -- 
> *Lewis*

-- 
Tobias Marx

Zentrum für Informations- und Medienverarbeitung - ZIM

Bergische Universität Wuppertal

Büro: T.11.08
++49 202 439 2237
tmarx@uni-wuppertal.de

OutOfMemoryError for bin/nutch elasticindex ocpnutch -all

2013-05-23T08:47:50Z

Dear List,
 I have been following the instructions at
http://wiki.apache.org/nutch/Nutch2Tutorial to see if I can get a nutch
installation running with ElasticSearch. I have successfully done a crawl
with no real issues, but then when I try and load the results into
elasticsearch I run into trouble.

I issue the command:

bin/nutch elasticindex ocpnutch -all
And it waits around for a long time and then comes back with an error:
Exception in thread "main" java.lang.RuntimeException: job failed:
name=elastic-index [ocpnutch], jobid=job_local_0001

If I look in the logs at:

~/apache-nutch-2.1/runtime/local/logs/hadoop.log
I see several errors like this:
Exception caught on netty layer [[id: 0x569764bd, /192.168.17.39:52554 => /
192.168.17.60:9300]]
java.lang.OutOfMemoryError: Java heap space

There is nothing in the logs on the elastic search.

I have tried changing:
elastic.max.bulk.docs and elastic.max.bulk.size to small sizes
and allocating large amounts of GB to nutch, but to no avail.

The jvm is:
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)

Does anyone have any idea what I am doing wrong - what other diagnostic
information would be helpful to solve this problem?

Thanks a lot,
Regards,
Nicholas W.

Re: error crawling

2013-05-23T01:22:38Z

I'm trying to crawl. I'm just running the script that I pulled from the
nutch site, so I assumed that it would be good to go, like the old
runbot.sh script. I could try removing that part, but I still get the error
farther down in the main body of the loop.

-- Christopher Gross
Sent from my nexus 7
On May 22, 2013 4:40 PM,  wrote:

> what are you trying to achieve? What is the reason running inject with a
> crawlIId?
>
>
>
>
>
>
> -----Original Message-----
> From: Christopher Gross 
> To: user 
> Sent: Wed, May 22, 2013 12:25 pm
> Subject: Re: error crawling
>
>
> Sure, I'll try.  I'm also confused about this -- I had it working at one
> point, and it stopped working after migrating to a new box (copied
> everything over but cleared out the HBase).
>
> My hadoop.log for today has:
> store.HBaseStore - Keyclass and nameclass match but mismatching table
> names  mappingfile schema is 'webpage' vs actual schema 'crawl_webpage' ,
> assuming they are the same.
>
> I have nothing in a config file for a "crawl_webpage".  I ran:
> grep crawl_webpage *
> and got nothing.
> Running:
> grep webpage *
> gets me hits on gora mapping files for accumulo, hbase, cassandra and sql,
> as well as the nutch-default.xml file.
> nutch-default.xml has a "storage.schema.webpage" which has a value of
> "webpage".
>
> Now, what I'm thinking is that my CRAWL_ID is set to crawl, and for
> whatever reason, that is the table that nutch is making is that CRAWL_ID +
> _ + "webpage".
>
> I tried making the gora mapping file use crawl_webpage but then I ended up
> with some crawl_crawl_webpage error messages, so I cleared out the HBase
> (again) and rolled back the file.
>
> Perhaps I'm running on an older one, can you point me in the right
> direction for getting that "crawl" script that replaces the 1.x "runbot.sh"
> script?
>
>
> -- Chris
>
>
> On Mon, May 20, 2013 at 1:55 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi Chris,
> >
> > On Mon, May 20, 2013 at 10:21 AM, Christopher Gross  > >wrote:
> >
> > > Lewis --
> > > Is the DEBUG something set in the conf/log4j.properties file?  I have
> the
> > > rootLogger set to INFO,DRFA and the threshold is ALL.  Everything else
> is
> > > INFO or WARN (no DEBUGs to be found.)
> > >
> > >
> > Well yes you can set it in the log4j.properties file, however if you are
> > working with anything older than 2.x HEAD then by default the logging is
> > hardcoded as INFO.
> > The DEBUG logging was implemented as of NUTCH-1496 and is now built into
> > 2.x HEAD. An example can be seen here
> >
> >
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/java/org/apache/nutch/parse/ParserJob.java?r1=1379438&r2=1408271
> >
> > BTW here is the HBase thread which I referred to before
> > http://www.mail-archive.com/user@nutch.apache.org/msg09245.html
> >
> >
> > > I'm still a bit lost on what I need to do for the gora-hbase portion.
>  My
> > > gora-hbase-mapping.xml is unchanged.  Also, from the nutch-default.xml
> > > file:
> > > 
> > >   storage.schema.webpage
> > >   webpage
> > >   This value holds the schema name used for Nutch web db.
> > >   Note that Nutch ignores the value in the gora mapping files, and uses
> > >   this as the webpage schema name.
> > >   
> > > 
> > >
> > > So that would lead me to believe that the gora file is just ignored.
> > > If I have the "crawlId" set to "crawlId" -- where do I need to tell
> nutch
> > > to look in the hbase for the "crawlId_webpage"?
> > >
> > > I am unsure as to what your problem is here Chris. Can you please try
> to
> > explain it in layman's terms for me so I understand what problem you are
> > facing?
> > Thanks
> > Lewis
> >
>
>
>

Re: error crawling

2013-05-22T20:39:57Z

what are you trying to achieve? What is the reason running inject with a crawlIId?

-----Original Message-----
From: Christopher Gross 
To: user 
Sent: Wed, May 22, 2013 12:25 pm
Subject: Re: error crawling

Sure, I'll try.  I'm also confused about this -- I had it working at one
point, and it stopped working after migrating to a new box (copied
everything over but cleared out the HBase).

My hadoop.log for today has:
store.HBaseStore - Keyclass and nameclass match but mismatching table
names  mappingfile schema is 'webpage' vs actual schema 'crawl_webpage' ,
assuming they are the same.

I have nothing in a config file for a "crawl_webpage".  I ran:
grep crawl_webpage *
and got nothing.
Running:
grep webpage *
gets me hits on gora mapping files for accumulo, hbase, cassandra and sql,
as well as the nutch-default.xml file.
nutch-default.xml has a "storage.schema.webpage" which has a value of
"webpage".

Now, what I'm thinking is that my CRAWL_ID is set to crawl, and for
whatever reason, that is the table that nutch is making is that CRAWL_ID +
_ + "webpage".

I tried making the gora mapping file use crawl_webpage but then I ended up
with some crawl_crawl_webpage error messages, so I cleared out the HBase
(again) and rolled back the file.

Perhaps I'm running on an older one, can you point me in the right
direction for getting that "crawl" script that replaces the 1.x "runbot.sh"
script?

-- Chris

On Mon, May 20, 2013 at 1:55 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Hi Chris,
>
> On Mon, May 20, 2013 at 10:21 AM, Christopher Gross  >wrote:
>
> > Lewis --
> > Is the DEBUG something set in the conf/log4j.properties file?  I have the
> > rootLogger set to INFO,DRFA and the threshold is ALL.  Everything else is
> > INFO or WARN (no DEBUGs to be found.)
> >
> >
> Well yes you can set it in the log4j.properties file, however if you are
> working with anything older than 2.x HEAD then by default the logging is
> hardcoded as INFO.
> The DEBUG logging was implemented as of NUTCH-1496 and is now built into
> 2.x HEAD. An example can be seen here
>
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/java/org/apache/nutch/parse/ParserJob.java?r1=1379438&r2=1408271
>
> BTW here is the HBase thread which I referred to before
> http://www.mail-archive.com/user@nutch.apache.org/msg09245.html
>
>
> > I'm still a bit lost on what I need to do for the gora-hbase portion.  My
> > gora-hbase-mapping.xml is unchanged.  Also, from the nutch-default.xml
> > file:
> > 
> >   storage.schema.webpage
> >   webpage
> >   This value holds the schema name used for Nutch web db.
> >   Note that Nutch ignores the value in the gora mapping files, and uses
> >   this as the webpage schema name.
> >   
> > 
> >
> > So that would lead me to believe that the gora file is just ignored.
> > If I have the "crawlId" set to "crawlId" -- where do I need to tell nutch
> > to look in the hbase for the "crawlId_webpage"?
> >
> > I am unsure as to what your problem is here Chris. Can you please try to
> explain it in layman's terms for me so I understand what problem you are
> facing?
> Thanks
> Lewis
>

Re: error crawling

2013-05-22T19:17:14Z

Sure, I'll try.  I'm also confused about this -- I had it working at one
point, and it stopped working after migrating to a new box (copied
everything over but cleared out the HBase).

My hadoop.log for today has:
store.HBaseStore - Keyclass and nameclass match but mismatching table
names  mappingfile schema is 'webpage' vs actual schema 'crawl_webpage' ,
assuming they are the same.

I have nothing in a config file for a "crawl_webpage".  I ran:
grep crawl_webpage *
and got nothing.
Running:
grep webpage *
gets me hits on gora mapping files for accumulo, hbase, cassandra and sql,
as well as the nutch-default.xml file.
nutch-default.xml has a "storage.schema.webpage" which has a value of
"webpage".

Now, what I'm thinking is that my CRAWL_ID is set to crawl, and for
whatever reason, that is the table that nutch is making is that CRAWL_ID +
_ + "webpage".

I tried making the gora mapping file use crawl_webpage but then I ended up
with some crawl_crawl_webpage error messages, so I cleared out the HBase
(again) and rolled back the file.

Perhaps I'm running on an older one, can you point me in the right
direction for getting that "crawl" script that replaces the 1.x "runbot.sh"
script?

-- Chris

On Mon, May 20, 2013 at 1:55 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Hi Chris,
>
> On Mon, May 20, 2013 at 10:21 AM, Christopher Gross  >wrote:
>
> > Lewis --
> > Is the DEBUG something set in the conf/log4j.properties file?  I have the
> > rootLogger set to INFO,DRFA and the threshold is ALL.  Everything else is
> > INFO or WARN (no DEBUGs to be found.)
> >
> >
> Well yes you can set it in the log4j.properties file, however if you are
> working with anything older than 2.x HEAD then by default the logging is
> hardcoded as INFO.
> The DEBUG logging was implemented as of NUTCH-1496 and is now built into
> 2.x HEAD. An example can be seen here
>
> http://svn.apache.org/viewvc/nutch/branches/2.x/src/java/org/apache/nutch/parse/ParserJob.java?r1=1379438&r2=1408271
>
> BTW here is the HBase thread which I referred to before
> http://www.mail-archive.com/user@nutch.apache.org/msg09245.html
>
>
> > I'm still a bit lost on what I need to do for the gora-hbase portion.  My
> > gora-hbase-mapping.xml is unchanged.  Also, from the nutch-default.xml
> > file:
> > 
> >   storage.schema.webpage
> >   webpage
> >   This value holds the schema name used for Nutch web db.
> >   Note that Nutch ignores the value in the gora mapping files, and uses
> >   this as the webpage schema name.
> >   
> > 
> >
> > So that would lead me to believe that the gora file is just ignored.
> > If I have the "crawlId" set to "crawlId" -- where do I need to tell nutch
> > to look in the hbase for the "crawlId_webpage"?
> >
> > I am unsure as to what your problem is here Chris. Can you please try to
> explain it in layman's terms for me so I understand what problem you are
> facing?
> Thanks
> Lewis
>

Re: Nutch 2.1 - Unauthorized

2013-05-22T16:14:57Z

Hi Feng,
Where is the patch please?
Thank you very much
Lewis

On Wednesday, May 22, 2013, feng lu  wrote:
> Hi Daniel
>
> Now Nutch 2.x can not support solr authentication, I have already open an
> issue and add a patch , you can patch this and try again.
>
> Thanks
>
>
> On Wed, May 22, 2013 at 7:34 PM, Daniel Hüsch wrote:
>
>> Hi,
>>
>> we used Nutch 1.x with an authentication for tomcat.
>> We used this:
>>
>> 
>>
>>  >        password="PASSWORD"
>>        roles="solr_admin"/>
>>
>> in /conf/tomcat-users.xml
>> And in configuration of Nutch 1.x we used "solr.auth" for the connection
>> between nutch and solr.
>> But this function (solr.auth) is not available in Nutch 2.1.
>>
>> So we got an error on hadoop.log at indexing:
>>
>> 2013-05-22 12:50:26,581 ERROR solr.SolrIndexerJob - SolrIndexerJob:
>> org.apache.solr.common.**SolrException: Unauthorized
>> Unauthorized
>>
>>
>> How can we indexing with an authentication and Nutch 2.1?
>>
>> Thanks,
>>
>> Daniel
>>
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>

-- 
*Lewis*

Re: Nutch 2.1 - Unauthorized

2013-05-22T15:36:31Z

Hi Daniel

Now Nutch 2.x can not support solr authentication, I have already open an
issue and add a patch , you can patch this and try again.

Thanks


On Wed, May 22, 2013 at 7:34 PM, Daniel Hüsch wrote:

> Hi,
>
> we used Nutch 1.x with an authentication for tomcat.
> We used this:
>
> 
>
>          password="PASSWORD"
>        roles="solr_admin"/>
>
> in /conf/tomcat-users.xml
> And in configuration of Nutch 1.x we used "solr.auth" for the connection
> between nutch and solr.
> But this function (solr.auth) is not available in Nutch 2.1.
>
> So we got an error on hadoop.log at indexing:
>
> 2013-05-22 12:50:26,581 ERROR solr.SolrIndexerJob - SolrIndexerJob:
> org.apache.solr.common.**SolrException: Unauthorized
> Unauthorized
>
>
> How can we indexing with an authentication and Nutch 2.1?
>
> Thanks,
>
> Daniel
>



-- 
Don't Grow Old, Grow Up... :-)

Nutch 2.1 - Unauthorized

2013-05-22T11:34:26Z

Hi,

we used Nutch 1.x with an authentication for tomcat.
We used this:



  

in /conf/tomcat-users.xml
And in configuration of Nutch 1.x we used "solr.auth" for the connection 
between nutch and solr.
But this function (solr.auth) is not available in Nutch 2.1.

So we got an error on hadoop.log at indexing:

2013-05-22 12:50:26,581 ERROR solr.SolrIndexerJob - SolrIndexerJob: 
org.apache.solr.common.SolrException: Unauthorized
Unauthorized


How can we indexing with an authentication and Nutch 2.1?

Thanks,

Daniel

Re: Using Nutch and Hive together

2013-05-22T01:22:17Z

I have another question.  As a side-solution (as in, something that
needs to be done soon and can be quick and dirty), would it be possible
to pipe the Nutch output somewhere and then using a cron job (or some
timed process) to import it into Hive?  Has anyone done this?


On Wed, May 1, 2013 at 11:16 PM, Renato Marroquín Mogrovejo <
renatoj.marroquin@gmail.com> wrote:

> Hi Yves,
>
> Just get your head around hadoop and start playing around with it.
> Nutch is a great place to start getting familiarized with Gora.
> We will help you any time you need it and then you can help us push
> Gora forward (:
>
>
> Renato M.
>
> 2013/5/1 Yves S. Garret :
> > Hi Renato,
> >
> > Sounds kinda fun :) .  I'll start reading here about Gora in
> > order to better understand.  I'm also reading Hadoop: The
> > Definitive Guide.
> >
> > http://gora.apache.org/
> >
> > Should I look into anything else in order to learn more about
> > Gora?  I'll have to admit, I'm new to Hadoop and don't have
> > the firmest grasp of the internals.  I'm not sure how useful
> > I'll be at this moment.
> >
> >
> > On Tue, Apr 30, 2013 at 6:21 PM, Renato Marroquín Mogrovejo <
> > renatoj.marroquin@gmail.com> wrote:
> >
> >> Hi Yves,
> >>
> >> Apache Gora does not support Apache Hive just yet, but we have it on
> >> our future plans. If you were willing to dive into an adventure with
> >> Gora we would be happy to help you out with that.
> >> There is a Pig-Gora adapter patch on JIRA, maybe you would like to
> >> give it a look? Although there is a bit of work involved in that one
> >> as well.
> >>
> >>
> >> Renato M.
> >>
> >> 2013/4/30 Yves S. Garret :
> >> > Ok.  I think I understand.  So there's an adapter involved here.
> >> >
> >> >
> >> > On Tue, Apr 30, 2013 at 5:04 PM, Tejas Patil <
> tejas.patil.cs@gmail.com
> >> >wrote:
> >> >
> >> >> Nutch 2.x series is built on Gora which offers storage abstraction.
> From
> >> >> the Gora project main page, I think gora has adapters for accessing
> the
> >> >> data and making analysis through Apache Hive but it wont support
> >> storing of
> >> >> nutch data into hive.
> >> >>
> >> >> There are Gora experts on this group who can answer better.
> >> >>
> >> >>
> >> >> On Tue, Apr 30, 2013 at 1:46 PM, Yves S. Garret
> >> >> wrote:
> >> >>
> >> >> > Hello,
> >> >> >
> >> >> > I'm curious.  If I wanted to store the URLs for Nutch (version
> 2.1) in
> >> >> Hive
> >> >> > (version
> >> >> > 0.9.0) and then store the output from Nutch in Hive, how would
I do
> >> that?
> >> >> > Any
> >> >> > pointers?
> >> >> >
> >> >> > I've googled for "nutch hive" (maybe there's a better term?),
but
> >> haven't
> >> >> > found
> >> >> > anything specific or very helpful.  I'll keep looking and
> >> experimenting.
> >> >> > Your help is
> >> >> > appreciated.
> >> >> >
> >> >> > --Yves
> >> >> >
> >> >>
> >>
>

Re: Nutch 2.1 generate: how to get multiple maps in deploy-mode

2013-05-21T14:34:51Z

Nutch 2.1 use apache gora to access the cassandra database, so that
implemented the hadoop inputsplit interface to generate input split from
cassandra. So you can find some documentation about gora- cassandra model.



On May 21, 2013 7:45 PM, "Martin Aesch"  wrote:

> Hi,
>
> I'm running nutch-2.1 (on top of cassandra) on a single-node hadoop
> "cluster". Sorry in case my question is noobish or more a hadoop issue.
>
> In short: How can I force nutch generate to provide a filesplitsize>1
> which seems to be necesarry to run multiple map jobs?
>
> I am seeing that only one input split is generated for nutch generate:
> For
> ./bin/nutch generate -topN 1000000 -noFilter
>
>
> 2013-05-21 13:36:33,960 INFO org.apache.hadoop.mapred.JobInProgress:
> Input size for job job_201305211335_0001 = 0. Number of splits = 1
> 2013-05-21 13:36:33,960 INFO org.apache.hadoop.mapred.JobInProgress:
> job_201305211335_0001 LOCALITY_WAIT_FACTOR=0.0
> 2013-05-21 13:36:33,961 INFO org.apache.hadoop.mapred.JobInProgress: Job
> job_201305211335_0001 initialized successfully with 1 map tasks and 2
> reduce tasks.
> 2013-05-21 13:36:34,278 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task (JOB_SETUP) 'attempt_201305211335_0001_m_000002_0' to tip
> task_201305211335_0001_m_000002, for tracker
> 'tracker_Ubuntu-1204-precise-64-minimal:localhost/127.0.0.1:34436'
> 2013-05-21 13:36:36,728 INFO org.apache.hadoop.mapred.JobInProgress:
> Task 'attempt_201305211335_0001_m_000002_0' has completed
> task_201305211335_0001_m_000002 successfully.
> 2013-05-21 13:36:36,735 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing a non-local task task_201305211335_0001_m_000000
> 2013-05-21 13:36:36,735 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task (MAP) 'attempt_201305211335_0001_m_000000_0' to tip
> task_201305211335_0001_m_000000, for tracker
> 'tracker_Ubuntu-1204-precise-64-minimal:localhost/127.0.0.1:34436'
>
>
> System load did not reach its maximum by far, neither in terms of CPU
> nor in terms of i/o-waiting. It took 100 minutes, which seems very fair
> for one single map, since I have about 50M webpages in my database.
> Jobtracker says max map tasks is 2 and max reduce tasks is 2, but only 1
> map task is running.
>
> This is my mapred-site.xml, which should be ok, in particular I
> overwrite mapred.job.tracker not to be "local":
>
> 
> 
> 
>
> 
>      
>          mapred.job.tracker
>          localhost:9001
>      
>      
>         mapreduce.jobtracker.staging.root.dir
>         /user
>      
>      
>         mapred.map.tasks
>         2
>         
>         define mapred.map tasks to be number of slave hosts
>         
>      
> 
>   mapred.reduce.tasks
>   2
>   
>     define mapred.reduce tasks to be number of slave hosts
>   
> 
>
> 
>
>
> Thanks and best regards,
> Martin
>
>
>
>
>
>
>

Nutch 2.1 generate: how to get multiple maps in deploy-mode

2013-05-21T11:44:47Z

Hi,

I'm running nutch-2.1 (on top of cassandra) on a single-node hadoop
"cluster". Sorry in case my question is noobish or more a hadoop issue.

In short: How can I force nutch generate to provide a filesplitsize>1
which seems to be necesarry to run multiple map jobs?

I am seeing that only one input split is generated for nutch generate:
For 
./bin/nutch generate -topN 1000000 -noFilter


2013-05-21 13:36:33,960 INFO org.apache.hadoop.mapred.JobInProgress:
Input size for job job_201305211335_0001 = 0. Number of splits = 1
2013-05-21 13:36:33,960 INFO org.apache.hadoop.mapred.JobInProgress:
job_201305211335_0001 LOCALITY_WAIT_FACTOR=0.0
2013-05-21 13:36:33,961 INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201305211335_0001 initialized successfully with 1 map tasks and 2
reduce tasks.
2013-05-21 13:36:34,278 INFO org.apache.hadoop.mapred.JobTracker: Adding
task (JOB_SETUP) 'attempt_201305211335_0001_m_000002_0' to tip
task_201305211335_0001_m_000002, for tracker
'tracker_Ubuntu-1204-precise-64-minimal:localhost/127.0.0.1:34436'
2013-05-21 13:36:36,728 INFO org.apache.hadoop.mapred.JobInProgress:
Task 'attempt_201305211335_0001_m_000002_0' has completed
task_201305211335_0001_m_000002 successfully.
2013-05-21 13:36:36,735 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing a non-local task task_201305211335_0001_m_000000
2013-05-21 13:36:36,735 INFO org.apache.hadoop.mapred.JobTracker: Adding
task (MAP) 'attempt_201305211335_0001_m_000000_0' to tip
task_201305211335_0001_m_000000, for tracker
'tracker_Ubuntu-1204-precise-64-minimal:localhost/127.0.0.1:34436'


System load did not reach its maximum by far, neither in terms of CPU
nor in terms of i/o-waiting. It took 100 minutes, which seems very fair
for one single map, since I have about 50M webpages in my database.
Jobtracker says max map tasks is 2 and max reduce tasks is 2, but only 1
map task is running.

This is my mapred-site.xml, which should be ok, in particular I
overwrite mapred.job.tracker not to be "local":






     
         mapred.job.tracker
         localhost:9001
     
     
        mapreduce.jobtracker.staging.root.dir
        /user
     
      
        mapred.map.tasks
        2
        
        define mapred.map tasks to be number of slave hosts
         
      
 
  mapred.reduce.tasks
  2
  
    define mapred.reduce tasks to be number of slave hosts
   
 




Thanks and best regards,
Martin

Re: Using Nutch and Hive together

2013-05-20T21:50:11Z

Hi, sorry for not writing back earlier.  But yes, I would like to take a
look,
where should I begin?


On Tue, Apr 30, 2013 at 6:21 PM, Renato Marroquín Mogrovejo <
renatoj.marroquin@gmail.com> wrote:

> Hi Yves,
>
> Apache Gora does not support Apache Hive just yet, but we have it on
> our future plans. If you were willing to dive into an adventure with
> Gora we would be happy to help you out with that.
> There is a Pig-Gora adapter patch on JIRA, maybe you would like to
> give it a look? Although there is a bit of work involved in that one
> as well.
>
>
> Renato M.
>
> 2013/4/30 Yves S. Garret :
> > Ok.  I think I understand.  So there's an adapter involved here.
> >
> >
> > On Tue, Apr 30, 2013 at 5:04 PM, Tejas Patil  >wrote:
> >
> >> Nutch 2.x series is built on Gora which offers storage abstraction. From
> >> the Gora project main page, I think gora has adapters for accessing the
> >> data and making analysis through Apache Hive but it wont support
> storing of
> >> nutch data into hive.
> >>
> >> There are Gora experts on this group who can answer better.
> >>
> >>
> >> On Tue, Apr 30, 2013 at 1:46 PM, Yves S. Garret
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm curious.  If I wanted to store the URLs for Nutch (version 2.1) in
> >> Hive
> >> > (version
> >> > 0.9.0) and then store the output from Nutch in Hive, how would I do
> that?
> >> > Any
> >> > pointers?
> >> >
> >> > I've googled for "nutch hive" (maybe there's a better term?), but
> haven't
> >> > found
> >> > anything specific or very helpful.  I'll keep looking and
> experimenting.
> >> > Your help is
> >> > appreciated.
> >> >
> >> > --Yves
> >> >
> >>
>

Re: error crawling

2013-05-20T17:55:00Z

Hi Chris,

On Mon, May 20, 2013 at 10:21 AM, Christopher Gross wrote:

> Lewis --
> Is the DEBUG something set in the conf/log4j.properties file?  I have the
> rootLogger set to INFO,DRFA and the threshold is ALL.  Everything else is
> INFO or WARN (no DEBUGs to be found.)
>
>
Well yes you can set it in the log4j.properties file, however if you are
working with anything older than 2.x HEAD then by default the logging is
hardcoded as INFO.
The DEBUG logging was implemented as of NUTCH-1496 and is now built into
2.x HEAD. An example can be seen here
http://svn.apache.org/viewvc/nutch/branches/2.x/src/java/org/apache/nutch/parse/ParserJob.java?r1=1379438&r2=1408271

BTW here is the HBase thread which I referred to before
http://www.mail-archive.com/user@nutch.apache.org/msg09245.html


> I'm still a bit lost on what I need to do for the gora-hbase portion.  My
> gora-hbase-mapping.xml is unchanged.  Also, from the nutch-default.xml
> file:
> 
>   storage.schema.webpage
>   webpage
>   This value holds the schema name used for Nutch web db.
>   Note that Nutch ignores the value in the gora mapping files, and uses
>   this as the webpage schema name.
>   
> 
>
> So that would lead me to believe that the gora file is just ignored.
> If I have the "crawlId" set to "crawlId" -- where do I need to tell nutch
> to look in the hbase for the "crawlId_webpage"?
>
> I am unsure as to what your problem is here Chris. Can you please try to
explain it in layman's terms for me so I understand what problem you are
facing?
Thanks
Lewis

Re: error crawling

2013-05-20T17:21:55Z

Lewis --
Is the DEBUG something set in the conf/log4j.properties file?  I have the
rootLogger set to INFO,DRFA and the threshold is ALL.  Everything else is
INFO or WARN (no DEBUGs to be found.)

Is there something I should set elsewhere that would be causing this?

I'm still a bit lost on what I need to do for the gora-hbase portion.  My
gora-hbase-mapping.xml is unchanged.  Also, from the nutch-default.xml file:

  storage.schema.webpage
  webpage
  This value holds the schema name used for Nutch web db.
  Note that Nutch ignores the value in the gora mapping files, and uses
  this as the webpage schema name.
  


So that would lead me to believe that the gora file is just ignored.
If I have the "crawlId" set to "crawlId" -- where do I need to tell nutch
to look in the hbase for the "crawlId_webpage"?


-- Chris


On Mon, May 20, 2013 at 11:56 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Please search the mailing list for the HBase logging. There was a
> conversation on this reasonably recently.
>
> Please see my other response for the rest.
> hth
> Lewis
>
> On Monday, May 20, 2013, Christopher Gross  wrote:
> > Ok, so the crawlId isn't like the directories used in the 1.x versions of
> > nutch.
> >
> > Well, changing that line makes that part work.  I still get the "Skipping
> > ; different batch id (null)" error.
> >
> > I'm not sure if this line from the hadoop.log file relates:
> > INFO  store.HBaseStore - Keyclass and nameclass match but mismatching
> table
> > names  mappingfile schema is 'webpage' vs actual schema 'crawl_webpage' ,
> > assuming they are the same.
> >
> > Any ideas for that one?
> >
> > -- Chris
> >
> >
> > On Fri, May 17, 2013 at 4:32 PM, Tejas Patil  >wrote:
> >
> >> The exception speaks about the problem:
> >>
> >> java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal
> >> first
> >> character <46> at 0.
> >> User-space table names can only start with 'word characters': i.e.
> >> [a-zA-Z_0-9]: ./crawl/_webpage
> >>
> >> The crawlId passed must follow the regex [a-zA-Z_0-9]. The one you
> passed
> >> has dot and slash.
> >> $ ./bin/nutch inject urls/ -crawlId ./crawl/
> >>
> >> Try this:
> >> $ ./bin/nutch inject urls/ -crawlId crawl
> >>
> >>
> >>
> >> On Fri, May 17, 2013 at 12:47 PM,  wrote:
> >>
> >> > What if you do bin/nutch inject urls/ ?
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Christopher Gross 
> >> > To: user 
> >> > Sent: Fri, May 17, 2013 11:26 am
> >> > Subject: error crawling
> >> >
> >> >
> >> > I'm having trouble getting my nutch working.  I had it on another
> server
> >> > and it was working fine.  I migrated it to a new server, and I've been
> >> > getting nothing but problems.  My old script wasn't working right
> >> (getting
> >> > a lot of "skipping" on the parser saying that the crawl id was null [a
> >> > separate point of frustration]), so now I'm trying the 'newer' crawl
> >> > script.  This one is worse, since I can't even get the inject to work.
> >> >
> >> > urls contains a "seed.txt" file that worked previously and contains a
> >> bunch
> >> > of urls.  crawl is empty.
> >> >
> >> > from my $NUTCH_HOME directory:
> >> >
> >> > $ ./bin/nutch inject urls/ -crawlId ./crawl/
> >> > InjectorJob: starting
> >> > InjectorJob: urlDir: urls
> >> > InjectorJob: org.apache.gora.util.GoraException:
> >> > java.lang.RuntimeException: java.lang.IllegalArgumentException:
> Illegal
> >> > first character <46> at 0. User-space table names can only start
with
> >> 'word
> >> > characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
> >> >         at
> >> >
> >> >
> >>
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> >> >         at
> >> >
> >> >
> >>
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
> >> >         at
> >> >
> >>
> org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
> >> >         at
> org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
> >> >         at
> >> org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
> >> >         at
> org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
> >> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >> >         at
> org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
> >> > Caused by: java.lang.RuntimeException:
> >> java.lang.IllegalArgumentException:
> >> > Illegal first character <46> at 0. User-space table names can only
> start
> >> > with 'word characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
> >> >         at
> >> > org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:125)
> >> >         at
> >> >
> >> >
> >>
>
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> >> >         at
> >> >
> >> >
> >>
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> >> >         ... 7 more
> >> > Caused by: java.lang.IllegalArgumentException: Illegal first character
> >> <46>
> >> > at 0. User-space table names can only start with 'word characters':
> i.e.
> >> > [a-zA-Z_0-9]: ./crawl/_webpage
> >> >         at
> >> >
> >> >
> >> org.apache.hadoop.hbase.HTableDescriptor.
>
> --
> *Lewis*
>

Re: error crawling

2013-05-20T15:56:38Z

Please search the mailing list for the HBase logging. There was a
conversation on this reasonably recently.

Please see my other response for the rest.
hth
Lewis

On Monday, May 20, 2013, Christopher Gross  wrote:
> Ok, so the crawlId isn't like the directories used in the 1.x versions of
> nutch.
>
> Well, changing that line makes that part work.  I still get the "Skipping
> ; different batch id (null)" error.
>
> I'm not sure if this line from the hadoop.log file relates:
> INFO  store.HBaseStore - Keyclass and nameclass match but mismatching
table
> names  mappingfile schema is 'webpage' vs actual schema 'crawl_webpage' ,
> assuming they are the same.
>
> Any ideas for that one?
>
> -- Chris
>
>
> On Fri, May 17, 2013 at 4:32 PM, Tejas Patil wrote:
>
>> The exception speaks about the problem:
>>
>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal
>> first
>> character <46> at 0.
>> User-space table names can only start with 'word characters': i.e.
>> [a-zA-Z_0-9]: ./crawl/_webpage
>>
>> The crawlId passed must follow the regex [a-zA-Z_0-9]. The one you passed
>> has dot and slash.
>> $ ./bin/nutch inject urls/ -crawlId ./crawl/
>>
>> Try this:
>> $ ./bin/nutch inject urls/ -crawlId crawl
>>
>>
>>
>> On Fri, May 17, 2013 at 12:47 PM,  wrote:
>>
>> > What if you do bin/nutch inject urls/ ?
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Christopher Gross 
>> > To: user 
>> > Sent: Fri, May 17, 2013 11:26 am
>> > Subject: error crawling
>> >
>> >
>> > I'm having trouble getting my nutch working.  I had it on another
server
>> > and it was working fine.  I migrated it to a new server, and I've been
>> > getting nothing but problems.  My old script wasn't working right
>> (getting
>> > a lot of "skipping" on the parser saying that the crawl id was null [a
>> > separate point of frustration]), so now I'm trying the 'newer' crawl
>> > script.  This one is worse, since I can't even get the inject to work.
>> >
>> > urls contains a "seed.txt" file that worked previously and contains a
>> bunch
>> > of urls.  crawl is empty.
>> >
>> > from my $NUTCH_HOME directory:
>> >
>> > $ ./bin/nutch inject urls/ -crawlId ./crawl/
>> > InjectorJob: starting
>> > InjectorJob: urlDir: urls
>> > InjectorJob: org.apache.gora.util.GoraException:
>> > java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal
>> > first character <46> at 0. User-space table names can only start with
>> 'word
>> > characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
>> >         at
>> >
>> >
>>
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
>> >         at
>> >
>> >
>>
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
>> >         at
>> >
>>
org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
>> >         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
>> >         at
>> org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
>> >         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
>> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >         at
org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
>> > Caused by: java.lang.RuntimeException:
>> java.lang.IllegalArgumentException:
>> > Illegal first character <46> at 0. User-space table names can only
start
>> > with 'word characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
>> >         at
>> > org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:125)
>> >         at
>> >
>> >
>>
org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
>> >         at
>> >
>> >
>>
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
>> >         ... 7 more
>> > Caused by: java.lang.IllegalArgumentException: Illegal first character
>> <46>
>> > at 0. User-space table names can only start with 'word characters':
i.e.
>> > [a-zA-Z_0-9]: ./crawl/_webpage
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.HTableDescriptor.

-- 
*Lewis*

Re: nutch crawl

2013-05-20T15:55:21Z

Hi Chris,

Please see the documentation I put up on the wiki for this phenomenon

http://wiki.apache.org/nutch/ErrorMessagesInNutch2#Nutch_logging_shows_Skipping_http:.2F.2FmyurlForParsing.com.3B_different_batch_id_.28null.29

Also, please search the mailing list for a recent discussion on the topic.

Finally, I logged and issue in Jira to improve logging for this scenario. I
don't agree with the logging of mark's as oppose to the identification of
batchId's to which those mark's should belong.
If we know the batchId(s) then we can at least attempt to generate (I use
the term generate not to specifically relate to the GeneratorJob, however
this is one of the tools that Generates Mark's) Mark's for the specific
WebPage.

Right now there is a bit of work to be done here as this has come up
several times and is still not quite fixed.

It just occured to me that as of now ALL of this logging has been silenced
to DEBUG level... I am not sure that this is useful enough for obtaining
metrics upon how many URLs are skipped due to various Mark's being absent.

https://issues.apache.org/jira/browse/NUTCH-1567

On Monday, May 20, 2013, Christopher Gross  wrote:
> I'm attempting to get a crawl working using scripts, but I've been getting
> a "Skipping ; different batch id (null)" error and then nothing new
in
> Solr.  So I've reverted back to trying out the "crawl" for the nutch
script:
>
> ./nutch crawl ../urls/ -solr "http://localhost/nutchsolr" -threads 5
-depth
> 3 -topN 100
>
> urls has the "seed.txt" file with some sites.  It definitely is able to
get
> pages (finding other hostnames in the lists scrolling through the screen),
> but then it is still skipping with the "batch id (null)" message for
> everything it finds.
>
> Any guidance/advice would be appreciated.
>
> Thanks!
>
> -- Chris
>

-- 
*Lewis*

Re: nutch crawl

2013-05-20T14:28:07Z

Hi Christopher

It will check the update db mark when indexing. but now the update db mark
is null. so it skip the url. maybe this url is not parsed success, you can
check the log to see what happens.


On Mon, May 20, 2013 at 9:44 PM, Christopher Gross wrote:

> I'm attempting to get a crawl working using scripts, but I've been getting
> a "Skipping ; different batch id (null)" error and then nothing new in
> Solr.  So I've reverted back to trying out the "crawl" for the nutch
> script:
>
> ./nutch crawl ../urls/ -solr "http://localhost/nutchsolr" -threads 5
> -depth
> 3 -topN 100
>
> urls has the "seed.txt" file with some sites.  It definitely is able to get
> pages (finding other hostnames in the lists scrolling through the screen),
> but then it is still skipping with the "batch id (null)" message for
> everything it finds.
>
> Any guidance/advice would be appreciated.
>
> Thanks!
>
> -- Chris
>



-- 
Don't Grow Old, Grow Up... :-)

nutch crawl

2013-05-20T13:44:06Z

I'm attempting to get a crawl working using scripts, but I've been getting
a "Skipping ; different batch id (null)" error and then nothing new in
Solr.  So I've reverted back to trying out the "crawl" for the nutch script:

./nutch crawl ../urls/ -solr "http://localhost/nutchsolr" -threads 5 -depth
3 -topN 100

urls has the "seed.txt" file with some sites.  It definitely is able to get
pages (finding other hostnames in the lists scrolling through the screen),
but then it is still skipping with the "batch id (null)" message for
everything it finds.

Any guidance/advice would be appreciated.

Thanks!

-- Chris

Re: error crawling

2013-05-20T13:05:39Z

Ok, so the crawlId isn't like the directories used in the 1.x versions of
nutch.

Well, changing that line makes that part work.  I still get the "Skipping
; different batch id (null)" error.

I'm not sure if this line from the hadoop.log file relates:
INFO  store.HBaseStore - Keyclass and nameclass match but mismatching table
names  mappingfile schema is 'webpage' vs actual schema 'crawl_webpage' ,
assuming they are the same.

Any ideas for that one?

-- Chris


On Fri, May 17, 2013 at 4:32 PM, Tejas Patil wrote:

> The exception speaks about the problem:
>
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal
> first
> character <46> at 0.
> User-space table names can only start with 'word characters': i.e.
> [a-zA-Z_0-9]: ./crawl/_webpage
>
> The crawlId passed must follow the regex [a-zA-Z_0-9]. The one you passed
> has dot and slash.
> $ ./bin/nutch inject urls/ -crawlId ./crawl/
>
> Try this:
> $ ./bin/nutch inject urls/ -crawlId crawl
>
>
>
> On Fri, May 17, 2013 at 12:47 PM,  wrote:
>
> > What if you do bin/nutch inject urls/ ?
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Christopher Gross 
> > To: user 
> > Sent: Fri, May 17, 2013 11:26 am
> > Subject: error crawling
> >
> >
> > I'm having trouble getting my nutch working.  I had it on another server
> > and it was working fine.  I migrated it to a new server, and I've been
> > getting nothing but problems.  My old script wasn't working right
> (getting
> > a lot of "skipping" on the parser saying that the crawl id was null [a
> > separate point of frustration]), so now I'm trying the 'newer' crawl
> > script.  This one is worse, since I can't even get the inject to work.
> >
> > urls contains a "seed.txt" file that worked previously and contains a
> bunch
> > of urls.  crawl is empty.
> >
> > from my $NUTCH_HOME directory:
> >
> > $ ./bin/nutch inject urls/ -crawlId ./crawl/
> > InjectorJob: starting
> > InjectorJob: urlDir: urls
> > InjectorJob: org.apache.gora.util.GoraException:
> > java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal
> > first character <46> at 0. User-space table names can only start with
> 'word
> > characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
> >         at
> >
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> >         at
> >
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
> >         at
> >
> org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
> >         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
> >         at
> org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
> >         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >         at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
> > Caused by: java.lang.RuntimeException:
> java.lang.IllegalArgumentException:
> > Illegal first character <46> at 0. User-space table names can only start
> > with 'word characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
> >         at
> > org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:125)
> >         at
> >
> >
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> >         at
> >
> >
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> >         ... 7 more
> > Caused by: java.lang.IllegalArgumentException: Illegal first character
> <46>
> > at 0. User-space table names can only start with 'word characters': i.e.
> > [a-zA-Z_0-9]: ./crawl/_webpage
> >         at
> >
> >
> org.apache.hadoop.hbase.HTableDescriptor.isLegalTableName(HTableDescriptor.java:280)
> >         at
> >
> org.apache.hadoop.hbase.HTableDescriptor.(HTableDescriptor.java:172)
> >         at
> >
> org.apache.hadoop.hbase.HTableDescriptor.(HTableDescriptor.java:158)
> >         at
> >
> >
> org.apache.gora.hbase.store.HBaseMapping$HBaseMappingBuilder.build(HBaseMapping.java:171)
> >         at
> > org.apache.gora.hbase.store.HBaseStore.readMapping(HBaseStore.java:592)
> >         at
> > org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:111)
> >         ... 9 more
> >
> > Where is the "_webpage" coming from?  Am I just missing something?
> >
> > Any help/ideas/references would be appreciated.
> >
> > Thanks!
> >
> > -- Chris
> >
> >
> >
>

[REQUEST] (NUTCH-1569) Upgrade 2.x to Gora 0.3

2013-05-19T21:39:16Z

Hi All,
I submitted a patch to upgrade the Nutch 2.x Branch codebase to the newly
released Gora 0.3.
The patch can be found here [0].
It would be excellent if folks could please test this patch and provide
feedback to the dev@ list.
The feedback will be very helpful in allowing us to progress towards a
Nutch 2.2 Release.
Thank you very much.
Lewis

[0] https://issues.apache.org/jira/browse/NUTCH-1569

-- 
*Lewis*

Re: Status of Elasticsearch indexer?

2013-05-19T02:09:59Z

Hi Chris,
Thanks for getting on the list and discussing these aspects of development
:0)
>From my perspective there are a number of observations

BRANCH 2.x
* NUTCH-1568 [0] is ripe for development. My sole justification for not
addressing this is that we wish to push Nutch 2.2 and it is safe to say
that there will not be enough testing to push toe code and mark it as
stable!
* NUTCH-1486 [1] is ready for testing (I know this is not elastic search
but I thought I'd throw it in there)
* Ferdy committed NUTCH-1445 [2] which enables you to index 2.x data to
Elastic Search but it is not pluggable so to speak. This will most likely
happen once we shift 2,x architecture pluggable in 2.3 development.

TRUNK

* Since Julien committed NUTCH-1047 trunk is pluggable to the tune of Solr
3.X, however Sebatian submitted a patch for a CSV indexer [4] and it would
nto be very hard to get the MongoDB patch ported to pluggable architecture
either I wouldn't imagine.
* Porting of Elastic Search from 2.x to pluggable trunk will most likely
happen in 1.8 development drive.

I think that wraps it up from me. Most likely there is something I've
missed out though!
It would be really great if you were able to chip in on any of the above...
we are always in need of porting stuff... and actually most critically
reviewing the mountain of patches we have in Jira :0)
hth
Lewis

[0] https://issues.apache.org/jira/browse/NUTCH-1568
[1] https://issues.apache.org/jira/browse/NUTCH-1486
[2] https://issues.apache.org/jira/browse/NUTCH-1445
[3] https://issues.apache.org/jira/browse/NUTCH-1047
[4] https://issues.apache.org/jira/browse/NUTCH-1541

On Fri, May 17, 2013 at 12:55 PM, Chris Hairfield <
chairfield@latitudegeo.com> wrote:

> Hello everyone,
>
> I've been eagerly awaiting some of the functionality slated for 2.x,
> especially around your work integrating with Elasticsearch. If possible,
> could you give any additional status on pluggable indexing (NUTCH-1568) and
> the nutch-elasticsearch-indexer (NUTCH-1527)?
>
> It's been a wonderful experience diving into Nutch for the last month and
> watching you guys do pretty awesome work. Now that I can finally say I no
> longer feel completely overwhelmed, I'd like to throw in my support for
> these items. Further, if there is work that still needs to be done, I might
> like to try helping out myself :)
>
> Thanks!
> Chris
>

-- 
*Lewis*

Re: Getting error while running nutch in eclips in window environment

2013-05-18T19:20:38Z

You need to follow the tutorial here
http://wiki.apache.org/nutch/RunNutchInEclipse
If after reading this thoroughly you have some problems please let us know
about them.
Thank you
Lewis


On Thu, May 16, 2013 at 12:07 PM, harsh yadav  wrote:

> 2013-05-17 00:33:09,376 WARN  crawl.Crawl (Crawl.java:run(97)) - solrUrl is
> not set, indexing will be skipped...
> 2013-05-17 00:33:09,522 INFO  crawl.Crawl (Crawl.java:run(108)) - crawl
> started in: crawl
> 2013-05-17 00:33:09,523 INFO  crawl.Crawl (Crawl.java:run(109)) -
> rootUrlDir = urls
> 2013-05-17 00:33:09,523 INFO  crawl.Crawl (Crawl.java:run(110)) - threads =
> 10
> 2013-05-17 00:33:09,523 INFO  crawl.Crawl (Crawl.java:run(111)) - depth = 3
> 2013-05-17 00:33:09,523 INFO  crawl.Crawl (Crawl.java:run(112)) -
> solrUrl=null
> 2013-05-17 00:33:09,524 INFO  crawl.Crawl (Crawl.java:run(114)) - topN = 50
> 2013-05-17 00:33:09,534 INFO  crawl.Injector (Injector.java:inject(257)) -
> Injector: starting at 2013-05-17 00:33:09
> 2013-05-17 00:33:09,535 INFO  crawl.Injector (Injector.java:inject(258)) -
> Injector: crawlDb: crawl/crawldb
> 2013-05-17 00:33:09,535 INFO  crawl.Injector (Injector.java:inject(259)) -
> Injector: urlDir: urls
> 2013-05-17 00:33:09,583 INFO  crawl.Injector (Injector.java:inject(269)) -
> Injector: Converting injected urls to crawl db entries.
> 2013-05-17 00:33:09,610 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) -
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2013-05-17 00:33:09,663 WARN  mapred.JobClient
> (JobClient.java:configureCommandLineOptions(661)) - No job jar file set.
>  User classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 2013-05-17 00:33:09,678 INFO  mapred.FileInputFormat
> (FileInputFormat.java:listStatus(192)) - Total input paths to process : 1
> 2013-05-17 00:33:10,158 INFO  mapred.JobClient
> (JobClient.java:monitorAndPrintJob(1275)) - Running job: job_local_0001
> 2013-05-17 00:33:10,161 INFO  mapred.FileInputFormat
> (FileInputFormat.java:listStatus(192)) - Total input paths to process : 1
> 2013-05-17 00:33:10,212 INFO  mapred.MapTask
> (MapTask.java:runOldMapper(347)) - numReduceTasks: 1
> 2013-05-17 00:33:10,217 INFO  mapred.MapTask (MapTask.java:(776)) -
> io.sort.mb = 100
> 2013-05-17 00:33:10,241 INFO  mapred.MapTask (MapTask.java:(788)) -
> data buffer = 79691776/99614720
> 2013-05-17 00:33:10,241 INFO  mapred.MapTask (MapTask.java:(789)) -
> record buffer = 262144/327680
> 2013-05-17 00:33:10,251 WARN  plugin.PluginRepository
> (PluginManifestParser.java:getPluginFolder(123)) - Plugins: directory not
> found: plugins
> 2013-05-17 00:33:10,252 INFO  plugin.PluginRepository
> (PluginRepository.java:displayStatus(313)) - Plugin Auto-activation mode:
> [true]
> 2013-05-17 00:33:10,252 INFO  plugin.PluginRepository
> (PluginRepository.java:displayStatus(314)) - Registered Plugins:
> 2013-05-17 00:33:10,252 INFO  plugin.PluginRepository
> (PluginRepository.java:displayStatus(317)) - NONE
> 2013-05-17 00:33:10,252 INFO  plugin.PluginRepository
> (PluginRepository.java:displayStatus(324)) - Registered Extension-Points:
> 2013-05-17 00:33:10,253 INFO  plugin.PluginRepository
> (PluginRepository.java:displayStatus(326)) - NONE
> 2013-05-17 00:33:10,255 WARN  mapred.LocalJobRunner
> (LocalJobRunner.java:run(256)) - job_local_0001
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 5 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 10 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 13 more
> Caused by: java.lang.RuntimeException: x point
> org.apache.nutch.net.URLNormalizer not found.
> at org.apache.nutch.net.URLNormalizers.(URLNormalizers.java:123)
> at org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:74)
> ... 18 more
> 2013-05-17 00:33:11,160 INFO  mapred.JobClient
> (JobClient.java:monitorAndPrintJob(1288)) -  map 0% reduce 0%
> 2013-05-17 00:33:11,163 INFO  mapred.JobClient
> (JobClient.java:monitorAndPrintJob(1343)) - Job complete: job_local_0001
> 2013-05-17 00:33:11,164 INFO  mapred.JobClient (Counters.java:log(514)) -
> Counters: 0
> Exception in thread "main" java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at org.apache.nutch.crawl.Injector.inject(Injector.java:281)
> at org.apache.nutch.crawl.Crawl.run(Crawl.java:132)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>
>
>
> On Fri, May 17, 2013 at 12:37 AM, harsh yadav 
> wrote:
>
> > Hello,
> >
> > I am running nutch 1.6 with hadoop 0.20.2 but not able to crawl in eclips
> > every time getting error:-
> >
> >
>



-- 
*Lewis*

[Nutch-newbie] Installation error

2013-05-18T00:36:21Z

Hi everyone,

This is my first post so apologies if this is not the correct question to ask.

I have followed the wiki tutorial and I am getting the below error. I am running in the local
mode and don't have hadoop installed. Can you please help as I have no clue what's going wrong.

Thanks.
Nishant

The Error:
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /home/local/ANT/nishans/nutch-1.6/apache-nutch-1.6/logs/hadoop.log
(No such file or directory)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:207)
at java.io.FileOutputStream.(FileOutputStream.java:131)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
at org.apache.log4j.LogManager.(LogManager.java:125)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
at org.apache.nutch.crawl.Injector.(Injector.java:53)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].
Injector: starting at 2013-05-17 17:22:22
Injector: crawlDb: Te stCrawl/crawldb
Injector: urlDir: urls/seed.txt
Injector: Converting injected urls to crawl db entries.
Injector: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
at org.apache.nutch.crawl.Injector.inject(Injector.java:281)
at org.apache.nutch.crawl.Injector.run(Injector.java:318)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Injector.main(Injector.java:308)

nishans@ua41f725d6547517ff08c:~/nutch-1.6/apache-nutch-1.6$
clear

nishans@ua41f725d6547517ff08c:~/nutch-1.6/apache-nutch-1.6$
bin/crawl urls/seed.txt TestCrawl http://localhost:8983/solr/ 2
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /home/local/ANT/nishans/nutch-1.6/apache-nutch-1.6/logs/hadoop.log
(No such file or directory)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.(File OutputStream.java:207)
at java.io.FileOutputStream.(FileOutputStream.java:131)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
at org.apache.log4j.LogManager.(LogManager.java:125)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
at org.apache.nutch.crawl.Injector.(Injector.java:53)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].
Injector: starting at 2013-05-17 17:30:07
Injector: crawlDb: TestCrawl/crawldb
Injector: urlDir: urls/seed.txt
Injector: Converting injected urls to crawl db entries.
Injector: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
at org.apache.nutch.crawl.Injector.inject(Injecto r.java:281)
at org.apache.nutch.crawl.Injector.run(Injector.java:318)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Injector.main(Injector.java:308)

Re: Example crawl script Nutch 2.1

2013-05-17T20:40:55Z

Hi Bai Shen,

Thanks for your comments. Can you kindly add those to the relevant jira [0]
so that it gets tracked ?

[0] https://issues.apache.org/jira/browse/NUTCH-1545

Thanks,
Tejas


On Fri, May 17, 2013 at 5:36 AM, Bai Shen  wrote:

> I just tested the GeneratorJob portion and it works fine.  I have two
> comments, though.
>
> 1.  I added braces around the -batchId arg if statement.  I don't like if's
> without them.
> 2.  BatchIds never get cleared.  So if you use the same batchId for
> multiple crawl cycles your urls per batch will continue to grow.  There
> should probably be some sort of note in the help printout.
>
>
>
>
> On Tue, Apr 30, 2013 at 10:37 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi James,
> > Please look for NUTCH-1545 capture batchid...
> > If you could review and use this patch it would be very very helpful.
> > thank you
> > lewis
> >
> > On Tuesday, April 30, 2013, James Ford  wrote:
> > > Thanks for your answer!
> > >
> > > I think I will create my own modified crawlscript then. But I am pretty
> > > confused of how to get a generated batchId? Should I just parse the id
> > from
> > > the output:
> > >
> > > GeneratorJob: generated batch id: 1367327604-149897259
> > >
> > > Or should I get the newly generated batchId from the datastore in my
> > script?
> > > Any best practices?
> > >
> > > Thanks
> > >
> > >
> > >
> > > --
> > > View this message in context:
> >
> >
> http://lucene.472066.n3.nabble.com/Example-crawl-script-Nutch-2-1-tp4059960p4059985.html
> > > Sent from the Nutch - User mailing list archive at Nabble.com.
> > >
> >
> > --
> > *Lewis*
> >
>

Re: error crawling

2013-05-17T20:32:24Z

The exception speaks about the problem:

java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal first
character <46> at 0.
User-space table names can only start with 'word characters': i.e.
[a-zA-Z_0-9]: ./crawl/_webpage

The crawlId passed must follow the regex [a-zA-Z_0-9]. The one you passed
has dot and slash.
$ ./bin/nutch inject urls/ -crawlId ./crawl/

Try this:
$ ./bin/nutch inject urls/ -crawlId crawl



On Fri, May 17, 2013 at 12:47 PM,  wrote:

> What if you do bin/nutch inject urls/ ?
>
>
>
>
>
>
> -----Original Message-----
> From: Christopher Gross 
> To: user 
> Sent: Fri, May 17, 2013 11:26 am
> Subject: error crawling
>
>
> I'm having trouble getting my nutch working.  I had it on another server
> and it was working fine.  I migrated it to a new server, and I've been
> getting nothing but problems.  My old script wasn't working right (getting
> a lot of "skipping" on the parser saying that the crawl id was null [a
> separate point of frustration]), so now I'm trying the 'newer' crawl
> script.  This one is worse, since I can't even get the inject to work.
>
> urls contains a "seed.txt" file that worked previously and contains a bunch
> of urls.  crawl is empty.
>
> from my $NUTCH_HOME directory:
>
> $ ./bin/nutch inject urls/ -crawlId ./crawl/
> InjectorJob: starting
> InjectorJob: urlDir: urls
> InjectorJob: org.apache.gora.util.GoraException:
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal
> first character <46> at 0. User-space table names can only start with 'word
> characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
>         at
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
>         at
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
>         at
> org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
>         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
>         at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
>         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
> Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException:
> Illegal first character <46> at 0. User-space table names can only start
> with 'word characters': i.e. [a-zA-Z_0-9]: ./crawl/_webpage
>         at
> org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:125)
>         at
>
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
>         at
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
>         ... 7 more
> Caused by: java.lang.IllegalArgumentException: Illegal first character <46>
> at 0. User-space table names can only start with 'word characters': i.e.
> [a-zA-Z_0-9]: ./crawl/_webpage
>         at
>
> org.apache.hadoop.hbase.HTableDescriptor.isLegalTableName(HTableDescriptor.java:280)
>         at
> org.apache.hadoop.hbase.HTableDescriptor.(HTableDescriptor.java:172)
>         at
> org.apache.hadoop.hbase.HTableDescriptor.(HTableDescriptor.java:158)
>         at
>
> org.apache.gora.hbase.store.HBaseMapping$HBaseMappingBuilder.build(HBaseMapping.java:171)
>         at
> org.apache.gora.hbase.store.HBaseStore.readMapping(HBaseStore.java:592)
>         at
> org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:111)
>         ... 9 more
>
> Where is the "_webpage" coming from?  Am I just missing something?
>
> Any help/ideas/references would be appreciated.
>
> Thanks!
>
> -- Chris
>
>
>

Re: crawl stopping randomly before the specified depth

2013-05-17T20:24:01Z

On Fri, May 17, 2013 at 1:39 AM, Sourajit Basak wrote:

> In order to control the number of part files generated, we made a minor
> change to handle 'numFetchers' argument in the one-step crawl command.
> Relevant portions of the code are given below.
>
> Although I set the depth to 3, the crawl stops at the first or second
> pass. Its random in nature.
>
> From the logs, I find
> *Generator: 0 records selected for fetching, exiting ...*
>
> I executed crawl with your code change on the latest codebase (trunk) and
didn't see this happening.


>     ....................
>     int depth = 5;
>     int numFetchers = -1;
>     long topN = Long.MAX_VALUE;
>     .............
>
>     for (int i = 0; i < args.length; i++) {
>       if ("-dir".equals(args[i])) {
>         dir = new Path(args[i+1]);
>         i++;
>       } else if ("-threads".equals(args[i])) {
>         threads = Integer.parseInt(args[i+1]);
>         i++;
>       } else if ("-depth".equals(args[i])) {
>         depth = Integer.parseInt(args[i+1]);
>         i++;
>       } else if ("-topN".equals(args[i])) {
>           topN = Integer.parseInt(args[i+1]);
>           i++;
>       } else if ("-solr".equals(args[i])) {
>         solrUrl = args[i + 1];
>         i++;
>       }* else if ("-numFetchers".equals(args[i])) {*
> *       numFetchers = Integer.parseInt(args[i+1]);*
> *       i++;*
> *      }* else if (args[i] != null) {
>         rootUrlDir = new Path(args[i]);
>       }
>     }
>
>     JobConf job = new NutchJob(getConf());
>
>    ..........................
>    ...................
>    .............
>
>     // initialize crawlDb
>     injector.inject(crawlDb, rootUrlDir);
>     int i;
>     for (i = 0; i < depth; i++) {             // generate new segment
> *      Path[] segs = generator.generate(crawlDb, segments, numFetchers,
> topN, System*
> *          .currentTimeMillis());*
>       if (segs == null) {
>         LOG.info("Stopping at depth=" + i + " - no more URLs to fetch.");
>         break;
>       }
>       fetcher.fetch(segs[0], threads);  // fetch it
>       if (!Fetcher.isParsing(job)) {
>         parseSegment.parse(segs[0]);    // parse it, if needed
>       }
>       crawlDbTool.update(crawlDb, segs, true, true); // update crawldb
>     }
>
> The code change is correct.


> From Generator source, I think that the 'status' is empty, because the
> flow doesn't reach partitionSegments(....
>
>     FileStatus[] status = fs.listStatus(tempDir);
>     try {
>       for (FileStatus stat : status) {
>         Path subfetchlist = stat.getPath();
>         if (!subfetchlist.getName().startsWith("fetchlist-")) continue;
>         // start a new partition job for this segment
>         Path newSeg = partitionSegment(fs, segments, subfetchlist,
> numLists);
>         generatedSegments.add(newSeg);
>       }
>
> There is a catch block below this which has a different message.

    } catch (Exception e) {
      LOG.warn("Generator: exception while partitioning segments, exiting
...");
      fs.delete(tempDir, true);
      return null;
    }

Had status been null, it would have not said "0 records ...." and given the
message above. So, in my opinion, 'status' was not null.
In case its empty, that can be verified by putting a log message inside the
for loop. When I did that, I could see that some valid path was logged.

What can be the cause for this behavior ?
>
> Can you set topN to a bigger value ? (I believe that you have already
verified that the urls are not getting filtered due to any filters). As you
said that it was kinda sporadic, I would retry it when I get free time.

Thanks,
> Sourajit
>

Status of Elasticsearch indexer?

2013-05-17T19:55:31Z

Hello everyone,

I've been eagerly awaiting some of the functionality slated for 2.x, especially around your
work integrating with Elasticsearch. If possible, could you give any additional status on
pluggable indexing (NUTCH-1568) and the nutch-elasticsearch-indexer (NUTCH-1527)?

It's been a wonderful experience diving into Nutch for the last month and watching you guys
do pretty awesome work. Now that I can finally say I no longer feel completely overwhelmed,
I'd like to throw in my support for these items. Further, if there is work that still needs
to be done, I might like to try helping out myself :)

Thanks!
Chris