Mailing-List: contact nutch-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: nutch-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of parvez@gmail.com designates
 209.85.211.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=elab7LurDqOf87+U1ZoqSe+F8Su0jhn5wi+YdcfF8tvOcO9ymZnC4v7cYajfsGU0g9
         qYanu2t03BMCis4vgfPz8Zvco8Ayqcv0hmtsG7l3Dqa4JBAXslTPj2/S2pfWkHui+zDg
         6wXCbQdPGch8TPZ3b4AP7TC6J8/AlIWCQOAeU=
MIME-Version: 1.0
In-Reply-To: 
 <53724.203.35.135.133.1252475482.squirrel@www.butterflycluster.com>
References: <f86738db0909081402y48d3df46h4e9c206a3859c487@mail.gmail.com>
	<f86738db0909082209gbd42bft3b39c385f63b3dc1@mail.gmail.com>
	<60682.203.35.135.133.1252473338.squirrel@www.butterflycluster.com>
	<f86738db0909082237n211e5347t71cafd7d9080b98d@mail.gmail.com>
	<53724.203.35.135.133.1252475482.squirrel@www.butterflycluster.com>
From: Mohamed Parvez <parvez@gmail.com>
Date: Wed, 9 Sep 2009 01:12:35 -0500
Message-ID: <f86738db0909082312m45caf692meafeef1204560e5d@mail.gmail.com>
Subject: Re: How to crawl pagination in sequence
To: nutch-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=001636ed722944af4504731ef990

--001636ed722944af4504731ef990
Content-Type: text/plain; charset=ISO-8859-1

I tried running with one thread, still same results.
Any hint on how do we make Nutch aware of session cookies

---
Thanks/Regards,
Parvez


On Wed, Sep 9, 2009 at 12:51 AM, <fadzi@butterflycluster.net> wrote:

> how many threads are you running at?
>
> nutch doesnt know about sessions;
>
> you might have to do something like fetching one thread at a time but
> thats slow.
>
> or maybe make nutch aware of session cookies.
>
> > I am crawling at depth 40 as there are 40 pages in the pagination.
> >
> > It works fine till the first 6 pages and after that it goes to the 7th
> > page,
> > but looks like its different session and hence the pagination wont work.
> >
> > I mean if you you directly hit page 7, using the URL, the pagination wont
> > work and will return empty set.
> >
> > But if you go in the sequence in the same session the pagination works.
> >
> >
> > ---
> > Thanks/Regards,
> > Parvez
> >
> >
> > On Wed, Sep 9, 2009 at 12:15 AM, <fadzi@butterflycluster.net> wrote:
> >
> >> could be tricky from what i've seen;
> >>
> >> theres limits on how many times you can hit one host/ip;
> >>
> >> also what depth you are crawling at may come to play in your case (which
> >> is probably what you want to look at in this case).
> >>
> >>
> >> > Any hint to increase the session time of the Nutch crawl thread.
> >> > I tried crawling with one thread, still no luck.
> >> >
> >> > ----
> >> > Thanks/Regards,
> >> > Parvez
> >> >
> >> >
> >> >
> >> > On Tue, Sep 8, 2009 at 4:02 PM, Mohamed Parvez <parvez@gmail.com>
> >> wrote:
> >> >
> >> >> I have a paginated pages, which will only work if its crawled in a
> >> given
> >> >> sequence, and in the same session.
> >> >>
> >> >> For example first URL is
> >> >>
> >> >> http://www.myhost.com/?page_number=1
> >> >> http://www.myhost.com/?page_number=2
> >> >> http://www.myhost.com/?page_number=3
> >> >>
> >> >> The first page has link to second page.
> >> >> Second page has link to first and second page.
> >> >> Third page has link to third and second page.
> >> >> So On...
> >> >>
> >> >> Nutch is able to crawl the the first 6 pages, but beyond that it is
> >> not
> >> >> able to crawl or is getting empty result.
> >> >>
> >> >> If I manually click through the pagination, in a browser, I can reach
> >> >> till
> >> >> the end with no problem.
> >> >>
> >> >> Is the Nutch Crawl Session timing out? How do we increase it.
> >> >>
> >> >> I tried crawling with on thread but still same result.
> >> >>
> >> >> Any suggestion ?
> >> >>
> >> >> ---
> >> >> Thanks/Regards,
> >> >> Parvez
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >
>
>
>

--001636ed722944af4504731ef990--