Mailing-List: contact user-help@nutch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@nutch.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Markus Jelsma <markus.jelsma@openindex.io>
Reply-To: markus.jelsma@openindex.io
Organization: Openindex
To: user@nutch.apache.org
Subject: Re: generate/update times and crawldb size
Date: Tue, 6 Dec 2011 12:11:05 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-35-generic; KDE/4.4.5; x86_64; ; )
References: <20111206103350.30210@gmx.com>
In-Reply-To: <20111206103350.30210@gmx.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-15"
Content-Transfer-Encoding: 7bit
Message-Id: <201112061211.05658.markus.jelsma@openindex.io>

Is this on Hadoop? Are the update and generate jobs doing filtering and 
normalizing? That's usually the problem.

On Tuesday 06 December 2011 11:33:49 Danicela nutch wrote:
> Hi,
> 
>  I have the impression that something is going wrong in my nutch cycle.
> 
>  4 millions pages
>  5.7 Gb crawldb
> 
>  One generate lasts 4:46 and gets 15 minutes more each segment (90 000
> pages produced for each segment) One update lasts 7h36 and gets 45 minutes
> more each segment.
> 
>  Are these times normal ?
> 
>  If not, what can I do to reduce these times ?
> 
>  Thanks.

-- 
Markus Jelsma - CTO - Openindex