Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: error (athena.apache.org: local policy)
Message-ID: <4EA92628.9070904@sendmail.cz>
Date: Thu, 27 Oct 2011 11:36:40 +0200
From: Radim Kolar <hsn@sendmail.cz>
User-Agent: Mozilla/5.0 (Windows NT 5.1;
 rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1
MIME-Version: 1.0
To: mapreduce-user@hadoop.apache.org
Subject: Re: unsort algorithmus in map/reduce
References: <4EA69361.4040401@sendmail.cz>
 <CADoiZqpohFY=g1rrGJOTub5sSzv-a_aTiUMZrGBPhybJVARU7Q@mail.gmail.com>
 <4EA6D72A.10906@sendmail.cz>
 <CAFQoU9Ffceu_ZX1dmP8W-HdWQRkBV+9D+NTbO8b7vis2ttLB9g@mail.gmail.com>
In-Reply-To: 
 <CAFQoU9Ffceu_ZX1dmP8W-HdWQRkBV+9D+NTbO8b7vis2ttLB9g@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

 > If on the other hand, you want to guarantee that you don't swamp the 
servers on each domain and you are trying to throttle
 > your fetchers, then you want to do something like re-write the urls 
to be backwards:
>
> com.test.www/http/page1.html
> com.test.www/http/page2.html
> com.test.www/http/page3.html
> com.test2.www/http/page1.html
> com.test2.www/http/page2.html
I didnt get why they have to be backwards because if we are interested 
in URL queue  distance from same origin server then distance is same.

or you wanted to reverse them like

page1.html/com.test.www/http
page1.html/com.test2.www/http

then i am not sure if this ordering is better then pure random or md5.

> and use a total ordering of the sort. (You'll need to sample the data 
> to pick the cut points.) That will limit each site to one or 
> occasionally two mappers and thus the maximum number of concurrent 
> fetchers will be the number of threads in each mapper.
I need to spread site between as much mappers as possible because there 
is crawl delay between requests per site.