Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B8705DF91 for ; Wed, 26 Dec 2012 10:44:19 +0000 (UTC) Received: (qmail 2249 invoked by uid 500); 26 Dec 2012 10:44:14 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 2119 invoked by uid 500); 26 Dec 2012 10:44:14 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 2104 invoked by uid 99); 26 Dec 2012 10:44:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Dec 2012 10:44:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of linlma@gmail.com designates 209.85.220.182 as permitted sender) Received: from [209.85.220.182] (HELO mail-vc0-f182.google.com) (209.85.220.182) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Dec 2012 10:44:07 +0000 Received: by mail-vc0-f182.google.com with SMTP id fy27so8661722vcb.41 for ; Wed, 26 Dec 2012 02:43:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=fMrRbXPFzWWJeQJmwwkzrPFak/cI3CD1dA06w3L6P2s=; b=V0ZCjMcLirH7NW3Zvutc762h1kwMT14xIcSaXzvhOomoGaU9nb5BYcx6O7Ihaf5e0J DMbmleHNC1vf7cAmr6Z/u9RlVSb762o/Gnsdp1O04OCnaAJdi9B9mS/tN4+4Z04PNlS5 a2C6Grs/yfOxON6rPe24Y+qYmYguwK3272ARE1kYhXFuQgKG4SZwb6ZemJIFaXPO7PgF bVIZFbVwRFL+pnKCRhIGsVA0XLV4GZwxPL44ba+rqePEtHll3E1AzYw8BfUzM0TURg10 eLFQDkp6AEIkuxBZjqAY/TGNvX801XNzsanbHBfwuVXx+agInCPZRg/MFfgs6Dt8EyEj RQDQ== MIME-Version: 1.0 Received: by 10.220.228.1 with SMTP id jc1mr40211675vcb.74.1356518627032; Wed, 26 Dec 2012 02:43:47 -0800 (PST) Received: by 10.58.68.135 with HTTP; Wed, 26 Dec 2012 02:43:46 -0800 (PST) In-Reply-To: References: <491FA550-FC92-4280-8FB5-186E5F7A4743@123.org> Date: Wed, 26 Dec 2012 18:43:46 +0800 Message-ID: Subject: Re: distributed cache From: Lin Ma To: Harsh J , user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae9cdc6bde53f8b04d1bf1a30 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9cdc6bde53f8b04d1bf1a30 Content-Type: text/plain; charset=ISO-8859-1 Thanks Harsh, multiple concurrent read is generally faster or? regards, Lin On Wed, Dec 26, 2012 at 6:21 PM, Harsh J wrote: > There is no limitation in HDFS that limits reads of a block to a > single client at a time (no reason to do so) - so downloads can be as > concurrent as possible. > > On Wed, Dec 26, 2012 at 3:41 PM, Lin Ma wrote: > > Thanks Harsh, > > > > Supposing DistributedCache is uploaded by client, for each replica, in > > Hadoop design, it could only serve one download session (download from a > > mapper or a reducer which requires the DistributedCache) at a time until > > DistributedCache file download is completed, or it could serve multiple > > concurrent parallel download session (download from multiple mappers or > > reducers which requires the DistributedCache). > > > > regards, > > Lin > > > > > > On Wed, Dec 26, 2012 at 4:51 PM, Harsh J wrote: > >> > >> Hi Lin, > >> > >> DistributedCache files are stored onto the HDFS by the client first. > >> The TaskTrackers download and localize it. Therefore, as with any > >> other file on HDFS, "downloads" can be efficiently parallel with > >> higher replicas. > >> > >> The point of having higher replication for these files is also tied to > >> the concept of racks in a cluster - you would want more replicas > >> spread across racks such that on task bootup the downloads happen with > >> rack locality. > >> > >> On Sat, Dec 22, 2012 at 6:54 PM, Lin Ma wrote: > >> > Hi Kai, > >> > > >> > Smart answer! :-) > >> > > >> > The assumption you have is one distributed cache replica could only > >> > serve > >> > one download session for tasktracker node (this is why you get > >> > concurrency > >> > n/r). The question is, why one distributed cache replica cannot serve > >> > multiple concurrent download session? For example, supposing a > >> > tasktracker > >> > use elapsed time t to download a file from a specific distributed > cache > >> > replica, it is possible for 2 tasktrackers to download from the > specific > >> > distributed cache replica in parallel using elapsed time t as well, or > >> > 1.5 > >> > t, which is faster than sequential download time 2t you mentioned > >> > before? > >> > "In total, r+n/r concurrent operations. If you optimize r depending on > >> > n, > >> > SRQT(n) is the optimal replication level." -- how do you get SRQT(n) > for > >> > minimize r+n/r? Appreciate if you could point me to more details. > >> > > >> > regards, > >> > Lin > >> > > >> > > >> > On Sat, Dec 22, 2012 at 8:51 PM, Kai Voigt wrote: > >> >> > >> >> Hi, > >> >> > >> >> simple math. Assuming you have n TaskTrackers in your cluster that > will > >> >> need to access the files in the distributed cache. And r is the > >> >> replication > >> >> level of those files. > >> >> > >> >> Copying the files into HDFS requires r copy operations over the > >> >> network. > >> >> The n TaskTrackers need to get their local copies from HDFS, so the n > >> >> TaskTrackers copy from r DataNodes, so n/r concurrent operation. In > >> >> total, > >> >> r+n/r concurrent operations. If you optimize r depending on n, > SRQT(n) > >> >> is > >> >> the optimal replication level. So 10 is a reasonable default setting > >> >> for > >> >> most clusters that are not 500+ nodes big. > >> >> > >> >> Kai > >> >> > >> >> Am 22.12.2012 um 13:46 schrieb Lin Ma : > >> >> > >> >> Thanks Kai, using higher replication count for the purpose of? > >> >> > >> >> regards, > >> >> Lin > >> >> > >> >> On Sat, Dec 22, 2012 at 8:44 PM, Kai Voigt wrote: > >> >>> > >> >>> Hi, > >> >>> > >> >>> Am 22.12.2012 um 13:03 schrieb Lin Ma : > >> >>> > >> >>> > I want to confirm when on each task node either mapper or reducer > >> >>> > access distributed cache file, it resides on disk, not resides in > >> >>> > memory. > >> >>> > Just want to make sure distributed cache file does not fully > loaded > >> >>> > into > >> >>> > memory which compete memory consumption with mapper/reducer tasks. > >> >>> > Is that > >> >>> > correct? > >> >>> > >> >>> > >> >>> Yes, you are correct. The JobTracker will put files for the > >> >>> distributed > >> >>> cache into HDFS with a higher replication count (10 by default). > >> >>> Whenever a > >> >>> TaskTracker needs those files for a task it is launching locally, it > >> >>> will > >> >>> fetch a copy to its local disk. So it won't need to do this again > for > >> >>> future > >> >>> tasks on this node. After a job is done, all local copies and the > HDFS > >> >>> copies of files in the distributed cache are cleaned up. > >> >>> > >> >>> Kai > >> >>> > >> >>> -- > >> >>> Kai Voigt > >> >>> k@123.org > >> >>> > >> >>> > >> >>> > >> >>> > >> >> > >> >> > >> >> -- > >> >> Kai Voigt > >> >> k@123.org > >> >> > >> >> > >> >> > >> >> > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J > --14dae9cdc6bde53f8b04d1bf1a30 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Harsh, multiple concurrent read is generally faster or?

regar= ds,
Lin

On Wed, Dec 26, 2012 at 6:21 P= M, Harsh J <harsh@cloudera.com> wrote:
There is no limitation in HDFS that limits r= eads of a block to a
single client at a time (no reason to do so) - so downloads can be as
concurrent as possible.

On Wed, Dec 26, 2012 at 3:41 PM, Lin Ma <linlma@gmail.com> wrote:
> Thanks Harsh,
>
> Supposing DistributedCache is uploaded by client, for each replica, in=
> Hadoop design, it could only serve one download session (download from= a
> mapper or a reducer which requires the DistributedCache) at a time unt= il
> DistributedCache file download is completed, or it could serve multipl= e
> concurrent parallel download session (download from multiple mappers o= r
> reducers which requires the DistributedCache).
>
> regards,
> Lin
>
>
> On Wed, Dec 26, 2012 at 4:51 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Hi Lin,
>>
>> DistributedCache files are stored onto the HDFS by the client firs= t.
>> The TaskTrackers download and localize it. Therefore, as with any<= br> >> other file on HDFS, "downloads" can be efficiently paral= lel with
>> higher replicas.
>>
>> The point of having higher replication for these files is also tie= d to
>> the concept of racks in a cluster - you would want more replicas >> spread across racks such that on task bootup the downloads happen = with
>> rack locality.
>>
>> On Sat, Dec 22, 2012 at 6:54 PM, Lin Ma <linlma@gmail.com> wrote:
>> > Hi Kai,
>> >
>> > Smart answer! :-)
>> >
>> > The assumption you have is one distributed cache replica coul= d only
>> > serve
>> > one download session for tasktracker node (this is why you ge= t
>> > concurrency
>> > n/r). The question is, why one distributed cache replica cann= ot serve
>> > multiple concurrent download session? For example, supposing = a
>> > tasktracker
>> > use elapsed time t to download a file from a specific distrib= uted cache
>> > replica, it is possible for 2 tasktrackers to download from t= he specific
>> > distributed cache replica in parallel using elapsed time t as= well, or
>> > 1.5
>> > t, which is faster than sequential download time 2t you menti= oned
>> > before?
>> > "In total, r+n/r concurrent operations. If you optimize = r depending on
>> > n,
>> > SRQT(n) is the optimal replication level." -- how do you= get SRQT(n) for
>> > minimize r+n/r? Appreciate if you could point me to more deta= ils.
>> >
>> > regards,
>> > Lin
>> >
>> >
>> > On Sat, Dec 22, 2012 at 8:51 PM, Kai Voigt <k@123.org> wrote:
>> >>
>> >> Hi,
>> >>
>> >> simple math. Assuming you have n TaskTrackers in your clu= ster that will
>> >> need to access the files in the distributed cache. And r = is the
>> >> replication
>> >> level of those files.
>> >>
>> >> Copying the files into HDFS requires r copy operations ov= er the
>> >> network.
>> >> The n TaskTrackers need to get their local copies from HD= FS, so the n
>> >> TaskTrackers copy from r DataNodes, so n/r concurrent ope= ration. In
>> >> total,
>> >> r+n/r concurrent operations. If you optimize r depending = on n, SRQT(n)
>> >> is
>> >> the optimal replication level. So 10 is a reasonable defa= ult setting
>> >> for
>> >> most clusters that are not 500+ nodes big.
>> >>
>> >> Kai
>> >>
>> >> Am 22.12.2012 um 13:46 schrieb Lin Ma <linlma@gmail.com>:
>> >>
>> >> Thanks Kai, using higher replication count for the purpos= e of?
>> >>
>> >> regards,
>> >> Lin
>> >>
>> >> On Sat, Dec 22, 2012 at 8:44 PM, Kai Voigt <k@123.org> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> Am 22.12.2012 um 13:03 schrieb Lin Ma <linlma@gmail.com>:
>> >>>
>> >>> > I want to confirm when on each task node either = mapper or reducer
>> >>> > access distributed cache file, it resides on dis= k, not resides in
>> >>> > memory.
>> >>> > Just want to make sure distributed cache file do= es not fully loaded
>> >>> > into
>> >>> > memory which compete memory consumption with map= per/reducer tasks.
>> >>> > Is that
>> >>> > correct?
>> >>>
>> >>>
>> >>> Yes, you are correct. The JobTracker will put files f= or the
>> >>> distributed
>> >>> cache into HDFS with a higher replication count (10 b= y default).
>> >>> Whenever a
>> >>> TaskTracker needs those files for a task it is launch= ing locally, it
>> >>> will
>> >>> fetch a copy to its local disk. So it won't need = to do this again for
>> >>> future
>> >>> tasks on this node. After a job is done, all local co= pies and the HDFS
>> >>> copies of files in the distributed cache are cleaned = up.
>> >>>
>> >>> Kai
>> >>>
>> >>> --
>> >>> Kai Voigt
>> >>> k@123.org
>> >>>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Kai Voigt
>> >> k@123.org
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



--
Harsh J

--14dae9cdc6bde53f8b04d1bf1a30--