Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id AD12E200AC5 for ; Sun, 5 Jun 2016 17:05:15 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id AA57E160A28; Sun, 5 Jun 2016 15:05:15 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A5788160A25 for ; Sun, 5 Jun 2016 17:05:14 +0200 (CEST) Received: (qmail 28942 invoked by uid 500); 5 Jun 2016 15:05:12 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 28931 invoked by uid 99); 5 Jun 2016 15:05:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2016 15:05:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id EBAF9C1CFA for ; Sun, 5 Jun 2016 15:05:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id aOudzk1Ttmxb for ; Sun, 5 Jun 2016 15:05:09 +0000 (UTC) Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E48C75F240 for ; Sun, 5 Jun 2016 15:05:08 +0000 (UTC) Received: by mail-wm0-f42.google.com with SMTP id z87so48392772wmh.0 for ; Sun, 05 Jun 2016 08:05:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=5ypTkXMNinNsuQzfRj4NIpYxs6nOpNamXv0XhkZwFnA=; b=iYGLoDR59yLYrCA0e///kuiH+8Vk5iCl3gKo6rLtER1vpCqGYmJZISM+0xjaVE4KEl +WO5WumocBNMm5L+RUDRDK9FibETkOfuHWn5XROfIR/WAo2lE6aXvUkOY+VA8A1V9kZf Pm1fp9BBmcpwBHj+CfAY8kKkZcy0AY6vQxmVRgn4IUIVJD9f3SLfqFSkGBEcgABOUzfb OKEp7/Gl/0y+/0ftb5O+kIV15qo9ESLzLKC2c8sxRbTBr1kXjsVUNpzmZS11zzU+8R4a Wk4fV8NQDE+x9osyoeTUcsN7DCu6phzlF3yLCbrkPs3tYoYegSS004Cbzcuc42jDezeQ O/9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=5ypTkXMNinNsuQzfRj4NIpYxs6nOpNamXv0XhkZwFnA=; b=ZFFmObGnex5ZiUcMu0dAEjnesfDo4iTaBacnihyjinBtrELuMDDpQUO8OXSxedTR2f taVzqYyCn3Zbc4OPBTXJGxwSNI2MtuSXkYFcCStmsWH6KjfWhsAhF+HgiKZTT5G/r1jO drzry8ZrsfRj0+lKuA4HHrhxZeT6LIRnt0SSA610ZEy/mNAIlQDXdI8dBMujFRnkkulC FRx8U6lJsqUFaDquKompeQKZL3Z8B/TbAUk4stwjVnDu8zVZ5l6PWA4cv2t80Ts3qGa5 MKhiTvrLHcu+6r8UvxcrzGcnFfgUR+WsJ3J2SuE63VoSMXPF2O/iFjm8URWy5SsEpQYM I5tQ== X-Gm-Message-State: ALyK8tI12K7KKjF/ImO/Ewqi9E3QsK+g+eEde8Bv5X2n8jq796lrnraXtTDJW5vV8jjxzISnM1q1zFwazb/g5Q== MIME-Version: 1.0 X-Received: by 10.28.184.83 with SMTP id i80mr7998702wmf.66.1465139108536; Sun, 05 Jun 2016 08:05:08 -0700 (PDT) Received: by 10.28.111.202 with HTTP; Sun, 5 Jun 2016 08:05:08 -0700 (PDT) Received: by 10.28.111.202 with HTTP; Sun, 5 Jun 2016 08:05:08 -0700 (PDT) In-Reply-To: References: <0D85B8BC-5D45-474E-A7B7-C21AA2F816CF@gmail.com> Date: Sun, 5 Jun 2016 18:05:08 +0300 Message-ID: Subject: Re: HDFS2 vs MaprFS From: Hayati Gonultas To: Ascot Moss Cc: daemeon reiydelle , user@hadoop.apache.org, Gavin Yue Content-Type: multipart/alternative; boundary=001a114b30a21c6bb405348948d9 archived-at: Sun, 05 Jun 2016 15:05:15 -0000 --001a114b30a21c6bb405348948d9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Another correction about the terminology needs to be made. i said 1gb =3D 1 million blocks. Pay attention to term block. it is not fil= e. A file may contain more than one block. Default block size 64mb so 640 mb file will hold 10 blocks. Each file has its name ,permissions, path, creation date and etc. These metadata is held in memory for all files but not blocks. So it is good to have files with many blocks. So by the terms of file count, the worst case scenerio is each file only contained in one block. Resulting my 1gb =3D 1million files. Typically file= s have many blocks and this count may increase. 5 Haz 2016 17:33 tarihinde "Hayati Gonultas" yazd=C4=B1: > it is written 128 000 000 million in my previous post. it was incorrect > (million million) > > what i mean is 128 million. > > 1gb raughly 1 million. > 5 Haz 2016 16:58 tarihinde "Ascot Moss" yazd=C4=B1= : > >> HDFS2 "Limit to 50-200 million files", is it really true like what MapR >> says? >> >> On Sun, Jun 5, 2016 at 7:55 PM, Hayati Gonultas < >> hayati.gonultas@gmail.com> wrote: >> >>> I forgot to mention about file system limit. >>> >>> Yes HDFS has limit, because for the performance considirations HDFS >>> filesystem is read from disk to RAM and rest of the work is done with R= AM. >>> So RAM should be big enough to fit the filesystem image. But HDFS has >>> configuration options like har files (Hadoop Archive) to defeat these >>> limitations. >>> >>> On Sun, Jun 5, 2016 at 11:14 AM, Ascot Moss >>> wrote: >>> >>>> Will the the common pool of datanodes and namenode federation be a mor= e >>>> effective alternative in HDFS2 than multiple clusters? >>>> >>>> On Sun, Jun 5, 2016 at 12:19 PM, daemeon reiydelle >>>> wrote: >>>> >>>>> There are indeed many tuning points here. If the name nodes and >>>>> journal nodes can be larger, perhaps even bonding multiple 10gbyte ni= cs, >>>>> one can easily scale. I did have one client where the file counts for= ced >>>>> multiple clusters. But we were able to differentiate by airframe type= s ... >>>>> eg fixed wing in one, rotary subsonic in another, etc. >>>>> >>>>> sent from my mobile >>>>> Daemeon C.M. Reiydelle >>>>> USA 415.501.0198 >>>>> London +44.0.20.8144.9872 >>>>> On Jun 4, 2016 2:23 PM, "Gavin Yue" wrote: >>>>> >>>>>> Here is what I found on Horton website. >>>>>> >>>>>> >>>>>> *Namespace scalability* >>>>>> >>>>>> While HDFS cluster storage scales horizontally with the addition of >>>>>> datanodes, the namespace does not. Currently the namespace can only = be >>>>>> vertically scaled on a single namenode. The namenode stores the ent= ire >>>>>> file system metadata in memory. This limits the number of blocks, fi= les, >>>>>> and directories supported on the file system to what can be accommod= ated in >>>>>> the memory of a single namenode. A typical large deployment at Yahoo= ! >>>>>> includes an HDFS cluster with 2700-4200 datanodes with 180 million >>>>>> files and blocks, and address ~25 PB of storage. At Facebook, HDFS = has >>>>>> around 2600 nodes, 300 million files and blocks, addressing up to 60= PB of >>>>>> storage. While these are very large systems and good enough for majo= rity of >>>>>> Hadoop users, a few deployments that might want to grow even larger = could >>>>>> find the namespace scalability limiting. >>>>>> >>>>>> >>>>>> >>>>>> On Jun 4, 2016, at 04:43, Ascot Moss wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I read some (old?) articles from Internet about Mapr-FS vs HDFS. >>>>>> >>>>>> https://www.mapr.com/products/m5-features/no-namenode-architecture >>>>>> >>>>>> It states that HDFS Federation has >>>>>> >>>>>> a) "Multiple Single Points of Failure", is it really true? >>>>>> Why MapR uses HDFS but not HDFS2 in its comparison as this would lea= d >>>>>> to an unfair comparison (or even misleading comparison)? (HDFS was = from >>>>>> Hadoop 1.x, the old generation) HDFS2 is available since 2013-10-15,= there >>>>>> is no any Single Points of Failure in HDFS2. >>>>>> >>>>>> b) "Limit to 50-200 million files", is it really true? >>>>>> I have seen so many real world Hadoop Clusters with over 10PB data, >>>>>> some even with 150PB data. If "Limit to 50 -200 millions files" wer= e true >>>>>> in HDFS2, why are there so many production Hadoop clusters in real w= orld? >>>>>> how can they mange well the issue of "Limit to 50-200 million files= "? For >>>>>> instances, the Facebook's "Like" implementation runs on HBase at We= b >>>>>> Scale, I can image HBase generates huge number of files in Facbook's= Hadoop >>>>>> cluster, the number of files in Facebook's Hadoop cluster should be = much >>>>>> much bigger than 50-200 million. >>>>>> >>>>>> From my point of view, in contrast, MaprFS should have true >>>>>> limitation up to 1T files while HDFS2 can handle true unlimited file= s, >>>>>> please do correct me if I am wrong. >>>>>> >>>>>> c) "Performance Bottleneck", again, is it really true? >>>>>> MaprFS does not have namenode in order to gain file system >>>>>> performance. If without Namenode, MaprFS would lose Data Locality wh= ich is >>>>>> one of the beauties of Hadoop If Data Locality is no longer availab= le, any >>>>>> big data application running on MaprFS might gain some file system >>>>>> performance but it would totally lose the true gain of performance f= rom >>>>>> Data Locality provided by Hadoop's namenode (gain small lose big) >>>>>> >>>>>> d) "Commercial NAS required" >>>>>> Is there any wiki/blog/discussion about Commercial NAS on Hadoop >>>>>> Federation? >>>>>> >>>>>> regards >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>> >>> >>> -- >>> Hayati Gonultas >>> >> >> --001a114b30a21c6bb405348948d9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Another correction about the terminology needs to be made.

i said 1gb =3D 1 million blocks. Pay attention to term block= . it is not file. A file may contain more than one block. Default block siz= e 64mb so 640 mb file will hold 10 blocks. Each file has its name ,permissi= ons, path, creation date and etc. These metadata is held in memory for all = files but not blocks. So it is good to have files with many blocks.

So by the terms of file count, the worst case scenerio is ea= ch file only contained in one block. Resulting my 1gb =3D 1million files. T= ypically files have many blocks and this count may increase.

5 Haz 2016 17:33 tarihinde "Hayati Gonultas= " <hayati.gonultas@gma= il.com> yazd=C4=B1:

it is written 128 000 000 million in my previous post= . it was incorrect (million million)

what i mean is 128 million.

1gb raughly 1 million.

5 Haz 2016 16:58 tarihinde "Ascot Moss"= ; <ascot.moss@= gmail.com> yazd=C4=B1:
HDFS2 "Lim= it to 50-200 million files", is it really true like what MapR says?=C2= =A0

On Sun, Jun 5, 2016 at 7:55 PM, Hayati Gonultas <hayati.gonulta= s@gmail.com> wrote:
I forgot to mention about file system limit.

Ye= s HDFS has limit, because for the performance considirations HDFS filesyste= m is read from disk to RAM and rest of the work is done with RAM. So RAM sh= ould be big enough to fit the filesystem image. But HDFS has configuration = options like har files (Hadoop Archive) to defeat these limitations.

On Sun, = Jun 5, 2016 at 11:14 AM, Ascot Moss <ascot.moss@gmail.com> wrote:
Will the the common pool of datanodes and namenode federation be a more e= ffective alternative in HDFS2 =C2=A0than multiple clusters?
=

On Sun, Jun 5, 20= 16 at 12:19 PM, daemeon reiydelle <daemeonr@gmail.com> wrot= e:

There are indeed many t= uning points here. If the name nodes and journal nodes can be larger, perha= ps even bonding multiple 10gbyte nics, one can easily scale. I did have one= client where the file counts forced multiple clusters. But we were able to= differentiate by airframe types ... eg fixed wing in one, rotary subsonic = in another, etc.

sent from my mobile
Daemeon C.M. Reiydelle
USA 4= 15.501.0198
London +44.0.20.8144.9872

On Jun 4, 2016 2:23 PM, "Gavin Yue" &l= t;yue.yuanyuan@= gmail.com> wrote:
Here is what I found on Horton websi= te. =C2=A0


Namespace scalability

While HDFS cluster storage scales horizo= ntally with the addition of datanodes, the namespace does not. Currently th= e namespace can only be vertically scaled on a single namenode.=C2=A0 The n= amenode stores the entire file system metadata in memory. This limits the n= umber of blocks, files, and directories supported on the file system to wha= t can be accommodated in the memory of a single namenode. A typical large d= eployment at Yahoo! includes an HDFS cluster with=C2=A02700-4200=C2=A0datanodes with 180 million= files and blocks, and address ~25 PB of storage.=C2=A0 At Facebook, HDFS h= as around 2600 nodes, 300 million files and blocks, addressing up to 60PB o= f storage. While these are very large systems and good enough for majority = of Hadoop users, a few deployments that might want to grow even larger coul= d find the namespace scalability limiting.


<= span style=3D"background-color:rgba(255,255,255,0)">

On Jun 4= , 2016, at 04:43, Ascot Moss <ascot.moss@gmail.com> wrote:

Hi,
=
I read some (old?) articles from Internet about Mapr-FS vs HDFS. =

https://www.mapr.com/products/m5-featur= es/no-namenode-architecture

It states that HDFS Federation= has

a) "Multiple Single Points of Failure", is it really= true?=C2=A0
Why MapR uses HDFS but not HDFS2 in its comparison as this= would lead to an unfair comparison (or even misleading comparison)?=C2=A0 = (HDFS was from Hadoop 1.x, the old generation) HDFS2 is available since 201= 3-10-15, there is no any Single Points of=C2=A0 Failure in HDFS2.

b)= "Limit to 50-200 million files", is it really true?
I have s= een so many real world Hadoop Clusters with over 10PB data, some even with = 150PB data.=C2=A0 If "Limit to 50 -200 millions files" were true = in HDFS2, why are there so many production Hadoop clusters in real world? h= ow can they mange well the issue of=C2=A0 "Limit to 50-200 million fil= es"? For instances,=C2=A0 the Facebook's "Like" implemen= tation runs on HBase at Web Scale, I can image HBase generates huge number = of files in Facbook's Hadoop cluster, the number of files in Facebook&#= 39;s Hadoop cluster should be much much bigger than 50-200 million.

=
From my point of view, in contrast, MaprFS should have true limi= tation up to 1T files while HDFS2 can handle true unlimited files, please d= o correct me if I am wrong.

c) "Performance Bo= ttleneck", again, is it really true?
MaprFS does not have namenode = in order to gain file system performance. If without Namenode, MaprFS would= lose Data Locality which is one of the beauties of Hadoop=C2=A0 If Data Lo= cality is no longer available, any big data application running on MaprFS m= ight gain some file system performance but it would totally lose the true g= ain of performance from Data Locality provided by Hadoop's namenode (ga= in small lose big)

d) "Commercial NAS required"
I= s there any wiki/blog/discussion about Commercial NAS on Hadoop Federation?=

regards
=C2=A0





--
Haya= ti Gonultas

--001a114b30a21c6bb405348948d9--