Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E7376D190 for ; Mon, 10 Sep 2012 07:40:37 +0000 (UTC) Received: (qmail 56520 invoked by uid 500); 10 Sep 2012 07:40:32 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 56235 invoked by uid 500); 10 Sep 2012 07:40:32 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 56221 invoked by uid 99); 10 Sep 2012 07:40:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Sep 2012 07:40:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of prabhu.hadoop@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Sep 2012 07:40:25 +0000 Received: by iecs9 with SMTP id s9so3010278iec.35 for ; Mon, 10 Sep 2012 00:40:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=phQmtbS+FPhE9gcZWoiXq7nuQnQ8ZCqA2QmikoIjSMk=; b=Wdj2d+mn9U7S/YiN0voz5Vyv8Lj0Ys1z6VcD6zDKC+X3hrDjEamT9qQYCgTCUw8r4/ wCdhQcVOayDYkuz6yuLSGCWBjKZPJPE69LmbrYNJ1isv+U/W9sOmxMie8J3rnqcjGtzP ffiItdVZ16t/8E4ol+be85XvzcdkkoNKB5rHndgDuyyERu9vHzExiJ2u3uWuAutEs4FI UC11+duGOu8huFJs9pAR53Q2GR1ckoY9JnUXh73O5MvWFFpcvRn9w+6DXEihUM8tYRQo uUJ1OrE6/H5rMolzdwerMdn89E4jEdNjRpPkBdiVa1m7oV0aA8kmYNpEIxomA4rKulrD VDAg== MIME-Version: 1.0 Received: by 10.50.149.137 with SMTP id ua9mr9475940igb.65.1347262804946; Mon, 10 Sep 2012 00:40:04 -0700 (PDT) Received: by 10.64.132.1 with HTTP; Mon, 10 Sep 2012 00:40:04 -0700 (PDT) In-Reply-To: References: Date: Mon, 10 Sep 2012 13:10:04 +0530 Message-ID: Subject: Re: One petabyte of data loading into HDFS with in 10 min. From: prabhu K To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=e89a8f3bac1de8707c04c9541006 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f3bac1de8707c04c9541006 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Users, Thanks for the response. We have loaded 100GB data loaded into HDFS, time taken 1hr.with below configuration. Each Node (1 machine master, 2 machines are slave) 1. 500 GB hard disk. 2. 4Gb RAM 3. 3 quad code CPUs. 4. Speed 1333 MHz Now, we are planning to load 1 petabyte of data (single file) into Hadoop HDFS and Hive table within 10-20 minutes. For this we need a clarification below. 1. what are the system configuration setup required for all the 3 machine= =92s ?. 2. Hard disk size. 3. RAM size. 4. Mother board 5. Network cable 6. How much Gbps Infiniband required. For the same setup we need cloud computing environment too? Please suggest and help me on this. Thanks, Prabhu. On Fri, Sep 7, 2012 at 7:30 PM, Michael Segel wr= ote: > Sorry, but you didn't account for the network saturation. > > And why 1GBe and not 10GBe? Also which version of hadoop? > > Here MapR works well with bonding two 10GBe ports and with the right > switch, you could do ok. > Also 2 ToR switches... per rack. etc... > > How many machines? 150? 300? more? > > Then you don't talk about how much memory, CPUs, what type of storage... > > Lots of factors. > > I'm sorry to interrupt this mental masturbation about how to load 1PB in > 10min. > There is a lot more questions that should be asked that weren't. > > Hey but look. Its a Friday, so I suggest some pizza, beer and then take i= t > to a white board. > > But what do I know? In a different thread, I'm talking about how to tame > HR and Accounting so they let me play with my team Ninja! > :-P > > On Sep 5, 2012, at 9:56 AM, zGreenfelder wrote: > > > On Wed, Sep 5, 2012 at 10:43 AM, Cosmin Lehene > wrote: > >> Here's an extremely na=EFve ballpark estimation: at theoretical hardwa= re > >> speed, for 3PB representing 1PB with 3x replication > >> > >> Over a single 1Gbps connection (and I'm not sure, you can actually rea= ch > >> 1Gbps) > >> (3 petabytes) / (1 Gbps) =3D 291.271111 days > >> > >> So you'd need at least 40,000 1Gbps network cards to get that in 10 > minutes > >> :) - (3PB/1Gbps)/40000 > >> > >> The actual number of nodes would depend a lot on the actual network > >> architecture, the type of storage you use (SSD, HDD), etc. > >> > >> Cosmin > > > > ah, I went te other direction with the math, and assumed no > > replication (completely unsafe and never reasonable for a real, > > production environment, but since we're all theory and just looking > > for starting point numbers) > > > > > > 1PB in 10 min =3D=3D > > 1,000,000gB in 10 min =3D=3D > > 8,000,000gb in 600 seconds =3D=3D > > > > 80,000/6 ~=3D 14k machines running at gigabit or about 1.5k machines i= f > you > > get 10Gb connected machines. > > > > all assuming there's no network or cluster sync overhead > > (of course there would be) > > > > > > that seems like some pretty deep pockets to get to < 10 minute load > > time for that much data. > > > > I could also be off, I just threw some stuff together somewhat > > quickly.between conf calls. > > > > -- > > Even the Magic 8 ball has an opinion on email clients: Outlook not so > good. > > > > --e89a8f3bac1de8707c04c9541006 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi Users,
=A0
Thanks for the response.
=A0

We have loaded 100GB data loaded into HDFS, time taken 1h= r.with below configuration.

Each Node (= 1 machine master, 2 machines =A0are slave)=

1.=A0=A0=A0 = 500 GB hard disk.

2.=A0=A0=A0 = 4Gb RAM

3.=A0=A0=A0 = 3 quad code CPUs.

4.=A0=A0=A0 = Speed 1333 MHz

=A0

Now, we are planning to load 1 petabyte of data (single f= ile) =A0into Hadoop HDFS and Hive table within 10-20 min= utes. For this we need a clarification below.

1. what are the system configuration setup required for a= ll the 3 machine=92s ?.

2. Hard disk size.

3. RAM size.

4. Mother board

5. Network cable

6. How much Gbps =A0Infiniband require= d.

=A0For the same= setup we need cloud computing environment too?

Please suggest and help me on this.

=A0Thanks,

Prabhu.

On Fri, Sep 7, 2012 at 7:30 PM, Michael Segel <michael_segel@hotmail.com> wrote:
Sorry, but you didn't account for= the network saturation.

And why 1GBe and not 10GBe? Also which vers= ion of hadoop?

Here MapR works well with bonding two 10GBe ports and with the right sw= itch, you could do ok.
Also 2 ToR switches... per rack. etc...

Ho= w many machines? 150? 300? more?

Then you don't talk about how m= uch memory, CPUs, what type of storage...

Lots of factors.

I'm sorry to interrupt this mental masturba= tion about how to load 1PB in 10min.
There is a lot more questions that = should be asked that weren't.

Hey but look. Its a Friday, so I s= uggest some pizza, beer and then take it to a white board.

But what do I know? In a different thread, I'm talking about how to= tame HR and Accounting so they let me play with my team Ninja!
:-P

On Sep 5, 2012, at 9:56 AM, zGreenfelder <zgreenfelder@gmail.com> wrote:
> On Wed, Sep 5, 2012 at 10:43 AM, Cosmin Lehene <clehene@adobe.com> wrote:
>> Here's an extremely na=EFve ballpark estimation: at theoretica= l hardware
>> speed, for 3PB representing 1PB with 3x replication<= br>>>
>> Over a single 1Gbps connection (and I'm not sur= e, you can actually reach
>> 1Gbps)
>> (3 petabytes) / (1 Gbps) =3D 291.271111 days>>
>> So you'd need at least 40,000 1Gbps network cards= to get that in 10 minutes
>> :) - (3PB/1Gbps)/40000
>> >> The actual number of nodes would depend a lot on the actual networ= k
>> architecture, the type of storage you use (SSD, =A0HDD), etc.=
>>
>> Cosmin
>
> ah, I went te other directi= on with the math, and assumed no
> replication (completely unsafe and never reasonable for a real,
>= ; production environment, but since we're all theory and just looking> for starting point numbers)
>
>
> 1PB in 10 min = =3D=3D
> 1,000,000gB in 10 min =3D=3D
> 8,000,000gb in 600 seconds =3D=3D=
>
> 80,000/6 =A0~=3D 14k machines running at gigabit or about = 1.5k machines if you
> get 10Gb connected machines.
>
> a= ll assuming there's no network or cluster sync overhead
> (of course there would be)
>
>
> that seems like som= e pretty deep pockets to get to < 10 minute load
> time for that m= uch data.
>
> I could also be off, I just threw some stuff toge= ther somewhat
> quickly.between conf calls.
>
> --
> Even the Magic = 8 ball has an opinion on email clients: Outlook not so good.
>

--e89a8f3bac1de8707c04c9541006--