Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8BC7FD3F1 for ; Mon, 10 Sep 2012 09:41:37 +0000 (UTC) Received: (qmail 90891 invoked by uid 500); 10 Sep 2012 09:41:32 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 90281 invoked by uid 500); 10 Sep 2012 09:41:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 90252 invoked by uid 99); 10 Sep 2012 09:41:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Sep 2012 09:41:24 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.220.176] (HELO mail-vc0-f176.google.com) (209.85.220.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Sep 2012 09:41:16 +0000 Received: by vcbfl11 with SMTP id fl11so1347778vcb.35 for ; Mon, 10 Sep 2012 02:40:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=8Ov1oknvYqeIFNW8zp5Yu6DaS4N3MgNbTGNHUtvef/0=; b=FH3Wci8LOQx5nNOv4tEqmyIdOqVyyxCiZebmFElfHoZWTN0YOC0fUGSSpaC4ygM9Io PJ8Uqa/dnSGgo6mV0kpe+sKvFX2qvjGqbakc+QKAq2V5FSkCR5MU5zRpbHfmJpTmVtkz lNWTB3T5bzVRUnQw9zLjS/qYMKNJrRZeQMByQacjS7KU7FTEdbQQkUIObNgT+2Viidrm 9eMv8VwQDZwDJe04K7Ek+FIzixcQ4smLNrqRDr8wP4BgZNWOC+bkMOyJPz457JAq6412 Ojh7Dd0j/nQW9RJzVCNe8XzgAl72EP3r3BtrEBKvS1WcAxntqKgZ0dkSSsKzQkroGaeI q/SA== MIME-Version: 1.0 Received: by 10.52.71.7 with SMTP id q7mr15100555vdu.113.1347270055327; Mon, 10 Sep 2012 02:40:55 -0700 (PDT) Received: by 10.52.64.235 with HTTP; Mon, 10 Sep 2012 02:40:55 -0700 (PDT) In-Reply-To: References: Date: Mon, 10 Sep 2012 10:40:55 +0100 Message-ID: Subject: Re: One petabyte of data loading into HDFS with in 10 min. From: Steve Loughran To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf3071c6ca10767004c955c168 X-Gm-Message-State: ALoCoQmiMTsWk9If1mmlcSPGjxy6RZlMnnMCO2LZb84QvDwMSK9/tNxz1H6HCo9Go2Xh146632np --20cf3071c6ca10767004c955c168 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 10 September 2012 08:40, prabhu K wrote: > Hi Users, > > Thanks for the response. > > > We have loaded 100GB data loaded into HDFS, time taken 1hr.with below > configuration. > > Each Node (1 machine master, 2 machines are slave) > > 1. 500 GB hard disk. > > 2. 4Gb RAM > > 3. 3 quad code CPUs. > > 4. Speed 1333 MHz > > > > Now, we are planning to load 1 petabyte of data (single file) into > Hadoop HDFS and Hive table within 10-20 minutes. For this we need a > clarification below. > > 1. what are the system configuration setup required for all the 3 > machine=E2=80=99s ?. > 2. Hard disk size. > At least a petabyte, maybe three. If you were planning to do some pre-storage processing, such as filter or compress the data, to it before the upload. > 3. RAM size. > > 4. Mother board > > 5. Network cable > > 6. How much Gbps Infiniband required. > > yes. > For the same setup we need cloud computing environment too? > > Please suggest and help me on this. > > Thanks, > Prabhu, I don't think you've been reading the replies fully. The data rate coming off the filtered Cern LHC experiments is 1.6 PB/month. Your "10 minute" upload is trying to handle two weeks' worth of CERN data in a fraction of time. Nobody can seriously point to your questions and say "this is the motherboard you need" as your project seems to have some unrealistic goals. If you do want to do a 1PB upload in 10 minutes -or even, say 30-60 minutes, the first actions in your project should be 1. Come up with some realistic deliverables rather than a a vague "1 PB/10 minute" requirements. 2. Include a realistic timetable as part of those deliverables. 3. Look at the data source(s) and work out how fast they can actually generate data off their hard disks, out of their database, or whatever. That's your maximum bandwidth irrespective of what you do with the data afterwards. 4. Hire someone who knows about these problems and how to solve them -or who at least is respected enough that when they say "you need realistic goals" they'd be believed. Someone could set up a network to transfer 1 PB of data into a Hadoop cluster in 10 Minutes, but it would be a bleeding edge exercise you'd end up writing papers about in VLDB or similar conferences. The cost of doing so would be utterly excessive unless you were planning to load (and then hopefully, discard) another PB in the next 10 minutes -and again, repeatedly. Otherwise you would be paying massive amounts for network bandwidth that would only ever be using for ten minutes. Asking for help on the -user list isn't going to solve your problems, as the "1 PB in 10 minutes" goal is the problem. Do you really need all that data? In 10 minutes? IF so, then you're going to have to find someone who really, really knows about networking, disk IO bandwidth, cluster commissioning, etc. I'm not volunteering. I may have some colleagues you could talk to, but that -as with other people on this list- would be in the category of action 5, "pay for consultancy" Sorry. --20cf3071c6ca10767004c955c168 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On 10 September 2012 08:40, prabhu K <prabhu.hadoop@gmail.com> wrote:
Hi Users,
=C2=A0
Thanks for the response.
=C2=A0

We have loaded 100GB data loaded into HDFS, time taken 1h= r.with below configuration.

Each Node (1 machine master= , 2 machines =C2=A0are slave)

1.=C2=A0=C2=A0=C2=A0 500 GB hard disk.

2.=C2=A0=C2=A0=C2=A0 4Gb RAM

3.=C2=A0=C2=A0=C2=A0 3 quad code CPUs.

4.=C2=A0=C2=A0=C2=A0 Speed 1333 MHz

=C2=A0

Now, we are planning to load 1 petabyte of data (single f= ile) =C2=A0into Hadoop HDFS and Hive table within 10-20 minute= s. For this we need a clarification below.

1. what are the system configuration setup required for a= ll the 3 machine=E2=80=99s ?.


=

2. Hard disk size.


At least a petabyte, maybe three.=C2=A0

If you were planning to do some pre-storage processing, such as filter or c= ompress the data, to it before the upload.
=C2=A0

3. RAM size.

4. Mother board

5. Network cable

=C2=A0=C2=A0

= 6. How much Gbps =C2=A0Infiniband requi= red.


yes.=C2=A0

=C2=A0

=C2=A0For t= he same setup we need cloud computing environment too?

Please suggest and help me on this.

=C2=A0Thanks,


Prabhu, I don't think you've been reading the replies fully.<= /div>

The data rate coming off the filtered Cern LHC experiments i= s 1.6 PB/month. Your "10 minute" upload is trying to handle two w= eeks' worth of CERN data in a fraction of time.

Nobody can seriously point to your questions and say "this is the= motherboard you need" as your project seems to have some unrealistic = goals. If you do want to do a 1PB upload in 10 minutes -or even, say 30-60 = minutes, the first actions in your project should be

  1. Come up with some realistic deliverables rather= than a a vague "1 PB/10 minute" requirements.
  2. Include a = realistic timetable as part of those deliverables.
  3. Look at the data= source(s) and work out how fast they can actually generate data off their = hard disks, out of their database, or whatever. That's your maximum ban= dwidth irrespective of what you do with the data afterwards.
  4. Hire someone who knows about these problems and how to solve them -or w= ho at least is respected enough that =C2=A0when they say "you need rea= listic goals" they'd be believed.
Someone could set = up a network to transfer 1 PB of data into a Hadoop cluster in 10 Minutes, = but it would be a bleeding edge exercise you'd end up writing papers ab= out in VLDB or similar conferences.=C2=A0

The cost of doing so would be utterly excessive unless = you were planning to load (and then hopefully, discard) another PB in the n= ext 10 minutes -and again, repeatedly. Otherwise you would be paying massiv= e amounts for network bandwidth that would only ever be using for ten minut= es.=C2=A0

Asking for help on the -user list isn't going to so= lve your problems, as the "1 PB in 10 minutes" goal is the proble= m. Do you really need all that data? In 10 minutes? IF so, then you're = going to have to find someone who really, really knows about networking, di= sk IO bandwidth, cluster commissioning, etc. I'm not volunteering. I ma= y have some colleagues you could talk to, but that -as with other people on= this list- would be in the category of action 5, "pay for consultancy= "

Sorry.=C2=A0

--20cf3071c6ca10767004c955c168--