Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7F06FCB34 for ; Wed, 23 Jan 2013 08:45:38 +0000 (UTC) Received: (qmail 59568 invoked by uid 500); 23 Jan 2013 08:45:33 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 59204 invoked by uid 500); 23 Jan 2013 08:45:33 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 58885 invoked by uid 99); 23 Jan 2013 08:45:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 08:45:22 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of vikascjadhav87@gmail.com designates 209.85.128.42 as permitted sender) Received: from [209.85.128.42] (HELO mail-qe0-f42.google.com) (209.85.128.42) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 08:45:16 +0000 Received: by mail-qe0-f42.google.com with SMTP id 9so523647qea.15 for ; Wed, 23 Jan 2013 00:44:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=/nYvau6iFnb3m1h3lZgFGSMxaJ42ZQtiWuZo5A5gugY=; b=A+XCAa6FHpajeohkJjLrlGhRSElFX1ne2NhtiMGGo8iJvrfhcD5MsidW9CaosyXAlX tV7++VxNgpmh+5ATZhi+t5fIJqM0RS7PsbBtwoi+c1Urg8dKXkJcJMN9Gvl8Ky9ZxGXP nj3cgVpur8cvKa04IFLmj96ws7ddD6SWm+gCQ/jtu1MBowucCfE5pklz/sS9fmDAvx4c K3HELdwEEVJuzVM/mnVUMmokfST5tfZT080PNBHty2APw+Q/j+keWGaimriKVX672Gl6 uAjCki6Bctvzhj1bANawis67METxx+Aa6tSa66jbGT12seOrbTaJdWZ1mBYuk0dt5bjJ dLgg== MIME-Version: 1.0 X-Received: by 10.49.118.138 with SMTP id km10mr618730qeb.18.1358930696020; Wed, 23 Jan 2013 00:44:56 -0800 (PST) Received: by 10.49.118.166 with HTTP; Wed, 23 Jan 2013 00:44:55 -0800 (PST) In-Reply-To: References: Date: Wed, 23 Jan 2013 14:14:55 +0530 Message-ID: Subject: Fwd: Bulk Loading DFS Space issue in Hbase From: Vikas Jadhav To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b6d97a269220604d3f0b599 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6d97a269220604d3f0b599 Content-Type: text/plain; charset=ISO-8859-1 ---------- Forwarded message ---------- From: Vikas Jadhav Date: Tue, Jan 22, 2013 at 5:23 PM Subject: Bulk Loading DFS Space issue in Hbase To: user@hbase.apache.org Hi I am trying to bulk load 700m CSV data with 31 colms into Hbase I have written MapReduce Program for but when run my program it takes whole disk space and fails Here is Status before running * * ** Configured Capacity : 116.16 GB DFS Used : 13.28 GB Non DFS Used : 61.41 GBDFS Remaining:41.47 GBDFS Used%:11.43 %DFS Remaining%:35.7 % Live Nodes : 1 Dead Nodes :0 Decommissioning Nodes : 0 Number of Under-Replicated Blocks : 68 After Runnign * * * Configured Capacity* : 116.16 GB * DFS Used* : 52.07 GB * Non DFS Used* : 61.47 GB * DFS Remaining* : 2.62 GB * DFS Used%* : 44.83 % * DFS Remaining%* : 2.26 % * **Live Nodes* * * : 1 * **Dead Nodes* * * : 0 * **Decommissioning Nodes* * * : 0 * Number of Under-Replicated Blocks* : 455 So what is taking so much DFS space. Has Anybody come across this issue. even though map and reduce complete 100% For incramental loading of HFILE it again keep on Demanding spcace until whole drive .. 52 GB for 700 MB csv File -- * * * Thanx and Regards* * Vikas Jadhav* -- * * * Thanx and Regards* * Vikas Jadhav* --047d7b6d97a269220604d3f0b599 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable


---------- Forwarded me= ssage ----------
From: Vikas Jadhav <vikascjadha= v87@gmail.com>
Date: Tue, Jan 22, 2013 at 5:23 PM
Subject: Bulk Loading DFS Space issue= in Hbase
To: user@hbase.apache= .org


Hi
I am trying to bulk= load 700m CSV data with 31 colms into Hbase
=A0
I have written MapReduce Program for but when run my pro= gram
it takes whole disk space and fails
=A0
Here is Status before running

=A0<= /strong>
<= td style=3D"padding-right:10px;padding-left:10px;font-weight:bolder;vertica= l-align:top"> DFS Remaining 11.43 %<= /td>
Configured = Capacity := 116.16 GB
DFS Used := 13.28 GB
Non DFS Used : 61.41 GB
: 41.4= 7 GB
DFS Used% :
DFS Remaining% : 35.= 7 %
Live Nodes : 1
Dead Nodes : 0
Decommissioni= ng Nodes : 0
Number of Under-Replicated Blocks : 68

=A0
=A0
Aft= er Runnign

=A0Configured Capacity

<= font color=3D"#000000">=A0:

=A0116= .16 GB

=A0DFS Used

<= font color=3D"#000000">=A0:

=A052.= 07 GB

=A0Non DFS Used

<= font color=3D"#000000">=A0:

=A061.= 47 GB

=A0DFS Remaining<= /p>

<= font color=3D"#000000">=A0:

=A02.6= 2 GB

=A0DFS Used%

<= font color=3D"#000000" size=3D"3" face=3D"Times New Roman">

<= font color=3D"#000000">=A0:

=A044.= 83 %

=A0DFS Remaining%=

<= font color=3D"#000000">=A0:

=A02.2= 6 %

=A0Live Nodes=

<= font color=3D"#000000">=A0:

=A01

=A0Dead Nodes=

<= font color=3D"#000000">=A0:

=A00

=A0Decommissioning Nodes

<= font color=3D"#000000">=A0:

=A00

=A0Number of Under-Replicated Block= s

<= font color=3D"#000000">=A0:

=A0455=

=A0

=A0

So what is taking so much DFS spac= e.

Has Anybody come across this issue.

= =A0

even though map and red= uce complete 100%

For incramental loading of HFILE it a= gain keep on

Demanding spcace until whole drive ..

= =A0

=A0

52 GB for 700=A0MB csv File

=A0

=A0

=A0

=A0



--

=

Thanx and Regards
=A0Vikas Jadhav=



--

=

Thanx and Regards
=A0Vikas Jadhav
--047d7b6d97a269220604d3f0b599--