Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D821310EEB for ; Mon, 10 Jun 2013 09:49:12 +0000 (UTC) Received: (qmail 35438 invoked by uid 500); 10 Jun 2013 09:49:06 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 35058 invoked by uid 500); 10 Jun 2013 09:49:06 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 35051 invoked by uid 99); 10 Jun 2013 09:49:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jun 2013 09:49:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mail2mayank@gmail.com designates 74.125.82.170 as permitted sender) Received: from [74.125.82.170] (HELO mail-we0-f170.google.com) (74.125.82.170) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jun 2013 09:48:59 +0000 Received: by mail-we0-f170.google.com with SMTP id w57so4741223wes.29 for ; Mon, 10 Jun 2013 02:48:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=EXp16SKvj/IILPkfyGaMnxNhDt0eFrOsA7oC+zDALNU=; b=MhptA1hnut973LM7nFv/LNq1xlW+0Z17xdqnSR/lGU8bytw64aN1uwYrxcVNq5SqcM AQIHYcEqJ0ej8H/KxrDsSYCEunkG4PVkLDqT1C+qCtOQbSKTyo6ryRz6BWEVQ0MaIJiK VyaBQOoh8VDuUHab+SbYD1swM56e47kJ7o/3sq51ZO2SSR1wits2r02jGlwOYD51PQGr XamiCJaSnNhQQx88CPhUDGyGTxBeECY+6cw1tg4Y+cX8s1VxUryxPICPYx/64XWUbnew 3fUEjfEG2vFQoxxjsxF5pWy+O73N0o4RlLtEdr5379W5KYKa6fk9lmd07sB2jo8aqr7z 8x6Q== MIME-Version: 1.0 X-Received: by 10.180.91.131 with SMTP id ce3mr4191044wib.55.1370857718670; Mon, 10 Jun 2013 02:48:38 -0700 (PDT) Received: by 10.194.173.162 with HTTP; Mon, 10 Jun 2013 02:48:38 -0700 (PDT) In-Reply-To: References: Date: Mon, 10 Jun 2013 15:18:38 +0530 Message-ID: Subject: Re: Application errors with one disk on datanode getting filled up to 100% From: Mayank To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d04388e555befc704dec9af38 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04388e555befc704dec9af38 Content-Type: text/plain; charset=UTF-8 No it's not a map-reduce job. We've a java app running on around 80 machines which writes to hdfs. The error that I'd mentioned is being thrown by the application and yes we've replication factor set to 3 and following is status of hdfs: Configured Capacity : 16.15 TB DFS Used : 11.84 TB Non DFS Used : 872.66 GB DFS Remaining : 3.46 TB DFS Used% : 73.3 % DFS Remaining% : 21.42 % Live Nodes : 10 Dead Nodes : 0 Decommissioning Nodes : 0 Number of Under-Replicated Blocks : 0 On Mon, Jun 10, 2013 at 3:11 PM, Nitin Pawar wrote: > when you say application errors out .. does that mean your mapreduce job > is erroring? In that case apart from hdfs space you will need to look at > mapred tmp directory space as well. > > you got 400GB * 4 * 10 = 16TB of disk and lets assume that you have a > replication factor of 3 so at max you will have datasize of 5TB with you. > I am also assuming you are not scheduling your program to run on entire > 5TB with just 10 nodes. > > i suspect your clusters mapred tmp space is getting filled in while the > job is running. > > > > > > On Mon, Jun 10, 2013 at 3:06 PM, Mayank wrote: > >> We are running a hadoop cluster with 10 datanodes and a namenode. Each >> datanode is setup with 4 disks (/data1, /data2, /data3, /data4), which each >> disk having a capacity 414GB. >> >> >> hdfs-site.xml has following property set: >> >> >> dfs.data.dir >> >> /data1/hadoopfs,/data2/hadoopfs,/data3/hadoopfs,/data4/hadoopfs >> Data dirs for DFS. >> >> >> Now we are facing a issue where in we find /data1 getting filled up >> quickly and many a times we see it's usage running at 100% with just few >> megabytes of free space. This issue is visible on 7 out of 10 datanodes at >> present. >> >> We've some java applications which are writing to hdfs and many a times >> we are seeing foloowing errors in our application logs: >> >> java.io.IOException: All datanodes xxx.xxx.xxx.xxx:50010 are bad. Aborting... >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3093) >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586) >> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790) >> >> >> I went through some old discussions and looks like manual rebalancing is >> what is required in this case and we should also have >> dfs.datanode.du.reserved set up. >> >> However I'd like to understand if this issue, with one disk getting >> filled up to 100% can result into the issue which we are seeing in our >> application. >> >> Also, are there any other peformance implications due to some of the >> disks running at 100% usage on a datanode. >> -- >> Mayank Joshi >> >> Skype: mail2mayank >> Mb.: +91 8690625808 >> >> Blog: http://www.techynfreesouls.co.nr >> PhotoStream: http://picasaweb.google.com/mail2mayank >> >> Today is tommorrow I was so worried about yesterday ... >> > > > > -- > Nitin Pawar > -- Mayank Joshi Skype: mail2mayank Mb.: +91 8690625808 Blog: http://www.techynfreesouls.co.nr PhotoStream: http://picasaweb.google.com/mail2mayank Today is tommorrow I was so worried about yesterday ... --f46d04388e555befc704dec9af38 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
No it's not a map-reduce job. We've a java app run= ning on around 80 machines which writes to hdfs. The error that I'd men= tioned is being thrown by the application and yes we've replication fac= tor set to 3 and following is status of hdfs:

<= td id=3D"col2"> : DFS Used DFS Remaining%
Configured Capacity 16.15 TB
: 11.84 T= B
Non DFS Used : = 872.66 GB
DFS Remaining : 3.46 TB
DFS Used% : 73.3 %
: 21= .42 %
Live Nodes <= /td> : 10
Dead Nodes : 0
Decommissioning= Nodes : 0
Number of Under-Replicated Blocks : 0


On Mon, Jun 10, 2013 at 3:11 PM, Nitin Pawar <nitinpawar432@gmail.com> wrote:
when you say application er= rors out .. does that mean your mapreduce job is erroring? In that case apa= rt from hdfs space you will need to look at mapred tmp directory space as w= ell.=C2=A0

you got 400GB * 4 * 10 =3D 16TB of disk and lets assume that you have a rep= lication factor of 3 so at max you will have datasize of 5TB with you.=C2= =A0
I am also assuming you are not scheduling your program to run= on entire 5TB with just 10 nodes.=C2=A0

i suspect your clusters mapred tmp space is getting fil= led in while the job is running.=C2=A0


<= div>


On Mon, Jun 10, 2013 at 3:06 PM, Mayank <mail2mayank@gmail.com> wrote:
We are running a hadoop cluster with 10 datanodes and= a namenode. Each datanode is setup with 4 disks (/data1, /data2, /data3, /= data4), which each disk having a capacity 414GB.


hdfs= -site.xml has following property set:

<property>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <name= >dfs.data.dir</name>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= <value>/data1/hadoopfs,/data2/hadoopfs,/data3/hadoopfs,/data4/hadoop= fs</value>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <descript= ion>Data dirs for DFS.</description>
</property>

Now we are facing a iss= ue where in we find /data1 getting filled up quickly and many a times we se= e it's usage running at 100% with just few megabytes of free space. Thi= s issue is visible on 7 out of 10 datanodes at present.

We've some java applications which are writing to hdfs a= nd many a times we are seeing foloowing errors in our application logs:
= 
java.io.IOException: All datanodes xxx.xxx.xxx.xxx:50010 are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(D=
FSClient.java:3093)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.=
java:2586)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCl=
ient.java:2790)

I went through some old discussions and looks like manual rebalan= cing is what is required in this case and we should also have dfs.datanode.= du.reserved set up.

However I'd like to understand if= this issue, with one disk getting filled up to 100% can result into the is= sue which we are seeing in our application.

Also, are there any other peformance implications due to some of the di= sks running at 100% usage on a datanode.
<= /font>
--
Mayank Joshi

Skype: mail2mayank
Mb.:=C2=A0 +91 86906258= 08

Blog: http://www.techynfreesouls.co.nr
PhotoStream: http://picasaweb.google.com/mail2mayank

Today is tommorr= ow I was so worried about yesterday ...



--
Nitin Pawar



--
Mayank Joshi

Sky= pe: mail2mayank
Mb.:=C2=A0 +91 8690625808

Blog: http://www.techynfreesouls.= co.nr
PhotoStream: http://picasaweb.google.com/mail2mayank

Today is tommorr= ow I was so worried about yesterday ... --f46d04388e555befc704dec9af38--