Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of marcin.mejran@hooklogic.com
 designates 216.32.181.183 as permitted sender)
Received-SPF: pass (mail49-ch1: domain of hooklogic.com designates
 132.245.2.21 as permitted sender) client-ip=132.245.2.21;
 envelope-from=marcin.mejran@hooklogic.com;
 helo=BN1PRD0512HT002.namprd05.prod.outlook.com ;.outlook.com ;
From: Marcin Mejran <marcin.mejran@hooklogic.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Jobtracker memory issues due to FileSystem$Cache
Thread-Topic: Jobtracker memory issues due to FileSystem$Cache
Thread-Index: Ac46yPoHY/l4H0CAT9iDQDNYY12sWA==
Date: Tue, 16 Apr 2013 17:47:10 +0000
Message-ID: 
 <80D53984EECD8940BE08AEC8C0F799B10F80C4CF@BN1PRD0512MB602.namprd05.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_80D53984EECD8940BE08AEC8C0F799B10F80C4CFBN1PRD0512MB602_"
MIME-Version: 1.0

--_000_80D53984EECD8940BE08AEC8C0F799B10F80C4CFBN1PRD0512MB602_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

We've recently run into jobtracker memory issues on our new hadoop cluster.=
 A heap dump shows that there are thousands of copies of DistributedFileSys=
tem kept in FileSystem$Cache, a bit over one for each job run on the cluste=
r and their jobconf objects support this view. I believe these are created =
when the .staging directories get cleaned up but I may be wrong on that.

>From what I can tell in the dump, the username (probably not ugi, hard to t=
ell), scheme and authority parts of the Cache$Key are the same across multi=
ple objects in FileSystem$Cache. I can only assume that the usergroupinform=
ation piece differs somehow every time it's created.

We're using CDH4.2, MR1, CentOS 6.3 and Java 1.6_31. Kerberos, ldap and so =
on are not enabled.

Is there any known reason for this type of behavior?

Thanks,
-Marcin

--_000_80D53984EECD8940BE08AEC8C0F799B10F80C4CFBN1PRD0512MB602_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri","sans-serif";}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">We&#8217;ve recently run into jobtracker memory issu=
es on our new hadoop cluster. A heap dump shows that there are thousands of=
 copies of DistributedFileSystem kept in FileSystem$Cache, a bit over one f=
or each job run on the cluster and their
 jobconf objects support this view. I believe these are created when the .s=
taging directories get cleaned up but I may be wrong on that.<o:p></o:p></p=
>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">From what I can tell in the dump, the username (prob=
ably not ugi, hard to tell), scheme and authority parts of the Cache$Key ar=
e the same across multiple objects in FileSystem$Cache. I can only assume t=
hat the usergroupinformation piece
 differs somehow every time it&#8217;s created.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">We&#8217;re using CDH4.2, MR1, CentOS 6.3 and Java 1=
.6_31. Kerberos, ldap and so on are not enabled.
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Is there any known reason for this type of behavior?=
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Thanks,<o:p></o:p></p>
<p class=3D"MsoNormal">-Marcin<o:p></o:p></p>
</div>
</body>
</html>

--_000_80D53984EECD8940BE08AEC8C0F799B10F80C4CFBN1PRD0512MB602_--