Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 16046103C7 for ; Thu, 4 Apr 2013 13:01:07 +0000 (UTC) Received: (qmail 73761 invoked by uid 500); 4 Apr 2013 13:01:02 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 73657 invoked by uid 500); 4 Apr 2013 13:01:01 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 73641 invoked by uid 99); 4 Apr 2013 13:01:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Apr 2013 13:01:01 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 98.139.253.104 is neither permitted nor denied by domain of daryn@yahoo-inc.com) Received: from [98.139.253.104] (HELO mrout1-b.corp.bf1.yahoo.com) (98.139.253.104) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Apr 2013 13:00:56 +0000 Received: from GQ1-EX10-CAHT01.y.corp.yahoo.com (gq1-ex10-caht01.corp.gq1.yahoo.com [10.73.118.80]) by mrout1-b.corp.bf1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id r34Cx8SB096795 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Thu, 4 Apr 2013 05:59:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1365080350; bh=dMDLm7ENBvoVWZup5XDmdUaDH8fCQgPTKb0UxJsXheI=; h=From:To:Subject:Date:Message-ID:References:In-Reply-To: Content-Type:MIME-Version; b=BdpXHsd1IUb/eXi6X6Rrh62Oy5tqGFDbwzvp3+vqFUEFsmn4bqsjvbCh7jMYNDLBA 0F4ntt3KBw+rWkGhQsTsdPaaXDgkNZUUUYKMwwgyAiiMU+jheuVqR2hQOnjqH2pR3C ZhOo9C5aDIOJYTwd6ZNWIHZI5Av2G4wZJWzfLhKw= Received: from GQ1-MB01-02.y.corp.yahoo.com ([fe80::a049:b5af:9055:ada6]) by GQ1-EX10-CAHT01.y.corp.yahoo.com ([fe80::7c21:dcd7:76a1:8f1b%12]) with mapi id 14.02.0342.003; Thu, 4 Apr 2013 05:59:08 -0700 From: Daryn Sharp To: "" Subject: Re: one minute delay in running a simple ls command on hadoop (maybe near security groups..): hadoop 0.23.5 Thread-Topic: one minute delay in running a simple ls command on hadoop (maybe near security groups..): hadoop 0.23.5 Thread-Index: AQHOLdefFwU2W6KKhUW+3YFjT6RGqpjGgguA Date: Thu, 4 Apr 2013 12:59:07 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [216.145.54.15] Content-Type: multipart/alternative; boundary="_000_F2C4E9662AC447F184FF68070D90E388yahooinccom_" MIME-Version: 1.0 X-Milter-Version: master.31+4-gbc07cd5+ X-CLX-ID: 080349005 X-Virus-Checked: Checked by ClamAV on apache.org --_000_F2C4E9662AC447F184FF68070D90E388yahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Gopi, Check if you can run "groups -F hduser". I think it's causing the delay. Daryn On Mar 31, 2013, at 1:17 AM, Gopi Krishna M wrote: Hi.. I have installed hadoop 0.23.5 and is working fine on two of my installatio= ns. In a new installation on Windows Azure VMs, I am seeing an inordinate = delay (of 1 minute) when doing a simple dfs ls command. This is a 3 node cluster, running 64 bit Ubuntu 12.04.1 LTS, Sun jre command : bin/hadoop dfs -ls / With log set to DEBUG, following is the output on the console. The logs in= bold show the sleep/timeout/delay of 1 minute. Any pointers on what to ch= eck for / what is the reason for this delay? 13/03/31 05:48:55 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.= metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess wit= h annotation @org.apache. hadoop.metrics2.annotation.Metric(valueName=3DTime, about=3D, value=3D[Rate= of successful kerberos logins and latency (milliseconds)], always=3Dfalse, type=3DDEFAULT, sampleName=3DO= ps) 13/03/31 05:48:55 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.= metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure wit= h annotation @org.apache. hadoop.metrics2.annotation.Metric(valueName=3DTime, about=3D, value=3D[Rate= of failed kerberos logins and latency (milliseconds)], always=3Dfalse, type=3DDEFAULT, sampleName=3DOps) 13/03/31 05:48:55 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group = related metrics 13/03/31 05:49:55 DEBUG security.Groups: Creating new Groups object 13/03/31 05:49:55 DEBUG security.Groups: Group mapping impl=3Dorg.apache.ha= doop.security.ShellBasedUni xGroupsMapping; cacheTimeout=3D300000 13/03/31 05:50:55 DEBUG security.UserGroupInformation: hadoop login 13/03/31 05:50:55 DEBUG security.UserGroupInformation: hadoop login commit 13/03/31 05:50:55 DEBUG security.UserGroupInformation: using local user:Uni= xPrincipal: hduser 13/03/31 05:50:55 DEBUG security.UserGroupInformation: UGI loginUser:hduser= (auth:SIMPLE) 13/03/31 05:50:56 DEBUG hdfs.DFSClient: Short circuit read is false 13/03/31 05:50:56 DEBUG ipc.Client: The ping interval is 60000 ms. 13/03/31 05:50:56 DEBUG ipc.Client: Use SIMPLE authentication for protocol = ClientProtocol 13/03/31 05:50:56 DEBUG ipc.Client: Connecting to XXXXXX:9000 Thanks Gopi --_000_F2C4E9662AC447F184FF68070D90E388yahooinccom_ Content-Type: text/html; charset="iso-8859-1" Content-ID: Content-Transfer-Encoding: quoted-printable
Hi Gopi,

Check if you can run "groups -F hduser".  I think it's causi= ng the delay.

Daryn

On Mar 31, 2013, at 1:17 AM, Gopi Krishna M wrote:

Hi..

I have installed hadoop 0.23.5 and is working fine on two o= f my installations.  In a new installation on Windows Azure VMs, I am = seeing an inordinate delay (of 1 minute) when doing a simple dfs ls command= .

This is a 3 node cluster, running 64 bit Ubuntu 12.04.1 LTS= , Sun jre

command : bin/hadoop dfs -ls /

With log set to DEBUG, following is the output on the conso= le.  The logs in bold show the sleep/timeout/delay of 1 minute.  Any pointers on what to check for / what is the reas= on for this delay?

13/03/31 05:48:55 DEBUG lib.MutableMetricsFactory: field org.apache.ha= doop.metrics2.lib.MutableRate
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSucces= s with annotation @org.apache.
hadoop.metrics2.annotation.Metric(valueName=3DTime, about=3D, value=3D= [Rate of successful kerberos logins
and latency (milliseconds)], always=3Dfalse, type=3DDEFAULT, sampleNam= e=3DOps)
13/03/31 05:48:55 DEBUG lib.MutableMetricsFactory: field org.apache.ha= doop.metrics2.lib.MutableRate
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailur= e with annotation @org.apache.
hadoop.metrics2.annotation.Metric(valueName=3DTime, about=3D, value=3D= [Rate of failed kerberos logins and
latency (milliseconds)], always=3Dfalse, type=3DDEFAULT, sampleName=3D= Ops)
13/03/31 05:48:55 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and g= roup related metrics
13/03/31 05:49:55 DEBUG security.Groups:  Creating new Groups obj= ect
13/03/31 05:49:55 DEBUG security.Groups: Group mapping impl=3Dorg.a= pache.hadoop.security.ShellBasedUni
xGroupsMapping; cacheTimeout=3D300000
13/03/31 05:50:55 DEBUG security.UserGroupInformation: hadoop login=
13/03/31 05:50:55 DEBUG security.UserGroupInformation: hadoop login co= mmit
13/03/31 05:50:55 DEBUG security.UserGroupInformation: using local use= r:UnixPrincipal: hduser
13/03/31 05:50:55 DEBUG security.UserGroupInformation: UGI loginUser:h= duser (auth:SIMPLE)
13/03/31 05:50:56 DEBUG hdfs.DFSClient: Short circuit read is false
13/03/31 05:50:56 DEBUG ipc.Client: The ping interval is 60000 ms.
13/03/31 05:50:56 DEBUG ipc.Client: Use SIMPLE authentication for prot= ocol ClientProtocol
13/03/31 05:50:56 DEBUG ipc.Client: Connecting to XXXXXX:9000


Thanks
Gopi


--_000_F2C4E9662AC447F184FF68070D90E388yahooinccom_--