Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BC3C1EE00 for ; Wed, 23 Jan 2013 21:07:59 +0000 (UTC) Received: (qmail 11755 invoked by uid 500); 23 Jan 2013 21:07:54 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 11548 invoked by uid 500); 23 Jan 2013 21:07:54 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 11535 invoked by uid 99); 23 Jan 2013 21:07:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 21:07:54 +0000 X-ASF-Spam-Status: No, hits=-1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_HI,SPF_PASS,SPOOFED_URL_HOST,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of swong@netflix.com designates 69.53.237.163 as permitted sender) Received: from [69.53.237.163] (HELO exout104.netflix.com) (69.53.237.163) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 21:07:50 +0000 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=s2048;d=netflix.com; h=from:to:subject:date:message-id:content-type:mime-version; bh=y6wompw6AqGsQulIs0CHrlkeoiE=; b=XOYe6JhU6Hr1mCNxMDUdCkFjXC3M4zguOlqJB/ZgracORXWrMOJ1degRO5nuq2BPC+43dIAL 2l939AyM2ySmPYeXTj2H2Qp8GvbrVjWh/PxB6fG9dEuT2kos7/WCBOFY2BS9IISS9YjYe+Hw /e8ODEqyYKOMXfF4FWlvtE3k6lk6SNXO6pNgV6ogqigKMYCI9eHI6kq3VxM9m5QW+RxRJF7H +c+EXmEYkwNSVh1hm4zZNcTfOddd9JMGVlLE6mKBMZ2Ar95a8eo3aPbiLDT8qHI+zKLo3IzD +UazGo434hqf+0bgZ1h8Q0SbCwoXb5yzwlaM528xRMIk5b7cI2m3uw== DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s2048;d=netflix.com; h=from:to:subject:date:message-id:content-type:mime-version; b=HKxf/tAjsYu1kWyN4pffNlP77U1KMJ/V8NBhVEV7Hhlf6rc46HgOMrEhZEEzsgsuXFzu0HRq +9+/yW3iRTRmC7yPITNOrStytDnaF5e0iJ2bEKLoXq6+dO+8xagTMEB+Ihjko7xW8pWTwwn0 +WsU+chkur+zcWou88Vb1BatsIuR3n6T+KCBveibvijyuUuvkKXYI9ytLrRDS4kg/mn5alNN fDiYbCsiHZOcpoc3F8EdegmaJQ+Y1m3hZFwlF+ANx8aNcYgU7eTJO3ZfbhomweQFMZjbBc5n L8Cx1SiFyP5Ev73K3CvpofgJqsyrLFZTJw+WUsYIt2ZijS6+syFBBQ== Received: from EXFE103.corp.netflix.com (10.64.32.103) by exout104.netflix.com (10.64.240.74) with Microsoft SMTP Server (TLS) id 14.2.298.4; Wed, 23 Jan 2013 13:07:22 -0800 Received: from EXMB102.corp.netflix.com ([169.254.2.85]) by exfe103.corp.netflix.com ([10.64.32.103]) with mapi id 14.02.0283.003; Wed, 23 Jan 2013 13:07:28 -0800 From: Steven Wong To: "user@hadoop.apache.org" Subject: Need help with cluster setup for performance [Impala] Thread-Topic: Need help with cluster setup for performance [Impala] Thread-Index: AQHN+a2mOsAqN6n6HkaOy/x4XLEgtw== Date: Wed, 23 Jan 2013 21:07:28 +0000 Message-ID: <62A90015252F9340B077D64464E2CD613C1590@EXMB102.corp.netflix.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.64.24.186] Content-Type: multipart/alternative; boundary="_000_62A90015252F9340B077D64464E2CD613C1590EXMB102corpnetfli_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_62A90015252F9340B077D64464E2CD613C1590EXMB102corpnetfli_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable My apologies for sending this message to this group, but I'm having trouble= sending to the right group. ________________________________ From: Steven Wong Sent: Wednesday, January 23, 2013 11:15 AM To: impala-user@cloudera.org Subject: RE: Need help with cluster setup for performance Thanks for the suggestions. The /metrics output looks good now, and the SEL= ECT COUNT(*) runs much faster than before. But I still have the "Unknown disk id" error message. My CDH version is: hadoop-client x86_64 2.0.0+552-1.cdh4.1.2.p0.27.el5 cloudera-cdh4 = 18 k hadoop-mapreduce x86_64 2.0.0+552-1.cdh4.1.2.p0.27.el5 cloudera-cdh4 9= .8 M hadoop-yarn x86_64 2.0.0+552-1.cdh4.1.2.p0.27.el5 cloudera-cdh4 8= .9 M On Tuesday, January 22, 2013 5:37:30 PM UTC-8, Henry wrote: On 22 January 2013 11:40, Steven Wong wrote: Hi, I followed http://zenfractal.com/2012/11/15/from-zero-to-impala-in-minutes/= to set up a cluster on EC2. After seeing disappointing performance numbers= from a SELECT COUNT(*), I am following https://ccp.cloudera.com/display/IM= PALA10BETADOC/Configuring+Impala+for+Performance#ConfiguringImpalaforPerfor= mance-TestingImpalaforHighPerformanceConfiguration to check my cluster setu= p. Questions: 1. My cluster has 3 data nodes. Is the following http://:/m= etrics output good? statestore.backend.state.map: { 127.0.0.1:23000 : OK } statestore.live.backends:3 statestore.live.backends.list:[127.0.0.1:22000] Hi Steven - This looks like your problem. Your machines are registering themselves with= 'localhost' as their hostname, and this means that they all look the same = to the statestore. I looked at Matt's zero-to-impala link - it's awesome, but now a little out= of date. You should modify where you run impalad to also have --ipaddress = and --hostname correctly set for each node. Then check the statestore metri= cs; things should look a lot better and your performance should improve. 2. My impalad logs contain "Unknown disk id. This will negatively affect p= erformance. Check your hdfs settings to enable block location metadata." a= nd my http://:/varz doesn't contain the string "dfs.datanod= e.hdfs-blocks-metadata.enabled". But my hdfs-site.xml sets dfs.datanode.hdf= s-blocks-metadata.enabled to true. Why? What version of CDH are you using? 3. My impalad.out doesn't contain "Unable to load native-hadoop library". T= his is good, I believe. 4. My impalad logs contain the following lines matching the word "scheduler= ", but none contains "locality percentage". Why? The locality percentage is printed only for GLOG_v=3D1 - and I note that th= e setup-impala.sh script has a typo where it has GVLOG_v=3D1. If you fix t= his, you should see the locality percentage. Hope this helps - let us know if things improve. Henry /tmp/impalad.INFO:I0122 00:19:09.137197 5121 simple-scheduler.cc:82] Start= ing simple scheduler /tmp/impalad.ip-10-170-17-154.impala.log.INFO.20130122-001901.5121:I0122 00= :19:09.137197 5121 simple-scheduler.cc:82] Starting simple scheduler Thanks. Steven -- -- Henry Robinson Software Engineer Cloudera 415-994-6679 --_000_62A90015252F9340B077D64464E2CD613C1590EXMB102corpnetfli_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
My apologies for sending this message to this group, but I'm having tr= ouble sending to the right group.


From: Steven Wong
Sent: Wednesday, January 23, 2013 11:15 AM
To: impala-user@cloudera.org
Subject: RE: Need help with cluster setup for performance

Thanks for the suggestions. The /metrics output looks good now, and th= e SELECT COUNT(*) runs much faster than before.

But I still have the "Unknown disk id" error message. My CDH= version is:

 hadoop-client        x86_64 2.0.0+552-1.= cdh4.1.2.p0.27.el5 cloudera-cdh4  18 k
 hadoop-mapreduce     x86_64 2.0.0+552-1.cdh4.1.2.p= 0.27.el5 cloudera-cdh4 9.8 M
 hadoop-yarn          x86_64 2.0.0+5= 52-1.cdh4.1.2.p0.27.el5 cloudera-cdh4 8.9 M



On Tuesday, January 22, 2013 5:37:30 PM= UTC-8, Henry wrote:
On 22 January 2013 11:40, Steven Wong <sw...@netflix.com> wrot= e:
Hi,

I followed http://zenfractal.com/2012/11/15/from-zero-to-impala-in-min= utes/ to set up a cluster on EC2. After seeing disappointing performance numbers fr= om a SELECT COUNT(*), I am following https://ccp= .cloudera.com/display/IMPALA10BETADOC/Configuring+Impala= +for+Performance#ConfiguringImpalaforPerformance-Tes= tingImpalaforHighPerformanceConfiguration to check my cluster setup. Questions:

1. My cluster has 3 data nodes. Is the following http://<hostname>:&= lt;port>/metrics output good?

statestore.backend.state.map:
{
  127.0.0.1:23000 : OK
} 
statestore.live.backends:3
statestore.live.backends.list:[127.0.0.1:22000= ]


Hi Steven - 

This looks like your problem. Your machines are registering themselves wit= h 'localhost' as their hostname, and this means that they all look the same= to the statestore.

I looked at Matt's zero-to-impala link - it's awesome, but now a little ou= t of date. You should modify where you run impalad to also have --ipaddress= and --hostname correctly set for each node. Then check the statestore metrics; things should look a lot bet= ter and your performance should improve.

 
2. My impalad logs contain "Unknown disk id.  This will negative= ly affect performance.  Check your hdfs settings to enable block locat= ion metadata." and my http://<hostname>:<port>/varz doesn't contain the string "dfs.datanode.hdfs-blocks-metadata.en= abled". But my hdfs-site.xml sets dfs.datanode.hdfs-blocks-metada= ta.enabled to true. Why?

What version of CDH are you using?
 

3. My impalad.out doesn't contain "Unable to load native-hadoop libra= ry". This is good, I believe.

4. My impalad logs contain the following lines matching the word "sch= eduler", but none contains "locality percentage". Why?


The locality percentage is printed only for GLOG_v=3D1 - and I note that t= he setup-impala.sh script has  a typo where it has GVLOG_v=3D1. If you= fix this, you should see the locality percentage.

Hope this helps - let us know if things improve.

Henry
 

/tmp/impalad.INFO:I0122 00:19:09.137197  5121 simple-scheduler.cc:82]= Starting simple scheduler
/tmp/impalad.ip-10-170-17-154.impala.log.INFO.20130122-001901.51= 21:I0122 00:19:09.137197  5121 simple-scheduler.cc:82] Starting simple= scheduler

Thanks.
Steven

-- 
 
 



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679
--_000_62A90015252F9340B077D64464E2CD613C1590EXMB102corpnetfli_--