Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E4D121870A for ; Thu, 14 Jan 2016 03:25:39 +0000 (UTC) Received: (qmail 60449 invoked by uid 500); 14 Jan 2016 03:25:34 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 60343 invoked by uid 500); 14 Jan 2016 03:25:34 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 60333 invoked by uid 99); 14 Jan 2016 03:25:34 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2016 03:25:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id EC2D5180503 for ; Thu, 14 Jan 2016 03:25:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.446 X-Spam-Level: ** X-Spam-Status: No, score=2.446 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, RP_MATCHES_RCVD=-0.554, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id pHsa4cVvGzP6 for ; Thu, 14 Jan 2016 03:25:19 +0000 (UTC) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [58.251.152.64]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 9308043F05 for ; Thu, 14 Jan 2016 03:25:18 +0000 (UTC) Received: from 172.24.1.51 (EHLO szxeml426-hub.china.huawei.com) ([172.24.1.51]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id DCZ12213; Thu, 14 Jan 2016 11:25:05 +0800 (CST) Received: from SZXEML505-MBX.china.huawei.com ([169.254.1.153]) by szxeml426-hub.china.huawei.com ([10.82.67.181]) with mapi id 14.03.0235.001; Thu, 14 Jan 2016 11:24:26 +0800 From: "Naganarasimha G R (Naga)" To: yaoxiaohua , "'Namikaze Minato'" CC: "user@hadoop.apache.org" Subject: RE: Determine HDP Memory Configuration Settings Thread-Topic: Determine HDP Memory Configuration Settings Thread-Index: AdFOXpwJfvs2tsfYQneobkbEBM+uuP//h7SAgAAEkICAAKrBig== Date: Thu, 14 Jan 2016 03:24:25 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.250.107] Content-Type: multipart/alternative; boundary="_000_AD354F56741A1B47882A625909A59C692BE69303SZXEML505MBXchi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020204.56971512.002A,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=169.254.1.153, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 3002e7ff2a049bd8d0672a78ebf1749b --_000_AD354F56741A1B47882A625909A59C692BE69303SZXEML505MBXchi_ Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable Hi Evan, In most of the scenarios hardware we have is like 8 dual core cpu's which = are hyper threaded so logically its 8*2*2 =3D 32 cores. Usually this logical cpu cores are g= ot by executing the command grep -c /proc/cpuinfo By logical, assumption is that in ideal case 32 threads/processess can run = concurrently. But based on the overload we can configure accordingly. Hope this helps or may be you can reframe your question so that we can help= better. Regards, Naga ________________________________ From: yaoxiaohua [yaoxiaohua@outlook.com] Sent: Thursday, January 14, 2016 06:36 To: 'Namikaze Minato' Cc: user@hadoop.apache.org Subject: RE: Determine HDP Memory Configuration Settings Thanks for your reply, What make you give this conclusion? Evan Yao From: Namikaze Minato [mailto:lloydsensei@gmail.com] Sent: 2016=1B$BG/=1B(B1=1B$B7n=1B(B14=1B$BF|=1B(B 8:50 To: yaoxiaohua Cc: common-user@hadoop.apache.org Subject: Re: Determine HDP Memory Configuration Settings Logical ones. On 14 January 2016 at 01:13, yaoxiaohua > wrote: Hi, When we configure memory in yarn-site.xml, we find a recomm= end in hortonworks website. Determine HDP Memory Configuration Settings http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manua= lly_book/content/determine-hdp-memory-config.html python yarn-utils.py -c 16 -m 64 -d 4 -k True my question is that, for first parameter is the -c CORES The number of cores on each host. How to calculate the number of cores on each host? Use nproc? Or other tools? I know that our host=1B$B!G=1B(Bs physical cores is 8, logi= cal cores is 32. I should use which number? 32 or 8? Best Regards, Evan Yao --_000_AD354F56741A1B47882A625909A59C692BE69303SZXEML505MBXchi_ Content-Type: text/html; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable
Hi Evan,

In most of the scenarios  hardware we ha= ve is like 8 dual core cpu's which are hyper threaded 
so logically its 8*2*2 =3D 32 cores.  Us= ually  this logical cpu cores are got by executing the command 
grep -c /proc/cpuinfo
By logical, assumption is that in ideal case = 32 threads/processess can run concurrently.
But based on the overload we can configure ac= cordingly. 
Hope this helps or may be you can reframe you= r question so that we can help better.

Regards,
Naga
From: yaoxiaohua [yaoxiaohua@outlook.com]=
Sent: Thursday, January 14, 2016 06:36
To: 'Namikaze Minato'
Cc: user@hadoop.apache.org
Subject: RE: Determine HDP Memory Configuration Settings

Thanks for your reply,

    = ;            What ma= ke you give this conclusion?

 

Evan Yao

From: Namika= ze Minato [mailto:lloydsensei@gmail.com]
Sent: 2016=1B$BG/=1B(B1=1B$B7n= =1B(B14=1B$BF|=1B(B 8:50
To: yaoxiaohua
Cc: common-user@hadoop.apache.org
Subject: Re: Determine HDP Memory Configuration Settings

 

Logical ones.

 

On 14 January 2016 at 01:13, yaoxiaohua <yaoxiaohua@outlook.com= > wrote:

Hi,

       = ;         When we configure memory = in yarn-site.xml, we find a recommend in hortonworks  website.

Determine HDP Memory= Configuration Settings

http://docs.horton= works.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/d= etermine-hdp-memory-config.html

       python yarn-utils.py -c 16 -m 64 =
-d 4 -k True

       = ;        

       = ;         my question is that, for = first parameter is the

-c CORES

The number of cores on each host.

       = ;         How to calculate the numb= er of cores on each host?

       = ;         Use nproc?  Or other= tools?

       = ;         I know that our host=1B$B= !G=1B(Bs physical cores is 8, logical cores is 32.

       = ;         I should use which number= ? 32 or 8?

 

Best Regards,

Evan Yao

 

--_000_AD354F56741A1B47882A625909A59C692BE69303SZXEML505MBXchi_--