Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BC7551166B for ; Fri, 12 Sep 2014 19:31:42 +0000 (UTC) Received: (qmail 21055 invoked by uid 500); 12 Sep 2014 19:31:34 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 20933 invoked by uid 500); 12 Sep 2014 19:31:34 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 20790 invoked by uid 99); 12 Sep 2014 19:31:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 19:31:34 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of stransky.ja@gmail.com designates 209.85.217.171 as permitted sender) Received: from [209.85.217.171] (HELO mail-lb0-f171.google.com) (209.85.217.171) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 19:31:09 +0000 Received: by mail-lb0-f171.google.com with SMTP id 10so1505838lbg.2 for ; Fri, 12 Sep 2014 12:31:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=xwe4lSykDBV/eFgjuzrCJVJDKrvRdCa+pg7h6MMB/vk=; b=s7xYHuTSRrmo5cVp/3vVCppQP5TiOqmD80e89ot1dd9uBgMZTpjLb4+iqTlyJOafcX a5D6g5AMPrz55Nql9RXvJl6wwEEY5+jLqUahOaujsS08L68bQmdr+iHlEDtTbMedHTJW eRqK6Ne3W1/UC1DzpNGPkNWL7Xj1HEZ1vLcCJqYAYjwvgf+AltOup5EMMo2Jq5UQfcAl rmWbvU73oVBrhMGunpPTqcsUMUUr5PIpnkxP4Kxt9Z9CAo719eNuGHUoQGTzZsGcgxrQ WxhP+U55NZVXgpuTyP0czU3HqUgEazAL0QVXdl4lrv5Td8QAhn2yvoSwi2b/s/S9OPWv 2Xpg== MIME-Version: 1.0 X-Received: by 10.112.34.143 with SMTP id z15mr10659908lbi.86.1410550268255; Fri, 12 Sep 2014 12:31:08 -0700 (PDT) Received: by 10.114.198.131 with HTTP; Fri, 12 Sep 2014 12:31:08 -0700 (PDT) In-Reply-To: References: Date: Fri, 12 Sep 2014 21:31:08 +0200 Message-ID: Subject: Re: CPU utilization From: Jakub Stransky To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae93d94d8ad62580502e35378 X-Virus-Checked: Checked by ClamAV on apache.org --14dae93d94d8ad62580502e35378 Content-Type: text/plain; charset=UTF-8 Adam, how did you come to the conclusion that it is memory bounded? I haven't found any such sign, even though the map phase were assigned 768MB, job counters reported that just something around 600MB were use and no significant GC time imposed. To be more specific about the job what in essence do is loading data out of kafka messaging in protocol buffers format deserialize those and remap to avro data format. And that is performed on per record bases except the kafka reader which performs bulk read via buffer. Increasing a buffer size and fetch size didn't have any significant impact. May be completely silly question: how do I recognize that I have a memory bound job? As having a ~600MB heap and GC time somewhere around 30sec out of 60 min long job doesn't seem to me as a sign of insufficient memory. I don't see any apparent bound except that I mentioned on CPU per task process via top command. On 12 September 2014 20:57, Adam Kawa wrote: > Your NodeManager can use 2048 MB (yarn.nodemanager.resource.memory-mb) for > allocating containers. > > If you run map task, you need 768 MB (mapreduce.map.memory.mb). > If you run reduce task, you need 1024 MB (mapreduce.reduce.memory.mb). > If you run the MapReduce app master, you need 1024 MB ( > yarn.app.mapreduce.am.resource.mb). > > Therefore, you run MapReduce job, you can run only 2 containers per > NodeManager (3 x 768 = 2304 < 2048) on your setup. > > 2014-09-12 20:37 GMT+02:00 Jakub Stransky : > > >> I thought that memory assigned has to be muliply of >> yarn.scheduler.minimum-allocation-mb and is rounded according that. >> > > That's right. It also specifies the minimum size of a container to prevent > from requesting unreasonable small containers (that are likely to cause > tasks failures). > >> >> any other I am not aware of. Are there any additional parameters like >> that you mentioned which should be set? >> > > There are also settings related to vcores in mapred-site.xml and > yarn-site.xml. But they don't change anything in your case (as you are > limited by the memory, not vcores). > > >> The job wasn't the smallest but wasn't PB of data. Was run on 1.5GB of >> data and run for 60min. I wasn't able to make any significant improvment. >> It is map only job. And wasn't able to achive more that 30% of total >> machine cpu utilization. Howewer top command were displaying 100 %cpu for >> process running on data node, that's why I was thinking that way about >> limit on container process limit. I didn't find any other boundary like io >> or network or memory. >> > > CPU utilization depends on type of your jobs (e.g. doing complex math > operations or just counting words) and the number of containers you run. If > you want to play with this, you can run more CPU-bound jobs or increase the > number of containers running on a node. > -- Jakub Stransky cz.linkedin.com/in/jakubstransky --14dae93d94d8ad62580502e35378 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Adam, how did you come to the conclusion that it is memory= bounded? I haven't found any such sign, even though the map phase were= assigned 768MB, job counters reported that just something around 600MB wer= e use and no significant GC time imposed.

To be more spe= cific about the job what in essence do is loading data out of kafka messagi= ng in protocol buffers format deserialize those and remap to avro data form= at. And that is performed on per record bases except the kafka reader which= performs bulk read via buffer. Increasing a buffer size and fetch size did= n't have any significant impact.

May be comple= tely silly question: how do I recognize that I have a memory bound job? As = having a ~600MB heap and GC time somewhere around 30sec out of 60 min long = job doesn't seem to me as a sign of insufficient memory.
I do= n't see any apparent bound except that I mentioned on CPU per task proc= ess via top command.

<= br>
On 12 September 2014 20:57, Adam Kawa <kaw= a.adam@gmail.com> wrote:
Your NodeManager can use=C2=A02048 MB (yarn.n= odemanager.resource.memory-mb)=C2=A0for allocating contai= ners.=C2=A0

If you run map task, you need=C2=A0768 M= B (mapreduce.map.memory.mb).
If you run reduce = task, you need=C2=A01024 MB (mapreduce.reduce.memory.mb).
If you run the MapReduce app master, you need=C2=A01024 = MB (y= arn.app.mapreduce.am.resource.mb).

Therefore, you run MapReduce job, you can run only 2 containers per = NodeManager (3 x=C2=A0768 =3D=C2=A02304 < 2048) on your setup.

=
2014-09-12 20:37 GMT+02:00 Jakub Stransky <stransky.ja@gmail.com>:
=C2=A0
= =C2=A0I thought that memory assigned has to be muliply of=C2=A0yarn.scheduler.minimum-allocation-= mb and is rounded according that.

<= /div>
That's right. It also specifies the minimum size of a = container to prevent from requesting unreasonable small containers (that ar= e likely to cause tasks failures).
=

any other I am not aware of. Are there any additional parameters li= ke that you mentioned which should be set?
There are also settings related to vcores in mapred-site= .xml and yarn-site.xml. But they don't change anything in your case (as= you are limited by the memory, not vcores).
=C2= =A0
The job wasn't the smallest but wasn't = PB of data. Was run on 1.5GB of data and run for 60min. I wasn't able t= o make any significant improvment. It is map only job. And wasn't able = to achive more that 30% of total machine cpu utilization. Howewer top comma= nd were displaying 100 %cpu for process running on data node, that's wh= y I was thinking that way about limit on container process limit. I didn= 9;t find any other boundary like io or network or memory.

CPU utilization depends on type of your j= obs (e.g. doing complex math operations or just counting words) and the num= ber of containers you run. If you want to play with this, you can run more = CPU-bound jobs or increase the number of containers running on a node.



--
--14dae93d94d8ad62580502e35378--