Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 54EBA11342 for ; Thu, 3 Jul 2014 04:22:26 +0000 (UTC) Received: (qmail 23104 invoked by uid 500); 3 Jul 2014 04:22:21 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 22955 invoked by uid 500); 3 Jul 2014 04:22:21 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 22945 invoked by uid 99); 3 Jul 2014 04:22:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 04:22:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,MIME_BASE64_TEXT,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of huozhanfeng@gmail.com designates 209.85.220.54 as permitted sender) Received: from [209.85.220.54] (HELO mail-pa0-f54.google.com) (209.85.220.54) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 04:22:16 +0000 Received: by mail-pa0-f54.google.com with SMTP id et14so13807838pad.27 for ; Wed, 02 Jul 2014 21:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:mime-version:message-id:content-type; bh=nYXKQxpqUT7vgy9kqRTfrfbn7MMDcGJZdJt1xsGXars=; b=hrLUCRdGWjPVemEPR47IvkxXkT1X4TuQbogoi4O0NNJSo0p+TNR6zxkSg3SuXFy8lO C4OTPPrPoWJkWzjqHVZ2aoRvf41a2TCDFUQTzQp3jSZ7A0JCOJsY6SedkqSweIzEoxk9 GwkCYT2Zsqm+wD3Cj4vL8vFcC1U2/uQxTB+Z9q+V26pc8CZhYMXv0uQHkbUDqWWtm020 3rfOOSfjUBhJh7phpFsG9uQXEnO/AZ0mHAXcOfPNIsOmqafZcC5dkeWK1ZnPSQU7YFcV /6azLpcoSXcUsSNeb49XIrsMt+s8PHnYQNi1qFp0wzYQ8jFKszW/9X77hyL+MTDn7B9r xZpw== X-Received: by 10.68.68.236 with SMTP id z12mr7692328pbt.147.1404361311450; Wed, 02 Jul 2014 21:21:51 -0700 (PDT) Received: from 1-PC ([218.30.180.178]) by mx.google.com with ESMTPSA id fu12sm137628481pad.42.2014.07.02.21.21.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Jul 2014 21:21:50 -0700 (PDT) Date: Thu, 3 Jul 2014 12:21:49 +0800 From: "huozhanfeng@gmail.com" To: user Subject: How to limit MRJob's stdout/stderr size(yarn2.3) X-Priority: 3 X-GUID: 2B595913-EF79-4E62-BE04-C68BB66721F0 X-Has-Attach: no X-Mailer: Foxmail 7, 2, 5, 140[cn] Mime-Version: 1.0 Message-ID: <201407031221469355837@gmail.com> Content-Type: multipart/alternative; boundary="----=_001_NextPart836851566672_=----" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. ------=_001_NextPart836851566672_=---- Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: base64 SGksZnJpZW5kOg0KDQogICAgV2hlbiBhIE1SSm9iIHByaW50IHRvbyBtdWNoIHN0ZG91dCBvciBz dGRlcnIgbG9nLCB0aGUgZGlzayB3aWxsIGJlIGZpbGxlZC4gTm93IGl0IGhhcyBpbmZsdWVuY2Ug b3VyIHBsYXRmb3JtIG1hbmFnZW1lbnQuIA0KDQogICAgSSBoYXZlIGltcHJvdmVkIG9yZy5hcGFj aGUuaGFkb29wLm1hcHJlZC5NYXBSZWR1Y2VDaGlsZEpWTS5qYXZhKGNvbWUgZnJvbUBvcmcuYXBh Y2hlLmhhZG9vcC5tYXByZWQuVGFza0xvZykgdG8gZ2VuZXJhdGUgdGhlIGV4ZWN1dGUgY21kIA0K YXMgZm9sbG93czogDQoNCmV4ZWMgL2Jpbi9iYXNoIC1jICIoICRKQVZBX0hPTUUvYmluL2phdmEg LURqYXZhLm5ldC5wcmVmZXJJUHY0U3RhY2s9dHJ1ZSAtRGhhZG9vcC5tZXRyaWNzLmxvZy5sZXZl bD1XQVJOIC1YbXgxMDI0bSAtRGphdmEuaW8udG1wZGlyPSRQV0QvdG1wIC0gICAgIERsb2c0ai5j b25maWd1cmF0aW9uPWNvbnRhaW5lci1sb2c0ai5wcm9wZXJ0aWVzIC1EeWFybi5hcHAuY29udGFp bmVyLmxvZy5kaXI9L2xvZ3MvdXNlcmxvZ3MvYXBwbGljYXRpb25fMTQwMzkzMDY1MzIwOF8wMDAz L2NvbnRhaW5lcl8xNDAzOTMwNjUzMjA4XzAwMDNfMDFfMDAwMDAyIC1EeWFybi5hcHAuY29udGFp bmVyLmxvZy5maWxlc2l6ZT0xMDI0MCAtRGhhZG9vcC5yb290LmxvZ2dlcj1ERUJVRyxDTEEgb3Jn LmFwYWNoZS5oYWRvb3AubWFwcmVkLllhcm5DaGlsZCAkdGVzdF9JUCA1MzkxMSBhdHRlbXB0XzE0 MDM5MzA2NTMyMDhfMDAwM19tXzAwMDAwMF8wIDIgfCB0YWlsIC1jIDEwMiA+L2xvZ3MvdXNlcmxv Z3MvYXBwbGljYXRpb25fMTQwMzkzMDY1MzIwOF8wMDAzL2NvbnRhaW5lcl8xNDAzOTMwNjUzMjA4 XzAwMDNfMDFfMDAwMDAyL3N0ZG91dCA7IGV4aXQgJFBJUEVTVEFUVVMgKSAyPiYxIHwgdGFpbCAt YyAxMDI0MCA+L2xvZ3MvdXNlcmxvZ3MvYXBwbGljYXRpb25fMTQwMzkzMDY1MzIwOF8wMDAzL2Nv bnRhaW5lcl8xNDAzOTMwNjUzMjA4XzAwMDNfMDFfMDAwMDAyL3N0ZGVyciA7IGV4aXQgJFBJUEVT VEFUVVMgIiANCg0KDQogICAgQnV0IGl0IGRvZXNuJ3QgdGFrZSBlZmZlY3QuIA0KDQogICAgQW5k IHRoZW4sIHdoZW4gSSB1c2UgImV4cG9ydCBZQVJOX05PREVNQU5BR0VSX09QVFM9LVhkZWJ1ZyAt WHJ1bmpkd3A6dHJhbnNwb3J0PWR0X3NvY2tldCxhZGRyZXNzPTg3ODgsc2VydmVyPXksc3VzcGVu ZD15IiBmb3IgZGVidWdpbmcgTm9kZU1hbmFnZXIsIEkgZmluZCB3aGVuIEkgc2V0IHRoZSBCcmVh a1BvaW50cyBhdCBvcmcuYXBhY2hlLmhhZG9vcC51dGlsLlNoZWxsKGxpbmUgNDUwOnByb2Nlc3Mg PSBidWlsZGVyLnN0YXJ0KCkpIGFuZCBvcmcuYXBhY2hlLmhhZG9vcC55YXJuLnNlcnZlci5ub2Rl bWFuYWdlci5jb250YWluZXJtYW5hZ2VyLmxhdW5jaGVyLkNvbnRhaW5lckxhdW5jaChsaW5lIDE2 MTpMaXN0PFN0cmluZz4gbmV3Q21kcyA9IG5ldyBBcnJheUxpc3Q8U3RyaW5nPihjb21tYW5kLnNp emUoKSkpIHRoZSBjbWQgd2lsbCB3b3JrLiANCg0KICAgIEkgZG91YnQgdGhlcmUncyBjb25jdXJy ZW5jeSBwcm9ibGVtIGNhdXNlZCBwaXBlIHNoZWxsIHdpbGwgbm90IHBlcmZvcm0gcHJvcGVybHku IEl0IG1hdHRlcnMsIGFuZCBJIG5lZWQgaGVscC4gDQoNCiAgIEBodHRwczovL2lzc3Vlcy5hcGFj aGUub3JnL2ppcmEvYnJvd3NlL1lBUk4tMjIzMQ0KDQp0aGFua3MNCg0KDQoNClpoYW5mZW5nIEh1 bw0K ------=_001_NextPart836851566672_=---- Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <= body>=0A
Hi,friend= :
    When a MRJob print too much stdout or stderr log, = the disk will be filled. Now it has influence our platform management.=0A<= br>=0A
    I have improved org.apache.hadoop.mapred.MapReduce= ChildJVM.java(come from@org.apache.hadoop.mapred.TaskLog) to generate the = execute cmd=0A
as follows: 

exec /bin/bash -c "( $JAVA_HOME/bin/jav= a -Djava.net.preferIPv4Stack=3Dtrue -Dhadoop.metrics.log.level=3DWARN -Xm= x1024m -Djava.io.tmpdir=3D$PWD/tmp -     Dlog4j.configuration=3D= container-log4j.properties -Dyarn.app.container.log.dir=3D/logs/userlogs/a= pplication_1403930653208_0003/container_1403930653208_0003_01_000002 -Dyar= n.app.container.log.filesize=3D10240 -Dhadoop.root.logger=3DDEBUG,CLA org.= apache.hadoop.mapred.YarnChild $test_IP 53911 attempt_1403930653208_0003_m= _000000_0 2 | tail -c 102 >/logs/userlogs/application_1403930653208_000= 3/container_1403930653208_0003_01_000002/stdout ; exit $PIPESTATUS ) 2>= &1 | tail -c 10240 >/logs/userlogs/application_1403930653208_0003/= container_1403930653208_0003_01_000002/stderr ; exit $PIPESTATUS "=0A
= =0A

    But it doesn't take effect.=0A
=0A
  &= nbsp; And then, when I use "export YARN_NODEMANAGER_OPTS=3D-Xdebug -Xrunjd= wp:transport=3Ddt_socket,address=3D8788,server=3Dy,suspend=3Dy" for debugi= ng NodeManager, I find when I set the BreakPoints at org.apache.hadoop.uti= l.Shell(line 450:process =3D builder.start()) and org.apache.hadoop.yarn.s= erver.nodemanager.containermanager.launcher.ContainerLaunch(line 161:List&= lt;String> newCmds =3D new ArrayList<String>(command.size())) the= cmd will work.=0A
=0A
    I doubt there's concurrency pro= blem caused pipe shell will not perform properly. It matters, and I need = help. 

   @https://issues.apache.org/jira/browse/YARN-2231
=0A
thanks
=0A
=0A
Zhanfeng Huo
= =0A ------=_001_NextPart836851566672_=------