Return-Path: 
 <issues-return-12653-apmail-tez-issues-archive=tez.apache.org@tez.apache.org>
X-Original-To: apmail-tez-issues-archive@minotaur.apache.org
Delivered-To: apmail-tez-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B84517FB2
	for <apmail-tez-issues-archive@minotaur.apache.org>;
 Fri, 31 Oct 2014 01:58:34 +0000 (UTC)
Received: (qmail 87541 invoked by uid 500); 31 Oct 2014 01:58:34 -0000
Delivered-To: apmail-tez-issues-archive@tez.apache.org
Received: (qmail 87499 invoked by uid 500); 31 Oct 2014 01:58:33 -0000
Mailing-List: contact issues-help@tez.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@tez.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@tez.apache.org>
List-Post: <mailto:issues@tez.apache.org>
List-Id: <issues.tez.apache.org>
Reply-To: dev@tez.apache.org
Delivered-To: mailing list issues@tez.apache.org
Received: (qmail 87489 invoked by uid 99); 31 Oct 2014 01:58:33 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2014 01:58:33 +0000
Date: Fri, 31 Oct 2014 01:58:33 +0000 (UTC)
From: "Gopal V (JIRA)" <jira@apache.org>
To: issues@tez.apache.org
Message-ID: <JIRA.12750274.1414127770000.378850.1414720713845@Atlassian.JIRA>
In-Reply-To: <JIRA.12750274.1414127770000@Atlassian.JIRA>
References: <JIRA.12750274.1414127770000@Atlassian.JIRA>
 <JIRA.12750274.1414127770986@arcas>
Subject: [jira] [Updated] (TEZ-1698) Cut down on
 ResourceCalculatorProcessTree overheads in Tez
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/TEZ-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gopal V updated TEZ-1698:
-------------------------
    Attachment: TEZ-1698.3.patch

[~rajesh.balamohan]: Can you test this version? 

Minor changes, the plugin is built only against Sun/Oracle JDKs.

And CumulativeRSS now returns total memory held by the JVM via Runtime.getTotalMemory(), since it includes the free heap memory which is held by the JVM (as finalized/collected garbage).

> Cut down on ResourceCalculatorProcessTree overheads in Tez
> ----------------------------------------------------------
>
>                 Key: TEZ-1698
>                 URL: https://issues.apache.org/jira/browse/TEZ-1698
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.5.2
>            Reporter: Gopal V
>            Assignee: Rajesh Balamohan
>         Attachments: ProcfsBasedProcessTree.png, ProcfsFiles.png, TEZ-1698.1.patch, TEZ-1698.2.patch, TEZ-1698.3.patch
>
>
> ResourceCalculatorProcessTree scraps all of /proc/ for PIDs which are part of the current task's process group.
> This is mostly wasted in Tez, since unlike YARN which has to do this since it has the PID for the container-executor process (bash) and has to trace the bash -> java spawn inheritance.
> !ProcfsBasedProcessTree.png!
> The latency effect of this is less clearly visible with the profiler turned on as this is primarily related to rate of syscalls + overhead in the kernel (via the following codepath in YARN).
> !ProcfsFiles.png!
> {code}
>  private List<String> getProcessList() {
>     String[] processDirs = (new File(procfsDir)).list();
> ...
>     for (String dir : processDirs) {
>       try {
>         if ((new File(procfsDir, dir)).isDirectory()) {
>           processList.add(dir);
>         }
> ...
>   public void updateProcessTree() {
>     if (!pid.equals(deadPid)) {
>       // Get the list of processes
>       List<String> processList = getProcessList();
> ...
>       for (String proc : processList) {
>         // Get information for each process
>         ProcessInfo pInfo = new ProcessInfo(proc);
>         if (constructProcessInfo(pInfo, procfsDir) != null) {
>           allProcessInfo.put(proc, pInfo);
>           if (proc.equals(this.pid)) {
>             me = pInfo; // cache 'me'
>             processTree.put(proc, pInfo);
>           }
>         }
>       }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)