tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (TEZ-1698) Cut down on ResourceCalculatorProcessTree overheads in Tez
Date Fri, 31 Oct 2014 06:41:34 GMT

     [ https://issues.apache.org/jira/browse/TEZ-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajesh Balamohan updated TEZ-1698:
----------------------------------
    Attachment: TEZ-1725.addendum.patch

Looks like IDEA setup caused this. Attaching the addendum patch which fixes the license. 
Will commit to master & branch-0.5 shortly.

> Cut down on ResourceCalculatorProcessTree overheads in Tez
> ----------------------------------------------------------
>
>                 Key: TEZ-1698
>                 URL: https://issues.apache.org/jira/browse/TEZ-1698
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.5.2
>            Reporter: Gopal V
>            Assignee: Rajesh Balamohan
>             Fix For: 0.5.2
>
>         Attachments: ProcfsBasedProcessTree.png, ProcfsFiles.png, TEZ-1698.1.patch, TEZ-1698.2.patch,
TEZ-1698.3.patch, TEZ-1698.4.patch, TEZ-1725.addendum.patch
>
>
> ResourceCalculatorProcessTree scraps all of /proc/ for PIDs which are part of the current
task's process group.
> This is mostly wasted in Tez, since unlike YARN which has to do this since it has the
PID for the container-executor process (bash) and has to trace the bash -> java spawn inheritance.
> !ProcfsBasedProcessTree.png!
> The latency effect of this is less clearly visible with the profiler turned on as this
is primarily related to rate of syscalls + overhead in the kernel (via the following codepath
in YARN).
> !ProcfsFiles.png!
> {code}
>  private List<String> getProcessList() {
>     String[] processDirs = (new File(procfsDir)).list();
> ...
>     for (String dir : processDirs) {
>       try {
>         if ((new File(procfsDir, dir)).isDirectory()) {
>           processList.add(dir);
>         }
> ...
>   public void updateProcessTree() {
>     if (!pid.equals(deadPid)) {
>       // Get the list of processes
>       List<String> processList = getProcessList();
> ...
>       for (String proc : processList) {
>         // Get information for each process
>         ProcessInfo pInfo = new ProcessInfo(proc);
>         if (constructProcessInfo(pInfo, procfsDir) != null) {
>           allProcessInfo.put(proc, pInfo);
>           if (proc.equals(this.pid)) {
>             me = pInfo; // cache 'me'
>             processTree.put(proc, pInfo);
>           }
>         }
>       }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message