hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1857) Ability to run a script when a task fails to capture stack traces
Date Mon, 08 Oct 2007 06:54:50 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley updated HADOOP-1857:
----------------------------------

    Status: Open  (was: Patch Available)

You need to fix the new find bugs warning.

I can't see the need to have different debug scripts for mappers and reducers.

I think all of the output (stdout and stderr) from the debug script should be put together
when it is stored on the task tracker.

I don't think adding the concept of executable to the file cache is appropriate. It is basically
compensating for the lack of permissions in hdfs, which will be addressed more directly. In
the mean time, I think that all files coming out of the cache should have the "x" permission
set. Note that pipes and streaming already do this...

Why were the config files for the pipes examples changed to add the "#" part of the url?

Rather than let the user specify a command line that has a bunch (of undocumented) @varaibles,
I think it would be better to always use the same parameters:

basically, something like:

$script @stdout@ @stderr@ @jobconf@

and let the script find the core file if it cares.

By default the entire output of the script should be added to diagnostic and 5 is much much
too small.

> Ability to run a script when a task fails to capture stack traces
> -----------------------------------------------------------------
>
>                 Key: HADOOP-1857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1857
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.15.0
>
>         Attachments: patch-1857.txt, patch-1857.txt, patch-1857.txt, patch-1857.txt,
patch1857.txt
>
>
> This basically is for providing a better user interface for debugging failed
> jobs. Today we see stack traces for failed tasks on the job ui if the job
> happened to be a Java MR job. For non-Java jobs like Streaming, Pipes, the
> diagnostic info on the job UI is not helpful enough to debug what might have
> gone wrong. They are usually framework traces and not app traces.
> We want to be able to provide a facility, via user-provided scripts, for doing
> post-processing on task logs, input, output, etc. There should be some default
> scripts like running core dumps under gdb for locating illegal instructions,
> the last few lines from stderr, etc.  These outputs could be sent to the
> tasktracker and in turn to the jobtracker which would then display it on the
> job UI on demand.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message