hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1857) Ability to run a script when a task fails to capture stack traces
Date Tue, 09 Oct 2007 06:10:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533280
] 

Owen O'Malley commented on HADOOP-1857:
---------------------------------------

Ok, I can see having different scripts for map and reduce, since you can mix stream or pipes
with Java. (We should probably even support combinations of the two, eventually. *smile*)
I don't think these scripts are things you want to turn on to "debug", but rather hooks that
you'll always leave on to give more details about problems when they occur.

{quote}
This can be done by concatenating the files if we want. But redirection in the command is
not possible, since we dont know the order.
{quote}
I don't understand this. I was thinking that we run the script something like:
{code}
bash -e "$script $stdout $stderr $jobconf 2>&1 > $debugout"
{code}
to tie the stdout and stderr streams together.

Doesn't the "file" command when run on a core file give the executable name? Why does the
executable need to be in the current working directory? That doesn't sound right.

In terms of the parameters, it just seems like the script should have a single interface rather
than supporting a bunch of variables that the user can put together. Especially since you
are adding a fair amount of code to find core files that could be done just as well, if not
better in the script itself.



> Ability to run a script when a task fails to capture stack traces
> -----------------------------------------------------------------
>
>                 Key: HADOOP-1857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1857
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.15.0
>
>         Attachments: patch-1857.txt, patch-1857.txt, patch-1857.txt, patch-1857.txt,
patch1857.txt, tt-no-warn.patch
>
>
> This basically is for providing a better user interface for debugging failed
> jobs. Today we see stack traces for failed tasks on the job ui if the job
> happened to be a Java MR job. For non-Java jobs like Streaming, Pipes, the
> diagnostic info on the job UI is not helpful enough to debug what might have
> gone wrong. They are usually framework traces and not app traces.
> We want to be able to provide a facility, via user-provided scripts, for doing
> post-processing on task logs, input, output, etc. There should be some default
> scripts like running core dumps under gdb for locating illegal instructions,
> the last few lines from stderr, etc.  These outputs could be sent to the
> tasktracker and in turn to the jobtracker which would then display it on the
> job UI on demand.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message