hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkady Borkovsky <ark...@yahoo-inc.com>
Subject Re: [jira] Created: (HADOOP-1328) Hadoop Streaming needs to provide a way for the stream plugin to update global counters
Date Fri, 04 May 2007 21:50:55 GMT
This is a really good suggestion.
In general, this a good way to let a streaming process to send  
information to the Java streaming class:
stderr lines that match a predefined regular expression are treated  
as commands.
For example, I use this to kill the whole job: if a tasks runs into a  
deterministic error (e.g. a syntax error in the script), there is no  
need to wait for N*4 tasks to fail)

However, before this implemented, hadnling of streaming stderr needs  
to be improved.
I still have to save the stderr in DFS with my own hacks.

On May 4, 2007, at 2:37 PM, Runping Qi (JIRA) wrote:

> Hadoop Streaming needs to provide a way for the stream plugin to  
> update global counters
> ---------------------------------------------------------------------- 
> -----------------
>
>                  Key: HADOOP-1328
>                  URL: https://issues.apache.org/jira/browse/ 
> HADOOP-1328
>              Project: Hadoop
>           Issue Type: New Feature
>             Reporter: Runping Qi
>
>
>
> Sometimes, the stream plugin may want to create/update its own  
> global counters.
> There is no way to do so currently.
> One possible way to enable that is to use the stderr output of the  
> stream process.
> The stream process can emit the global counter update information  
> (such as GLOCAL_COUNTER COUNTER_NAME NUM)
> to the stderr of the process. The Stderr handling thread can call  
> the global counter API whenever it encounters the global counter  
> update information line.
>
>
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>


Mime
View raw message