hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Ding (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-1181) Need to find the right place to store streaming error messages when running multi-query scripts
Date Wed, 06 Jan 2010 19:16:54 GMT
Need to find the right place to store streaming error messages when running multi-query scripts

------------------------------------------------------------------------------------------------

                 Key: PIG-1181
                 URL: https://issues.apache.org/jira/browse/PIG-1181
             Project: Pig
          Issue Type: Bug
            Reporter: Richard Ding



Pig Latin allows user to specify a HDFS directory to store the streaming stderr ourput (if
necessary). For instance, the following script

{code}
DEFINE Y `stream.pl` stderr('stream_err' limit 100);
X = STREAM A THROUGH Y;
STORE X INTO '/tmp/stream_out';
{code} 

will put streaming stderr into the directory _/tmp/stream_out/_logs/stream_err_.  Namely,
in the _logs directory of the job's output directory.

But the problem occurs with multiquery scripts where a single job can have multiple output
directories. The current implementation stores streamig stderr in the _logs directory of a
ramdom generated tmp directory and it would be hard for user to find if she needs to look
ino streaming stderr messages.

A better solution is needed to store the streaming stderr in HDFS for multiquery scripts.




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message