Mailing-List: contact pig-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@incubator.apache.org
Message-ID: <12747562.1196285443038.JavaMail.jira@brutus>
Date: Wed, 28 Nov 2007 13:30:43 -0800 (PST)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: pig-dev@incubator.apache.org
Subject: [jira] Commented: (PIG-12) Please add timestamps to pig map/reduce
 progress messages
In-Reply-To: <1470723.1193971910927.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/PIG-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546411 ] 

Patrick Hunt commented on PIG-12:
---------------------------------

I have an alternative suggestion that I'd like to get your feedback on.

Both hadoop and pig make use of log4j for all logging. Why write special purpose code when we can just have the users specify something like this:
-----
bash-3.00$ cat log4j.properties
log4j.rootLogger=INFO, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout

# Print the date in ISO 8601 format
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
------
in their classpath for pig. The result is something like:

2007-11-28 21:23:25,632 [main] DEBUG org.apache.hadoop.ipc.Client - IPC Client connection to mithril-nn1.inktomisearch.com/68.180.187.193:8020 sending #0


I did notice an issue in Pig - Pig has hardcoded a configuration for it's logger. If we replace lines 159-164 in Main.java with something like:

BasicConfigurator.configure();

from http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/BasicConfigurator.html

this should provide the users with timestamps as well as a whole lot more flexibility than if we hardcoded something ourselves.

If you agree then we are basically done (you just need to apply this change I suggested and test it out).


> Please add timestamps to pig map/reduce progress messages
> ---------------------------------------------------------
>
>                 Key: PIG-12
>                 URL: https://issues.apache.org/jira/browse/PIG-12
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Olga Natkovich
>
> From one of the users: 
> ------------------------------
> I'm spending a lot of time trying to optimize my pig queries for short
> run-times.  This process would be much easier if, in the progress output
> from pig (currently on stdout, but hopefully soon moving to  
> stderr?!), the
> initiation and completion of each map/reduce job could be  
> timestamped.  Pig
> already spits out messages of the form "----- MapReduce Job -----",  
> "Input:
> ...", "Combine: ...", etc; could you just add a "Timestamp: ..."
> field as well?	Or ideally, both "Starting timestamp: ..." and	
> "Finishing
> timestamp ...".
> Additional comments from another user:
> ------------------------------------------------------
> I'm adding my vote for this as well.
> I'd like to know timestamp and "running time" in seconds or D;H:M:S:
> Thu Oct 25 10:06:01 GMT 2007 (0:00:12:56): 56% done
> Starting and stopping timestamps in the log would also be valuable.
> Unforutately, there's no "workaround" such as putting a date command before and after the pig command in logging --
> queuing times can be seconds to hours and completely mess up any notion of job execution time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.