pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Hunt (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-12) Please add timestamps to pig map/reduce progress messages
Date Tue, 04 Dec 2007 19:11:43 GMT

    [ https://issues.apache.org/jira/browse/PIG-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548356
] 

Patrick Hunt commented on PIG-12:
---------------------------------

Pig could provide a set of "default" log config files - say:

1) a log file specing the "true" defaults - run pig with no options and you get X logging,
this could include timestamps

2) a second log file that is similar to what we have now - no timestamps

we could then add a command line option to switch btw the two logging files, or optionally
to select a user specified log file that could be as complex as the user likes (for example
it could calculate delta btw log messages rather than timestamps).

We still need to decide what to do wrt "hadoop logging" - we probably want to have a root
logger that is the same for the pig application log messages (pig code, hadoop code, etc....).

If this works for ppl we would remove almost all of the pig specific log4j configuration that
we have in main and pigcontext and just move it into the log configuration file(s)

> Please add timestamps to pig map/reduce progress messages
> ---------------------------------------------------------
>
>                 Key: PIG-12
>                 URL: https://issues.apache.org/jira/browse/PIG-12
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Olga Natkovich
>
> From one of the users: 
> ------------------------------
> I'm spending a lot of time trying to optimize my pig queries for short
> run-times.  This process would be much easier if, in the progress output
> from pig (currently on stdout, but hopefully soon moving to  
> stderr?!), the
> initiation and completion of each map/reduce job could be  
> timestamped.  Pig
> already spits out messages of the form "----- MapReduce Job -----",  
> "Input:
> ...", "Combine: ...", etc; could you just add a "Timestamp: ..."
> field as well?	Or ideally, both "Starting timestamp: ..." and	
> "Finishing
> timestamp ...".
> Additional comments from another user:
> ------------------------------------------------------
> I'm adding my vote for this as well.
> I'd like to know timestamp and "running time" in seconds or D;H:M:S:
> Thu Oct 25 10:06:01 GMT 2007 (0:00:12:56): 56% done
> Starting and stopping timestamps in the log would also be valuable.
> Unforutately, there's no "workaround" such as putting a date command before and after
the pig command in logging --
> queuing times can be seconds to hours and completely mess up any notion of job execution
time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message