hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-943) Pig crash when it cannot get counter from hadoop
Date Fri, 04 Sep 2009 18:25:57 GMT

    [ https://issues.apache.org/jira/browse/PIG-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751560#action_12751560
] 

Pradeep Kamath commented on PIG-943:
------------------------------------

I am wondering if we should report that we were unable to determine the number of records/bytes
written - 
 {code}
           if (stats.getRecordsWritten()!=-1)                                            
                                                                                         
                                                      
                log.info("Records written : " + stats.getRecordsWritten());              
                                                                                         
                                                       
            if (stats.getBytesWritten()!=-1)                                             
                                                                                         
                                                       
                log.info("Bytes written : " + stats.getBytesWritten());                  
                                                                                         
                                                       
        }                              
 {code}
could change to
{code}
	if(stats.getRecordsWritten()==-1) {
        log.info("Records written : Unable to determine number of records written");
    } else {
        log.info("Records written : stats.getRecordsWritten()");
    }
	if(stats.getBytesWritten()==-1) {
        log.info("Bytes written : Unable to determine number of bytes written");
    } else {
        log.info("Bytes written : stats.getBytesWritten()");
    }
{code}

If we are using -1 to indicate we did not get counter information but the warning has occured
some number of times (maybe in one of the
jobs where we did get counter information),  the following function's output message should
change
:
{code}
public static void logAggregate(Map<Enum, Long> aggMap, MessageType messageType, Log
log) {
    	for(Enum e: aggMap.keySet()) {
    		Long count = aggMap.get(e);
    		if(count != null && count > 0) {
    			String message = "Encountered " + messageType + " " + e.toString() + " " + count +
" time(s).";
    			logMessage(message, messageType, log);
    		}
    	}	
    }
{code}
to
{code}
public static void logAggregate(Map<Enum, Long> aggMap, MessageType messageType, Log
log) {
    	for(Enum e: aggMap.keySet()) {
    		Long count = aggMap.get(e);
    		if(count != null && (count == -1 || count > 0)) {
    			String message = "Encountered " + messageType + " " + e.toString() + " " + (count ==
-1 ? "unknown number of ": count.toString()) + " time(s).";
    			logMessage(message, messageType, log);
    		}
    	}	
    }
{code}

This way we still warn the user about the warning message which got aggregated but just not
report the number of occurences.

To enable the above change the code in MapReduceLauncher.java should also change to ensure
we report 3 types of counts:
0 -> the warning did not occur
-1 ->the warning occured in some of the jobs but in one or more jobs we get a null counter
and hence we cannot report an accurate count
>0 -> we can accurately report the count of occurences of the warning.


> Pig crash when it cannot get counter from hadoop
> ------------------------------------------------
>
>                 Key: PIG-943
>                 URL: https://issues.apache.org/jira/browse/PIG-943
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.3.0
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.4.0
>
>         Attachments: PIG-943-1.patch
>
>
> We see following call stacks in Pig:
> Case 1:
> Caused by: java.lang.NullPointerException
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:390)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:238)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:265)
> Case 2:
> Caused by: java.lang.NullPointerException
>         at org.apache.pig.tools.pigstats.PigStats.accumulateMRStats(PigStats.java:150)
>         at org.apache.pig.tools.pigstats.PigStats.accumulateStats(PigStats.java:91)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:192)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:265)
> In both cases, hadoop jobs finishes without error. The cause of both problems is RunningJob.getCounters()
returns a null, and Pig do not currently check for that. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message