hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinay Kumar Thota (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1793) Exception exculsion functionality is not working correctly.
Date Tue, 18 May 2010 04:26:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868526#action_12868526
] 

Vinay Kumar Thota commented on MAPREDUCE-1793:
----------------------------------------------

It's my suggestion  where exactly the code needs to change to resolve the issue.So that I
have mentioned the part of the code in description field. 

The pattern always  should be either ERROR,WARN and FATAL and we need to fetch the exceptions
based on the pattern from the file.Once we got the exceptions, we need to exclude the exceptions
from the output list.Later we need to count the new exceptions.

For example, In my above suggestion the shell command generates like this.

grap ERROR logfiles* | grep -v IOExceptin | grep -vc java.net.ConnectException | awk -F :
'{s+=$2} END {print s}'

here {{filePattern}} is  logfile* and {{pattern}} is ERROR

I would say my suggestion is 100% correct and there is no faulty in that.

> Exception exculsion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-1793
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1793
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>
> Exception exclusion functionality is not working correctly because of that tests are
failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating
in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions
in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be
in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message