hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Subelsky (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-1475) Large number of FILTER matching clauses causes internal error 2998
Date Tue, 29 Jun 2010 19:39:51 GMT
Large number of FILTER matching clauses causes internal error 2998
------------------------------------------------------------------

                 Key: PIG-1475
                 URL: https://issues.apache.org/jira/browse/PIG-1475
             Project: Pig
          Issue Type: Bug
          Components: grunt
    Affects Versions: 0.6.0
         Environment: Amazon Elastic MapReduce with Pig 0.6 and Hadoop 0.20.2
            Reporter: Mike Subelsky
            Priority: Minor


I'm generating reports using Pig where I only want to report on rows matching a set of regular
expressions, but those regular expressions are pretty numerous.

Pig fails with this internal error when I run FILTER with 500 terms:

2010-06-29 15:46:04,579 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled
internal error. null

This only occurred when I ran my Pig script against Hadoop with the full dataset. When I ran
Pig in local mode, with a smaller sample file, Pig handled the FILTER command just fine.

The workaround has been to split my list into two separate lists of 250 then UNION the results,
but I assume this is something that could be addressed in the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message