hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-965) PERFORMANCE: optimize common case in matches (PORegex)
Date Thu, 17 Sep 2009 18:19:57 GMT

    [ https://issues.apache.org/jira/browse/PIG-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756673#action_12756673
] 

Thejas M Nair commented on PIG-965:
-----------------------------------

Hive like clause implementation is here - http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop
/hive/ql/udf/UDFLike.java?revision=802066&view=markup

I ran simple tests with a simple java program to see the impact of these optimizations. Optimization
1 reduces runtime to 1/2, optimization 2 reduces runtime to 1/4 . 

{code}
        int matches =0;
        int tot = 0;
        String prefix = "123";
        Pattern p =  Pattern.compile("123.*");
        while((str = in.readLine()) != null ){



            //without proposed optimizations
            //test setups 1 and 2 took 9secs, 126 secs respectively
//            if(str.matches("123.*"))
//                matches++;



            // with optimization 1
//            test sestups 1, 2 took  4, 57 secs respectively
//            if((p.matcher(str).matches()))
//                matches++;
            

            // with optimization 1
//            test sestups 1, 2 took  2.5, 25 secs respectively
            //takes 2.5, 25 secs
//            int len = prefix.length();
//            boolean matched = true;
//            for(int i=0; i<len; i++){
//                if(prefix.charAt(i) != str.charAt(i)){
//                    matched = false;
//                    break;
//                }
//            }
//            if(matched)
//                matches++;

            tot++;
        }
       }
        System.out.println("matches " + matches + " tot " + tot);
{code}

> PERFORMANCE: optimize common case in matches (PORegex)
> ------------------------------------------------------
>
>                 Key: PIG-965
>                 URL: https://issues.apache.org/jira/browse/PIG-965
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Thejas M Nair
>
> Some frequently seen use cases of 'matches' comparison operator have follow properties
-
> 1. The rhs is a constant string . eg "c1 matches 'abc%' "
> 2. Regexes such that look for matching prefix , suffix etc are very common. eg - "abc%',
"%abc", '%abc%' 
> To optimize for these common cases , PORegex.java can be changed to -
> 1. Compile the pattern (rhs of matches) re-use it if the pattern string has not changed.

> 2. Use string comparisons for simple common regexes (in 2 above).
> The implementation of Hive like clause uses similar optimizations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message