hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-39) BufferedPositionedInputStream drastically reduces read performance because it doesn't override read([], o, l) in InputStream
Date Mon, 10 Dec 2007 21:57:43 GMT

    [ https://issues.apache.org/jira/browse/PIG-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550206
] 

Olga Natkovich commented on PIG-39:
-----------------------------------

I incorporated the change and ran performance tests. Unfortunately, I did not see any change
in performance. By looking at Hadoop, code, I think they already buffering the data, so our
code just going against data cached in memory.

I am still going to commit the patch since this is a bug.

> BufferedPositionedInputStream drastically reduces read performance because it doesn't
override read([], o, l) in InputStream
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-39
>                 URL: https://issues.apache.org/jira/browse/PIG-39
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>         Environment: Java 1.6, Mac OS X 10.5
>            Reporter: Sam Pullara
>
> Simple fix can have a huge effect on performance of certain kinds of PIG programs:
> Index: src/org/apache/pig/impl/io/BufferedPositionedInputStream.java
> ===================================================================
> --- src/org/apache/pig/impl/io/BufferedPositionedInputStream.java	(revision 597597)
> +++ src/org/apache/pig/impl/io/BufferedPositionedInputStream.java	(working copy)
> @@ -49,7 +49,14 @@
>          pos += rc;
>          return rc;
>      }
> -    
> +
> +    @Override
> +    public int read(byte b[], int off, int len) throws IOException {
> +        int read = in.read(b, off, len);
> +        pos += read;
> +        return read;
> +    }
> +
>      /**
>       * Returns the current position in the tracked InputStream.
>       */

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message