pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Lindeman (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2619) HBaseStorage constructs a Scan with cacheBlocks = false
Date Tue, 27 Mar 2012 19:55:29 GMT

    [ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239872#comment-13239872
] 

Andy Lindeman commented on PIG-2619:
------------------------------------

My understanding is that the block cache is not very useful (and actually harmful because
of the wasted cycles populating it and evicting it) when scanning through large numbers of
rows that will only be needed once in a map-reduce job.
                
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
>                 Key: PIG-2619
>                 URL: https://issues.apache.org/jira/browse/PIG-2619
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Andy Lindeman
>              Labels: hbase
>         Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce
jobs should disable cache blocks.
>     scan.setCacheBlocks(false);  // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message