Mailing-List: contact hive-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hive-dev@hadoop.apache.org
Message-ID: <990182374.1231704779582.JavaMail.jira@brutus>
Date: Sun, 11 Jan 2009 12:12:59 -0800 (PST)
From: "Joydeep Sen Sarma (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Subject: [jira] Created: (HIVE-224) implement lfu based flushing policy for
 map side aggregates
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

implement lfu based flushing policy for map side aggregates
-----------------------------------------------------------

                 Key: HIVE-224
                 URL: https://issues.apache.org/jira/browse/HIVE-224
             Project: Hadoop Hive
          Issue Type: Improvement
            Reporter: Joydeep Sen Sarma


currently we flush some random set of rows when the map side hash table approaches memory limits.

we have discussed a strategy of flushing hash table entries that have the been seen the least number of times (effectively LFU flushing strategy). This will be very effective at reducing the amount of data sent from map to reduce step - as well as reduce the chances for any skews.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.