Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Date: Mon, 2 Apr 2012 14:49:23 +0000 (UTC)
From: "John Vines (Created) (JIRA)" <jira@apache.org>
To: dev@accumulo.apache.org
Message-ID: 
 <1140816865.672.1333378163888.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (ACCUMULO-508) Multi-range input format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Multi-range input format
------------------------

                 Key: ACCUMULO-508
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-508
             Project: Accumulo
          Issue Type: New Feature
          Components: client
            Reporter: John Vines
             Fix For: 1.5.0


Maybe for 1.4.1.

Our current input format will always apply one range (potentially split at tablet boundaries) per mapper. This is great for situations where you have a few larger ranges. However, there is a potential use case for many small ranges. Aside from the problem with a large job configuration (ACCUMULO-507), this will result in a LOT of mappers doing very little work. We should have an expanded input format which will bundle ranges together to a single mapper, ideally while trying to maintain locality. This will optimize jobs with a lot of ranges by reducing the amount of mapper overhead involved. I think very little will change with the RecordReader. The onus should still go to the end user to detect when a range change has been made (via Key change), so it will still emit Key/Value pairs, just like the regular input format.

This could possibly be extended to the whole row input format as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira