Return-Path: X-Original-To: apmail-accumulo-dev-archive@www.apache.org Delivered-To: apmail-accumulo-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DCDA710E3B for ; Thu, 21 Nov 2013 05:32:26 +0000 (UTC) Received: (qmail 35553 invoked by uid 500); 21 Nov 2013 05:32:09 -0000 Delivered-To: apmail-accumulo-dev-archive@accumulo.apache.org Received: (qmail 35475 invoked by uid 500); 21 Nov 2013 05:31:54 -0000 Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list dev@accumulo.apache.org Received: (qmail 35423 invoked by uid 99); 21 Nov 2013 05:31:49 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Nov 2013 05:31:49 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 1B7501D3BF4; Thu, 21 Nov 2013 05:31:45 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============7717138602635366004==" MIME-Version: 1.0 Subject: Review Request 15752: ACCUMULO-1854 Persist AccumuloInputFormat information from Configuration into RangeInputSplit From: "Josh Elser" To: "accumulo" , "Josh Elser" Date: Thu, 21 Nov 2013 05:31:45 -0000 Message-ID: <20131121053145.1417.15751@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Josh Elser" X-ReviewGroup: accumulo X-ReviewRequest-URL: https://reviews.apache.org/r/15752/ X-Sender: "Josh Elser" Reply-To: "Josh Elser" X-ReviewRequest-Repository: accumulo --===============7717138602635366004== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15752/ ----------------------------------------------------------- Review request for accumulo. Bugs: ACCUMULO-1854 https://issues.apache.org/jira/browse/ACCUMULO-1854 Repository: accumulo Description ------- The current way that AccumuloInputFormat works requires that the same *exact* Configuration that was used to invoke getSplits() is also provided when createRecordReader() is called on the InputFormat. In practice, notably looking at InputFormat implementations which merge or delegate other InputFormats, this is a bad idea. By serializing the necessary information into the RangeInputSplit from the provided Configuration object in getSplits() we can completely avoid this problem, at the minimal expense of serialization this information into each InputSplit. I tried to implement the changes in such a way that would be backwards compatible. If the information is not provided (is null) in the RangeInputSplit, the RecordReader will still attempt to pull a value from the Configuration object so as to not fail immediately. This should provide a little more flexibility if users have custom code built on top of the AccumuloInputFormat and RangeInputSplit Diffs ----- src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloInputFormat.java 4de131f src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/InputFormatBase.java 8e238f1 src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/RangeInputSplit.java PRE-CREATION src/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloInputFormatTest.java ba647e9 src/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloRowInputFormatTest.java 0673f1b src/core/src/test/java/org/apache/accumulo/core/client/mapreduce/RangeInputSplitTest.java PRE-CREATION src/examples/simple/src/test/java/org/apache/accumulo/examples/simple/filedata/ChunkInputFormatTest.java c31c738 Diff: https://reviews.apache.org/r/15752/diff/ Testing ------- Verified changes work as intended using PigInputFormat (which may delegate to many InputFormats). Added additional unit tests and verified sufficient coverage using cobertura. Thanks, Josh Elser --===============7717138602635366004==--