incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills" <jwi...@cloudera.com>
Subject Re: Review Request: Adding Sources for NLine and KeyValueText InputFormats
Date Sun, 02 Dec 2012 22:10:38 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8215/
-----------------------------------------------------------

(Updated Dec. 2, 2012, 10:10 p.m.)


Review request for crunch.


Description
-------

Added support for the NLine and KeyValueText InputFormats to the o.a.c.io.text package. This
completes Crunch's support for the InputFormats that ship as part of hadoop-client.

In the process, I refactored the ReaderFactory code that is used to read SequenceFiles and
text files during materialization to eliminate some duplicate code.


This addresses bug CRUNCH-119.
    https://issues.apache.org/jira/browse/CRUNCH-119


Diffs (updated)
-----

  crunch/src/it/java/org/apache/crunch/io/CompositePathIterableIT.java 796b821 
  crunch/src/it/java/org/apache/crunch/io/NLineInputIT.java PRE-CREATION 
  crunch/src/it/java/org/apache/crunch/io/TextFileTableIT.java PRE-CREATION 
  crunch/src/main/java/org/apache/crunch/io/ReadableSource.java 73a13a3 
  crunch/src/main/java/org/apache/crunch/io/avro/AvroFileReaderFactory.java 6f21dd2 
  crunch/src/main/java/org/apache/crunch/io/avro/AvroFileSource.java 2226556 
  crunch/src/main/java/org/apache/crunch/io/impl/AutoClosingIterator.java d58f290 
  crunch/src/main/java/org/apache/crunch/io/impl/FileTableSourceImpl.java f6e8f1d 
  crunch/src/main/java/org/apache/crunch/io/seq/SeqFileReaderFactory.java ad1b81b 
  crunch/src/main/java/org/apache/crunch/io/seq/SeqFileSource.java e8f3dcf 
  crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableReaderFactory.java 20c749a 
  crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableSource.java 56ed985 
  crunch/src/main/java/org/apache/crunch/io/text/LineParser.java PRE-CREATION 
  crunch/src/main/java/org/apache/crunch/io/text/NLineFileSource.java PRE-CREATION 
  crunch/src/main/java/org/apache/crunch/io/text/TextFileReaderFactory.java a0c48e0 
  crunch/src/main/java/org/apache/crunch/io/text/TextFileSource.java ee51c04 
  crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSource.java PRE-CREATION 
  crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSourceTarget.java PRE-CREATION

  crunch/src/main/java/org/apache/crunch/io/text/TextFileTarget.java c7e06d3 
  crunch/src/test/java/org/apache/crunch/io/avro/AvroFileReaderFactoryTest.java 66863ba 

Diff: https://reviews.apache.org/r/8215/diff/


Testing
-------

Integration tests that use the new formats.


Thanks,

Josh Wills


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message