samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman" <initialcont...@gmail.com>
Subject Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza
Date Sun, 14 Jun 2015 22:17:48 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35445/
-----------------------------------------------------------

Review request for samza.


Repository: samza


Description
-------

SAMZA-693: Very basic HDFS Producer service for Samza


Diffs
-----

  build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala PRE-CREATION 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala PRE-CREATION

  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemFactory.scala PRE-CREATION

  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala PRE-CREATION

  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducerMetrics.scala PRE-CREATION

  samza-hdfs/src/test/org/apache/samza/system/hdfs/TestHdfsSystemProducer.scala PRE-CREATION

  samza-hdfs/src/test/resources/samza-hdfs-test-job.properties PRE-CREATION 
  settings.gradle bb07a3b84b14dcef94da1bb166eab6aa3d0026bb 

Diff: https://reviews.apache.org/r/35445/diff/


Testing
-------

New unit test, but it's fairly rudimentary. Passes "./gradlew test" and "./gradlew check"

This only supplies an HDFS Producer, and this producer only writes SequenceFiles of ByteWriteables
so far. If the patch were accepted as-is, I'd suggest future tickets for a matching HDFS Consumer,
and a pluggable set of output formats, configurable via HdfsConfig settings.

On the upside, this patch has been tested on a real cluster with real data, using several
serdes, with good results.


Thanks,

Eli Reisman


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message