samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman" <>
Subject Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza
Date Sun, 14 Jun 2015 22:17:48 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for samza.

Repository: samza


SAMZA-693: Very basic HDFS Producer service for Samza


  build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala PRE-CREATION 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala PRE-CREATION

  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemFactory.scala PRE-CREATION

  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala PRE-CREATION

  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducerMetrics.scala PRE-CREATION

  samza-hdfs/src/test/org/apache/samza/system/hdfs/TestHdfsSystemProducer.scala PRE-CREATION

  samza-hdfs/src/test/resources/ PRE-CREATION 
  settings.gradle bb07a3b84b14dcef94da1bb166eab6aa3d0026bb 



New unit test, but it's fairly rudimentary. Passes "./gradlew test" and "./gradlew check"

This only supplies an HDFS Producer, and this producer only writes SequenceFiles of ByteWriteables
so far. If the patch were accepted as-is, I'd suggest future tickets for a matching HDFS Consumer,
and a pluggable set of output formats, configurable via HdfsConfig settings.

On the upside, this patch has been tested on a real cluster with real data, using several
serdes, with good results.


Eli Reisman

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message