samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yan Fang" <yanfang...@gmail.com>
Subject Re: Review Request 34974: SAMZA-676: implement broadcast stream
Date Thu, 20 Aug 2015 20:13:16 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34974/
-----------------------------------------------------------

(Updated Aug. 20, 2015, 8:13 p.m.)


Review request for samza.


Changes
-------

fixed javadoc malformed
uniformed all the "global" to "broadcast" stream
added missed license


Bugs: SAMZA-676
    https://issues.apache.org/jira/browse/SAMZA-676


Repository: samza


Description
-------

1. added offsetComparator method in SystemAdmin Interface

2. added "task.global.inputs" config

3. rewrote Grouper classes using Java; allows to assign global streams during grouping

4. used LinkedHashSet instead of HashSet in CoordinatorStreamSystemConsumer to preserve messages
order

5. added taskNames to the offsets in OffsetManager

6. allowed to assign one SSP to multiple taskInstances

7. skipped already-processed messages in RunLoop

8. unit tests for all changes


Diffs (updated)
-----

  checkstyle/import-control.xml aaa235a 
  docs/learn/documentation/versioned/container/samza-container.md 9f46414 
  docs/learn/documentation/versioned/jobs/configuration-table.html 8177fe5 
  samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java a920a10 
  samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java
63a1666 
  samza-core/src/main/java/org/apache/samza/config/TaskConfigJava.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartition.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartitionFactory.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartitionFactory.java
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 20e5d26 
  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4 
  samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 24da35f 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 85b012b 
  samza-core/src/main/scala/org/apache/samza/container/TaskInstance.scala c5a5ea5 
  samza-core/src/main/scala/org/apache/samza/container/TaskInstanceMetrics.scala 9dc7051 
  samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupByPartition.scala
44e95fc 
  samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.scala
3c0acad 
  samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala
c29853d 
  samza-core/src/test/java/org/apache/samza/config/TestTaskConfigJava.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupByPartition.java
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.java
PRE-CREATION 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala daa5eab 
  samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala b9d9e73 
  samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 4db6d5c 
  samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 7caad28 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/GroupByTestBase.scala
a14169b 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupByPartition.scala
74daf72 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.scala
deb3895 
  samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala d47de7d

  samza-elasticsearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemAdmin.java
1f5751e 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala c18e90d 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-text.properties b590e29 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job.properties ab90548 
  samza-hdfs/src/test/resources/samza-hdfs-test-job-text.properties 9df1397 
  samza-hdfs/src/test/resources/samza-hdfs-test-job.properties e69de29 
  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala 4391e24

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumer.scala de00320

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 1629035

  samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemConsumer.scala 2a84328

  samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java f23b8f9 
  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
6f67cf5 

Diff: https://reviews.apache.org/r/34974/diff/


Testing
-------


Thanks,

Yan Fang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message