samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yan Fang" <yanfang...@gmail.com>
Subject Re: Review Request 34974: SAMZA-676: implement broadcast stream
Date Tue, 16 Jun 2015 21:16:09 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34974/
-----------------------------------------------------------

(Updated June 16, 2015, 9:16 p.m.)


Review request for samza.


Changes
-------

rebase to the master (it was wrong in the previous upload)


Bugs: SAMZA-676
    https://issues.apache.org/jira/browse/SAMZA-676


Repository: samza


Description
-------

1. added offsetComparator method in SystemAdmin Interface

2. added "task.global.inputs" config

3. rewrote Grouper classes using Java; allows to assign global streams during grouping

4. used LinkedHashSet instead of HashSet in CoordinatorStreamSystemConsumer to preserve messages
order

5. added taskNames to the offsets in OffsetManager

6. allowed to assign one SSP to multiple taskInstances

7. skipped already-processed messages in RunLoop

8. unit tests for all changes


Diffs (updated)
-----

  checkstyle/import-control.xml 3374f0c 
  samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 7a588eb 
  samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java
249b8ae 
  samza-core/src/main/java/org/apache/samza/config/TaskConfigJava.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartition.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartitionFactory.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartitionFactory.java
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 20e5d26 
  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4 
  samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala c292ae4 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala cbacd18 
  samza-core/src/main/scala/org/apache/samza/container/TaskInstance.scala c5a5ea5 
  samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupByPartition.scala
44e95fc 
  samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.scala
3c0acad 
  samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala
097f410 
  samza-core/src/test/java/org/apache/samza/config/TestTaskConfigJava.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupByPartition.java
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.java
PRE-CREATION 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 8d54c46 
  samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala 64a5844 
  samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 9fb1aa9 
  samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 7caad28 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/GroupByTestBase.scala
a14169b 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupByPartition.scala
74daf72 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.scala
deb3895 
  samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala d9ae187

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala 35086f5

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumer.scala de00320

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 1629035

  samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemConsumer.scala 2a84328

  samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java b063366 
  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
1e936b4 

Diff: https://reviews.apache.org/r/34974/diff/


Testing
-------


Thanks,

Yan Fang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message