samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navina Ramesh" <nram...@linkedin.com>
Subject Re: Review Request 34974: SAMZA-676: implement broadcast stream
Date Thu, 20 Aug 2015 22:56:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34974/#review96013
-----------------------------------------------------------

Ship it!


I am fine if we are not adding the isBroadcast flag. We can add that if there is enough user
interest.


samza-core/src/main/java/org/apache/samza/config/TaskConfigJava.java (line 53)
<https://reviews.apache.org/r/34974/#comment151212>

    Broadcast sounds much better! :)


- Navina Ramesh


On Aug. 20, 2015, 8:13 p.m., Yan Fang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34974/
> -----------------------------------------------------------
> 
> (Updated Aug. 20, 2015, 8:13 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-676
>     https://issues.apache.org/jira/browse/SAMZA-676
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> 1. added offsetComparator method in SystemAdmin Interface
> 
> 2. added "task.global.inputs" config
> 
> 3. rewrote Grouper classes using Java; allows to assign global streams during grouping
> 
> 4. used LinkedHashSet instead of HashSet in CoordinatorStreamSystemConsumer to preserve
messages order
> 
> 5. added taskNames to the offsets in OffsetManager
> 
> 6. allowed to assign one SSP to multiple taskInstances
> 
> 7. skipped already-processed messages in RunLoop
> 
> 8. unit tests for all changes
> 
> 
> Diffs
> -----
> 
>   checkstyle/import-control.xml aaa235a 
>   docs/learn/documentation/versioned/container/samza-container.md 9f46414 
>   docs/learn/documentation/versioned/jobs/configuration-table.html 8177fe5 
>   samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java a920a10 
>   samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java
63a1666 
>   samza-core/src/main/java/org/apache/samza/config/TaskConfigJava.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartition.java
PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartitionFactory.java
PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.java
PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartitionFactory.java
PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 20e5d26 
>   samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4 
>   samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 24da35f 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 85b012b 
>   samza-core/src/main/scala/org/apache/samza/container/TaskInstance.scala c5a5ea5 
>   samza-core/src/main/scala/org/apache/samza/container/TaskInstanceMetrics.scala 9dc7051

>   samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupByPartition.scala
44e95fc 
>   samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.scala
3c0acad 
>   samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala
c29853d 
>   samza-core/src/test/java/org/apache/samza/config/TestTaskConfigJava.java PRE-CREATION

>   samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupByPartition.java
PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.java
PRE-CREATION 
>   samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala daa5eab

>   samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala b9d9e73 
>   samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 4db6d5c

>   samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 7caad28

>   samza-core/src/test/scala/org/apache/samza/container/grouper/stream/GroupByTestBase.scala
a14169b 
>   samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupByPartition.scala
74daf72 
>   samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.scala
deb3895 
>   samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala d47de7d

>   samza-elasticsearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemAdmin.java
1f5751e 
>   samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala c18e90d

>   samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-text.properties b590e29 
>   samza-hdfs/src/test/resources/samza-hdfs-test-batch-job.properties ab90548 
>   samza-hdfs/src/test/resources/samza-hdfs-test-job-text.properties 9df1397 
>   samza-hdfs/src/test/resources/samza-hdfs-test-job.properties e69de29 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala 4391e24

>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumer.scala
de00320 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 1629035

>   samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemConsumer.scala
2a84328 
>   samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java f23b8f9

>   samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
6f67cf5 
> 
> Diff: https://reviews.apache.org/r/34974/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Yan Fang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message