samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Zuljevic" <>
Subject Re: Review Request 36163: SAMZA-690: changelog topic creation should not be in the container code
Date Wed, 05 Aug 2015 13:46:56 GMT

> On July 27, 2015, 7:36 a.m., Yi Pan (Data Infrastructure) wrote:
> > The code LGTM. For testing, if we can verify this fix w/ a stateful StreamTask w/
changelog enabled with some partition numbers that are different from the default auto-creation
partition number (i.e. 8) in Kafka, it would be good. The integration test suite in samza-test
should be a good place to add the test there. Try following the steps in samza-test/src/main/config/join/README
and run the integration test. The joiner task has a changelog configured with partition number
of 2. You can verify the test passed w/ your fix.

Hi Yi, sorry for bothering you so much with this task : ) I'll just write down what I managed
to do regarding integration tests:

1. I ran integration tests via Zopkio and they all finished successfully.
2. I ran the integration per guide in samza-test/src/main/config/join/README and I suspect
they ran successfully, since none of them had an abnormal final status. I also ran the failure
tests (albeit after some limited fidling with the python scripts involved).
3. I ran "./gradlew clean build" (which runs TestStatefulTask). It finished with a STANDARD_ERROR,
which I assume is a good thing, but here is the output, just in case:

What I suspect are the next (possible) steps:

1. Create integration tests to be used with Zopkio. Here I am uncertain how I would kill/stop
Samza task to verify that changelog stream is being consumed properly.
2. Create another set of tasks similar to Checker/Emitter/Joiner/Watcher. I believe this is
unnecessary since they have their changelogs and their restartability is being tested. Of
course, I might be wrong.
3. Add another test similar to TestStatefulTask.
   a. Or add num.partitions param to TestStatefulTask.
4. None of the above : )
Again, I am very sorry for relying on you this much, but I'm really unclear on how to proceed
regarding this.

- Robert

This is an automatically generated e-mail. To reply, visit:

On July 9, 2015, 2:39 p.m., Robert Zuljevic wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> -----------------------------------------------------------
> (Updated July 9, 2015, 2:39 p.m.)
> Review request for samza.
> Repository: samza
> Description
> -------
> Removed trailing whitespaces
> Diffs
> -----
>   samza-api/src/main/java/org/apache/samza/system/ 7a588ebc99b5f07d533e48e10061a3075a63665a

>   samza-api/src/main/java/org/apache/samza/util/
>   samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 8ee034a642a13af1b0fdb4ebbb3b2592bb8e2be1

>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala aeba61a95371faaba23c97d896321b8d95467f87

>   samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala
>   samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 8d54c4639fc226b34e64915935c1d90e5917af2e

>   samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala d9ae187c7707673fe15c8cb7ea854e02c4a89a54

>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala 35086f54f526d5d88ad3bc312b71fce40260e7c6

>   samza-test/src/main/java/org/apache/samza/system/mock/ b063366f0f60e401765a000fa265c59dee4a461e

>   samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
> Diff:
> Testing
> -------
> I wasn't really sure what kind of test (unit test / integration test) I should make here,
so any pointers would be greatly appreaciated! I tested the change with the unit/integration
tests already available.
> Thanks,
> Robert Zuljevic

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message