samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato Marroquín Mogrovejo <renatoj.marroq...@gmail.com>
Subject Re: How to create partitions from input
Date Mon, 03 Nov 2014 09:38:33 GMT
Thanks for the help Yang!
I think I understand more things now, but I also have more questions :)

1) In the task.inputs option you told me that I should use something like
"ordersystem.order", I guess I have to use ordersystem because of the
factory name "OrderSystemFactory"? I tried using "order.order" but I got an
<o.a.s.s.SystemConsumersException: can't register order's consumer> So I
imagine there is a naming convention for classes, inputs, and factories?
2) I tried using the "ordersystem.order" name but I kept on getting the
NoPartitions exception
3) The BlockingEnvelopeMap has a put method to put the incoming messages
into the partition, and I am putting the table values on the start method,
is this a bad practice?
<
https://github.com/renato2099/hello-samza/blob/master/samza-wikipedia/src/main/java/samza/examples/order/system/OrderConsumer.java
>
4) WikipediaFeedStreamTask takes an envelope object and puts that to Kafka,
but where does it get the envelope from? the consumer right?

Thanks again Yang!


Renato M.

2014-11-03 3:18 GMT+01:00 Yan Fang <yanfang724@gmail.com>:

> Hi Renato,
>
> on the hello-samza example for each new incoming message which belongs
> to a channel, a new partition is created, right?
>
> -- In WikipediaFeed job of hello-samza, each channel actually is treated as
> one stream of the Wikipedia System. Each stream has on partition 0. This is
> the code:
> SystemStreamPartition systemStreamPartition = new
> SystemStreamPartition(systemName,
> event.getChannel(), new Partition(0);
>
> So in your code, I think each table could be thought as one stream, with
> only partition 0. So it will be like task.input=ordersystem.tableName
>
> -- Then you maybe confused by what happened in Kafka. All the messages sent
> to  wikipedia-raw topic is done in WikipediaFeedStreamTask
> <
> https://github.com/renato2099/hello-samza/blob/master/samza-wikipedia/src/main/java/samza/examples/wikipedia/task/WikipediaFeedStreamTask.java
> >
> .
>
> Exception in thread "main" org.apache.samza.SamzaException: No partitions
> for this task. Can't run a task without partition assignments. It's likely
> that the partition manager for this system doesn't know about the stream
> you're trying to read.
> at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:77)
> at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
>
> -- Is the stream registered to the system?
> SystemStreamPartition systemStreamPartition = new
> SystemStreamPartition(systemName,
> "order", new Partition(0);
> Is the "order" registered ? such as task.input=ordersystem.order ?
>
> Let me know if you have other questions.
>
> Thanks,
>
> Fang, Yan
> yanfang724@gmail.com
> +1 (206) 849-4108
>
> On Sun, Nov 2, 2014 at 4:56 AM, Renato Marroquín Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
> > Hello Samza team,
> >
> >
> > I am trying to modify the hello-samza example application to replay
> events
> > which are in a table. But I am having some troubles.
> > So on the hello-samza example for each new incoming message which belongs
> > to a channel, a new partition is created, right?
> > Now in my case, how (where) do I create these partitions? I create them
> in
> > [1] but I am almost sure that is wrong because I keep getting the
> exception
> > saying that there are no partitions for this task. I mean ideally I would
> > like to create partitions based on the keys I am reading from the table.
> > Could anybody help me on this task please? Many thanks in advance!
> >
> >
> > Renato M.
> >
> >
> > Exception in thread "main" org.apache.samza.SamzaException: No partitions
> > for this task. Can't run a task without partition assignments. It's
> likely
> > that the partition manager for this system doesn't know about the stream
> > you're trying to read.
> > at
> org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:77)
> > at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
> >
> >
> > [1]
> >
> >
> https://github.com/renato2099/hello-samza/blob/master/samza-wikipedia/src/main/java/samza/examples/order/system/OrderConsumer.java#L47
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message