atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Dossett (JIRA)" <>
Subject [jira] [Commented] (ATLAS-182) Add data model for Storm topology elements
Date Fri, 16 Oct 2015 16:34:05 GMT


Aaron Dossett commented on ATLAS-182:

Comments about ATLAS-182.patch:

- The code refers to the Storm DAG.  Storm topologies can contain cycles,
so it¹s not literally a DAG.  I don¹t think this has any implications for
the code I see though.

- The type hierarchy looks like NODE -> SPOUT -> BOLT.  I think it would
be better to have SPOUT and BOLT both be subtypes of NODE since SPOUT has
some behaviors and characteristics that a BOLT does not.

- Each NODE also has a set of configurations options associated with it.
These could be primitives (e.g. ³debug=true² or
³namenode=hdfs://²) or more complex configurations such as a
java class describing a file rotation policy for a bolt that writes data
to a file.  I don¹t know if that level of detail is appropriate yet, but
those options are the key metadata about the NODE to reason about its

- Some BOLTS act like ³sinks² (although Storm doesn¹t use that term) in
that the output is outside of the storm topology (e.g. writing to HDFS,
streaming to a Hive table, or enqueuing to Kafka). I don¹t know how that
should be reflected in the model. It could be a NODE with no ³outputs² but
whose configuration options would describe where the data goes, or SINK
could be a datatype in the model.  After typing that out, the first option
sounds better.

- I don¹t believe KAFKA and KAFKA TOPIC need to be first class citizens in
this data model.  A Kafka Spout would be represented as a SPOUT with a
JAVA_CLASS of KafkaSpout and a set of configuration options identifying
brokers, topics, etc.

- Very minor, but the patch places the .java files in addons/hive-bridge/,
I assume that should be addons/storm-bridge/?

> Add data model for Storm topology elements
> ------------------------------------------
>                 Key: ATLAS-182
>                 URL:
>             Project: Atlas
>          Issue Type: Sub-task
>    Affects Versions: 0.6-incubating
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>             Fix For: 0.6-incubating
>         Attachments: ATLAS-182.patch

This message was sent by Atlassian JIRA

View raw message