flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Question on Storm Split Streams
Date Wed, 14 Oct 2015 20:10:01 GMT
SplitStreamKeySelector was build for TupleX output type only
(FlinkTopologyBuilder never used primitive or POJO types). So splitting
a POJO type stream is currently not supported by the Storm layer. And
therefore, there is also no test for it.

It would not be too complicate to add this feature.


On 10/14/2015 09:07 PM, Stephan Ewen wrote:
> I am running into some issues with the Storm Compatibility layer when
> dealing with split streams.
> Specifically, the situation tested in
> "FlinkTopologyBuilderTest.testFieldsGroupingOnMultipleSpoutOutputStreams()"
> The topology builder creates a SplitStreamKeySelector, which internally
> uses an array key selector. The type of the stream is, however, not
> "array", but "SplitStreamType".
> With my changes to the KeyedStream and the state handling, there are now
> more checks for consistency, and this now throws an exception. From an
> initial look, it seems absolutely correct that this fails, because it
> attempts to build a program where a POJO stream is accessed as an array
> selector (generically via the java.util.Array class at runtime), which
> would fail.
> Is this a bug in the Storm API and simply is untested at runtime (client
> time can only produce such a program in the first place because all types
> are raw and no generic checks can be performed), or is there something else
> going on implicitly behind the scenes?
> Thanks for helping me out,
> Stephan

View raw message