flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Question on Storm Split Streams
Date Wed, 14 Oct 2015 20:10:01 GMT
SplitStreamKeySelector was build for TupleX output type only
(FlinkTopologyBuilder never used primitive or POJO types). So splitting
a POJO type stream is currently not supported by the Storm layer. And
therefore, there is also no test for it.

It would not be too complicate to add this feature.

-Matthias


On 10/14/2015 09:07 PM, Stephan Ewen wrote:
> I am running into some issues with the Storm Compatibility layer when
> dealing with split streams.
> Specifically, the situation tested in
> "FlinkTopologyBuilderTest.testFieldsGroupingOnMultipleSpoutOutputStreams()"
> 
> The topology builder creates a SplitStreamKeySelector, which internally
> uses an array key selector. The type of the stream is, however, not
> "array", but "SplitStreamType".
> 
> With my changes to the KeyedStream and the state handling, there are now
> more checks for consistency, and this now throws an exception. From an
> initial look, it seems absolutely correct that this fails, because it
> attempts to build a program where a POJO stream is accessed as an array
> selector (generically via the java.util.Array class at runtime), which
> would fail.
> 
> Is this a bug in the Storm API and simply is untested at runtime (client
> time can only produce such a program in the first place because all types
> are raw and no generic checks can be performed), or is there something else
> going on implicitly behind the scenes?
> 
> Thanks for helping me out,
> Stephan
> 


Mime
View raw message