Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAC27z=MDp-wLp9x0_vbJyF18c6zp42ZOBGt356tqszbMDWHGFw@mail.gmail.com>
References: 
 <CANZa=GsxwRkdppFS9LjHKVWwQREkPAt-APAZ3e9u2OjC8VixdA@mail.gmail.com>
 <CAGr9p8CE2nDK-0pMgkYx4fWvbMSrmYjnM=eH0d2j1LOBZZ1row@mail.gmail.com>
 <CANZa=Gun50nEG5H2tj3kuvCRwFS_HqF5sjN8BuA88oZHdRBz8g@mail.gmail.com>
 <CANC1h_vxyxvgAAqXcmApBqMVGNhCnHE4qtm1Hu5u1MvkAob_eQ@mail.gmail.com>
 <CANZa=GtxUi=-Lvjh0t6qvRsiiGkpOecEmrdWXt5_oLvSjSuXtA@mail.gmail.com>
 <CAC27z=MDp-wLp9x0_vbJyF18c6zp42ZOBGt356tqszbMDWHGFw@mail.gmail.com>
From: Nick Dimiduk <ndimiduk@gmail.com>
Date: Fri, 6 Nov 2015 09:03:39 -0800
Message-ID: 
 <CANZa=Gs42+X++k5wJxK2-LcJzFBt4pDF+B3UxeVY3Y+8F-vP6g@mail.gmail.com>
Subject: Re: Published test artifacts for flink streaming
To: user@flink.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Promising observation, Till. Is it possible to access Table API's
select and where operators from within such a flatMap?

-n

On Fri, Nov 6, 2015 at 6:19 AM, Till Rohrmann <trohrmann@apache.org> wrote:
> Hi Nick,
>
> I think a flatMap operation which is instantiated with your list of
> predicates should do the job. Thus, there shouldn=E2=80=99t be a need to =
dig deeper
> than the DataStream for the first version.
>
> Cheers,
> Till
>
>
> On Fri, Nov 6, 2015 at 3:58 AM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>>
>> Thanks Stephan, I'll check that out in the morning. Generally speaking, =
it
>> would be great to have some single-jvm example tests for those of us get=
ting
>> started. Following the example of WindowingIntegrationTest is mostly
>> working, though reusing my single sink instance with its static collecti=
on
>> results in non-deterministic results; there appears to be a race between
>> instances clearing the collection in their open method and the runtime
>> returning the collection to my test harness.
>>
>> I'd also appreciate some guidance on stream composition. It's nice to us=
e
>> the fluent API when exploring data in a shell, but it seems to me like t=
hat
>> API is cumbersome when composing data pipelines of reusable partials. Or
>> maybe I'm doing it all wrong... Hence the request for more examples :)
>>
>> While I'm asking, how might you model this: I have a set of predicates I=
'd
>> like to flatMap over a stream. An input item should be compared vs every
>> predicate (basically, I want a Clojure juxt of predicates over each stre=
am
>> element). Imagine those predicates expressed as where clauses via the Ta=
ble
>> API. Say I have hundreds of thousands of these predicates to run over ev=
ery
>> stream event. Is the java client API rich enough to express such a flow,=
 or
>> should I examine something lower than DataStream?
>>
>> Thanks a lot, and sorry for all the newb questions.
>> -n
>>
>>
>> On Thursday, November 5, 2015, Stephan Ewen <sewen@apache.org> wrote:
>>>
>>> Hey!
>>>
>>> There is also a collect() sink in the "flink-streaming-contrib" project=
,
>>> see here:
>>> https://github.com/apache/flink/blob/master/flink-contrib/flink-streami=
ng-contrib/src/main/java/org/apache/flink/contrib/streaming/DataStreamUtils=
.java
>>>
>>> It should work well locally for testing. In that case you can write a
>>> program as usual an use "DataStreamUtils.collect(stream)", so you need =
to
>>> stop reading it once you know the stream is exhausted...
>>>
>>> Stephan
>>>
>>>
>>> On Thu, Nov 5, 2015 at 10:38 PM, Nick Dimiduk <ndimiduk@gmail.com> wrot=
e:
>>>>
>>>> Hi Robert,
>>>>
>>>> It seems "type" was what I needed. This it also looks like the test
>>>> jar has an undeclared dependency. In the end, the following allowed me
>>>> to use TestStreamEnvironment for my integration test. Thanks a lot!
>>>>
>>>> -n
>>>>
>>>>     <dependency>
>>>>       <groupId>org.apache.flink</groupId>
>>>>       <artifactId>flink-streaming-core</artifactId>
>>>>       <version>${flink.version}</version>
>>>>       <type>test-jar</type>
>>>>       <scope>test</scope>
>>>>     </dependency>
>>>>     <dependency>
>>>>       <groupId>org.apache.flink</groupId>
>>>>       <artifactId>flink-test-utils</artifactId>
>>>>       <version>${flink.version}</version>
>>>>       <scope>test</scope>
>>>>     </dependency>
>>>>
>>>> On Thu, Nov 5, 2015 at 12:47 PM, Robert Metzger <rmetzger@apache.org>
>>>> wrote:
>>>> > Hi Nick,
>>>> >
>>>> > we are usually publishing the test  artifacts. Can you try and repla=
ce
>>>> > the
>>>> > <classifier> tag by <type>test-jar<type>:
>>>> >
>>>> > <dependency>
>>>> >    <groupId>org.apache.flink</groupId>
>>>> >    <artifactId>flink-streaming-core</artifactId>
>>>> >    <version>${flink.version}</version>
>>>> >    <type>test-jar</type>
>>>> >    <scope>test</scope>
>>>> > </dependency>
>>>> >
>>>> >
>>>> > On Thu, Nov 5, 2015 at 8:18 PM, Nick Dimiduk <ndimiduk@gmail.com>
>>>> > wrote:
>>>> >>
>>>> >> Hello,
>>>> >>
>>>> >> I'm attempting integration tests for my streaming flows. I'd like t=
o
>>>> >> produce an input stream of java objects and sink the results into a
>>>> >> collection for verification via JUnit asserts.
>>>> >> StreamExecutionEnvironment provides methods for the former, however=
,
>>>> >> how to achieve the latter is not evident based on my internet
>>>> >> searching. I think I've found a solution in the TestStreamEnvironme=
nt
>>>> >> class, ie, as used by WindowingIntegrationTest. However, this class
>>>> >> appears to be packaged in the flink-streaming-core test artifact,
>>>> >> which is not published to maven.
>>>> >>
>>>> >> For reference, this is the maven dependency stanza I'm using. Pleas=
e
>>>> >> let me know if I've got it wrong.
>>>> >>
>>>> >> Thanks,
>>>> >> Nick
>>>> >>
>>>> >>     <dependency>
>>>> >>       <groupId>org.apache.flink</groupId>
>>>> >>       <artifactId>flink-streaming-core</artifactId>
>>>> >>       <version>${flink.version}</version>
>>>> >>       <classifier>test</classifier>
>>>> >>       <scope>test</scope>
>>>> >>     </dependency>
>>>> >
>>>> >
>>>
>>>
>