flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Flink Stream: collect in an array all records within a window
Date Tue, 19 Jan 2016 12:32:33 GMT
What type is your DataStream? It must be DataStream[String] to work with
SimpleStringSchema.

If you have a different type, just implement a customized
SerializationSchema.

-Matthias


On 01/19/2016 11:26 AM, Saiph Kappa wrote:
> When I use SimpleStringSchema I get the error: Type mismatch, expected:
> SerializationSchema[String, Array[Byte]], actual: SimpleStringSchema. I
> think SimpleStringSchema extends SerializationSchema[String], and
> therefore cannot be used as argument of writeToSocket. Can you confirm
> this please?
> 
> s.writeToSocket(host, port.toInt, new SimpleStringSchema())
> 
> 
> Thanks.
> 
> On Tue, Jan 19, 2016 at 10:34 AM, Matthias J. Sax <mjsax@apache.org
> <mailto:mjsax@apache.org>> wrote:
> 
>     There is SimpleStringSchema.
> 
>     -Matthias
> 
>     On 01/18/2016 11:21 PM, Saiph Kappa wrote:
>     > Hi Matthias,
>     >
>     > Thanks for your response. The method .writeToSocket seems to be what I
>     > was looking for. Can you tell me what kind of serialization schema
>     > should I use assuming my socket server receives strings. I have
>     > something like this in scala:
>     >
>     > |val server =newServerSocket(9999)while(true){val s =server.accept()val
>     > in=newBufferedSource(s.getInputStream()).getLines()println(in.next())s.close()}
>     >
>     > |
>     >
>     > Thanks|
>     > |
>     >
>     >
>     >
>     > On Mon, Jan 18, 2016 at 8:46 PM, Matthias J. Sax <mjsax@apache.org <mailto:mjsax@apache.org>
>     > <mailto:mjsax@apache.org <mailto:mjsax@apache.org>>> wrote:
>     >
>     >     Hi Saiph,
>     >
>     >     you can use AllWindowFunction via .apply(...) to get an
>     .collect method:
>     >
>     >     From:
>     >   
>      https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html
>     >
>     >     > // applying an AllWindowFunction on non-keyed window stream
>     >     > allWindowedStream.apply (new
>     >     AllWindowFunction<Tuple2<String,Integer>, Integer, Window>()
{
>     >     >     public void apply (Window window,
>     >     >             Iterable<Tuple2<String, Integer>> values,
>     >     >             Collector<Integer> out) throws Exception {
>     >     >         int sum = 0;
>     >     >         for (value t: values) {
>     >     >             sum += t.f1;
>     >     >         }
>     >     >         out.collect (new Integer(sum));
>     >     >     }
>     >     > });
>     >
>     >     If you consume all those value via an sink, the sink will run
>     an the
>     >     cluster. You can use .writeToSocket(...) as sink:
>     >   
>      https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#data-sinks
>     >
>     >     -Matthias
>     >
>     >
>     >     On 01/18/2016 06:30 PM, Saiph Kappa wrote:
>     >     > Hi,
>     >     >
>     >     > After performing a windowAll() on a DataStream[String], is
>     there any
>     >     > method to collect and return an array with all Strings
>     within a window
>     >     > (similar to .collect in Spark).
>     >     >
>     >     > I basically want to ship all strings in a window to a remote
>     server
>     >     > through a socket, and want to use the same socket connection
>     for all
>     >     > strings that I send. The method .addSink iterates over all
>     >     records, but
>     >     > does the provided function runs on the flink client or on
>     the server?
>     >     >
>     >     > Thanks.
>     >
>     >
> 
> 


Mime
View raw message