Return-Path: 
 <user-return-2464-apmail-flink-user-archive=flink.apache.org@flink.apache.org>
X-Original-To: apmail-flink-user-archive@minotaur.apache.org
Delivered-To: apmail-flink-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 306B618560
	for <apmail-flink-user-archive@minotaur.apache.org>;
 Sun, 23 Aug 2015 19:41:04 +0000 (UTC)
Received: (qmail 90063 invoked by uid 500); 23 Aug 2015 19:40:52 -0000
Delivered-To: apmail-flink-user-archive@flink.apache.org
Received: (qmail 89986 invoked by uid 500); 23 Aug 2015 19:40:52 -0000
Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@flink.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@flink.apache.org>
List-Post: <mailto:user@flink.apache.org>
List-Id: <user.flink.apache.org>
Reply-To: user@flink.apache.org
Delivered-To: mailing list user@flink.apache.org
Received: (qmail 89976 invoked by uid 99); 23 Aug 2015 19:40:52 -0000
Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2015 19:40:52 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org)
 with ESMTP id 505D0ED00B
	for <user@flink.apache.org>; Sun, 23 Aug 2015 19:40:52 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 3.487
X-Spam-Level: ***
X-Spam-Status: No, score=3.487 tagged_above=-999 required=6.31
	tests=[DKIM_ADSP_CUSTOM_MED=0.001, NML_ADSP_CUSTOM_MED=1.2,
	SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001, URI_HEX=1.313]
	autolearn=disabled
Received: from mx1-eu-west.apache.org ([10.40.0.8])
	by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new,
 port 10024)
	with ESMTP id n5OYzddQZSsw for <user@flink.apache.org>;
	Sun, 23 Aug 2015 19:40:43 +0000 (UTC)
Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43])
	by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with
 ESMTP id 1E7E4205B2
	for <user@flink.apache.org>; Sun, 23 Aug 2015 19:40:43 +0000 (UTC)
Received: from mjoe.nabble.com (unknown [162.253.133.57])
	by mwork.nabble.com (Postfix) with ESMTP id 5137826EE20E
	for <user@flink.apache.org>; Sun, 23 Aug 2015 12:41:05 -0700 (PDT)
Date: Sun, 23 Aug 2015 12:36:02 -0700 (PDT)
From: defstat <defstat@gmail.com>
To: user@flink.apache.org
Message-ID: <1440358562442-2494.post@n4.nabble.com>
Subject: Statefull computation
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi. I am struggling the past few days to find a solution on the following
problem, using Apache Flink: 

I have a stream of vectors, represented by files in a local folder. After a
new text file is located using DataStream<String> text =
env.readFileStream(...), I transform (flatMap), the Input into a
SingleOutputStreamOperator<Tuple2&lt;String, Integer>, ?>, with the Integer
being the score coming from a scoring function. 

I want to persist a global HashMap containing the top-k vectors, using their
scores. I approached the problem using a statefull transformation. 
1. The first problem I have is that the HashMap retains per-sink data (so
for each thread of workers, one HashMap of data). How can I make that a
Global collection 

2. Using Apache Spark, I made that possible by 
JavaPairDStream<String, Integer> stateDstream =
tuples.updateStateByKey(updateFunction); 

and then making transformations on the stateDstream. Is there a way I can
get the same functionality using FLink? 

Thanks in advance! 



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Statefull-computation-tp2494.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.