Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Sun, 25 Sep 2016 22:18:46 -0700 (MST)
From: backtrack5 <sornalgm@live.com>
To: user@spark.apache.org
Message-ID: <1474867126940-27792.post@n3.nabble.com>
In-Reply-To: <1474656968668-27786.post@n3.nabble.com>
References: <1474471173266-27770.post@n3.nabble.com> <1474656968668-27786.post@n3.nabble.com>
Subject: Re: spark stream based deduplication
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
archived-at: Mon, 26 Sep 2016 05:18:53 -0000

Thank you @markcitizen . What I want to achieve is , say for an example

My historic rdd has
(Hash1, recordid1)
(Hash2,recordid2)

And in the new steam I have the following,
(Hash3, recordid3)
(Hash1,recordid5)

In this above scenario,
1) for recordid5,I should get recordid5 is duplicate of recordid1.
2) the new values (hash3,recordid3) should added in the historic rdd.

And I have one another question to ask,
If the problem crashes at any point, is it possible to recover that historic
rdd ?
Can i use state full stream. ?


--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-stream-based-deduplication-tp27770p27792.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org