flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1901) Create sample operator for Dataset
Date Fri, 07 Aug 2015 18:06:46 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662173#comment-14662173
] 

ASF GitHub Bot commented on FLINK-1901:
---------------------------------------

Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/949#issuecomment-128781613
  
    I have worked on this problem before. The idea is to divide the data into blocks and find
the probability of selection of an element from a block.
    Thus, suppose there are blocks B_1, B_2, ..., B_N with probabilities P_1, P_2, ..., P_N,
then you sample k points by first sampling from the distribution {P_1, P_2, ..., P_N} and
find the number of elements you require from each block. After that you select the required
number of points from each block and take a union.
    It is pretty easy to implement in a shared memory system but with a distributed system,
it is hard. I tried the following approach some time before, although didn't quite finish
working on it:
    blockedData = Data -> (block_id, data)
    
    blockNumbers = (block_id, data) -> (block_id, count)
    
    (1...k) -> (list of block ids we'll be sampling from)
    After this, I tried broadcasting the list and selecting the required number of elements
from each block, which can be done quite easily. But what if k is very large?


> Create sample operator for Dataset
> ----------------------------------
>
>                 Key: FLINK-1901
>                 URL: https://issues.apache.org/jira/browse/FLINK-1901
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Theodore Vasiloudis
>            Assignee: Chengxiang Li
>
> In order to be able to implement Stochastic Gradient Descent and a number of other machine
learning algorithms we need to have a way to take a random sample from a Dataset.
> We need to be able to sample with or without replacement from the Dataset, choose the
relative size of the sample, and set a seed for reproducibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message