datafu-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ohad Raviv (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DATAFU-148) Setup Spark sub-project
Date Mon, 21 Jan 2019 09:15:00 GMT

    [ https://issues.apache.org/jira/browse/DATAFU-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16747771#comment-16747771
] 

Ohad Raviv commented on DATAFU-148:
-----------------------------------

all this new code should work. we also added unitests to all the methods.

about pyspark, all the functionallity runs through UDFs and dataframe operations so in principle
it should be easy expoure to pyspark, but we need to add stub methods in python for that.

we have this functionality in our private code base but it's not clean enough yet to use it.
hope we will get to it soon.

> Setup Spark sub-project
> -----------------------
>
>                 Key: DATAFU-148
>                 URL: https://issues.apache.org/jira/browse/DATAFU-148
>             Project: DataFu
>          Issue Type: New Feature
>            Reporter: Eyal Allweil
>            Assignee: Eyal Allweil
>            Priority: Major
>         Attachments: patch.diff, patch.diff
>
>
> Create a skeleton Spark sub project for Spark code to be contributed to DataFu



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message