spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Erlandson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2315) drop, dropRight and dropWhile which take RDD input and return RDD
Date Wed, 30 Jul 2014 14:03:38 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14079300#comment-14079300
] 

Erik Erlandson commented on SPARK-2315:
---------------------------------------

Updated the PR with a proper lazy-transform implementation:
http://erikerlandson.github.io/blog/2014/07/29/deferring-spark-actions-to-lazy-transforms-with-the-promise-rdd/


> drop, dropRight and dropWhile which take RDD input and return RDD
> -----------------------------------------------------------------
>
>                 Key: SPARK-2315
>                 URL: https://issues.apache.org/jira/browse/SPARK-2315
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>            Reporter: Erik Erlandson
>              Labels: features
>
> Last time I loaded in a text file, I found myself wanting to just skip the first element
as it was a header.     I wrote candidate methods drop, dropRight and dropWhile to satisfy
this kind of need:
> val txt = sc.textFile("text_with_header.txt")
> val data = txt.drop(1)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message