datafu-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eyal Allweil (Jira)" <j...@apache.org>
Subject [jira] [Updated] (DATAFU-154) Spark Explode Array method
Date Thu, 08 Oct 2020 18:59:00 GMT

     [ https://issues.apache.org/jira/browse/DATAFU-154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eyal Allweil updated DATAFU-154:
--------------------------------
    Description: 
Spark has an [explode|https://spark.apache.org/docs/latest/api/sql/index.html#explode] method
that divides an array into multiple rows. It is sometimes convenient to divide an array into
multiple columns.


h5. For example, for input:


|label|sentence_arr |
|0.0 |[Hi, I heard, about, Spark] |
|0.0 |[I wish, Java, could use, case classes] |
|1.0 |[Logistic, regression, models, are neat]|


h5. the output could be

|label|sentence_arr |token0 |token1 |token2 |token3 |
|0.0 |[Hi, I heard, about, Spark] |Hi |I heard |about |Spark |
|0.0 |[I wish, Java, could use, case classes] |I wish |Java |could use|case classes|
|1.0 |[Logistic, regression, models, are neat]|Logistic|regression|models |are neat |


 

  was:
Spark has an [explode|https://spark.apache.org/docs/latest/api/sql/index.html#explode] method
that divides an array into multiple rows. It is sometimes convenient to divide an array into
multiple columns.

 

For example, for input

+-----+----------------------------------------+
|label|sentence_arr |
+-----+----------------------------------------+
|0.0 |[Hi, I heard, about, Spark] |
|0.0 |[I wish, Java, could use, case classes] |
|1.0 |[Logistic, regression, models, are neat]|
+-----+----------------------------------------+

 

the output could be

+-----+----------------------------------------+--------+----------+---------+------------+
|label|sentence_arr |token0 |token1 |token2 |token3 |
+-----+----------------------------------------+--------+----------+---------+------------+
|0.0 |[Hi, I heard, about, Spark] |Hi |I heard |about |Spark |
|0.0 |[I wish, Java, could use, case classes] |I wish |Java |could use|case classes|
|1.0 |[Logistic, regression, models, are neat]|Logistic|regression|models |are neat |
+-----+----------------------------------------+--------+----------+---------+------------+

 


> Spark Explode Array method
> --------------------------
>
>                 Key: DATAFU-154
>                 URL: https://issues.apache.org/jira/browse/DATAFU-154
>             Project: DataFu
>          Issue Type: New Feature
>         Environment: {noformat}
> *no* further _formatting_ is done here{noformat}
> {code:java}
>  
> {code}
> {code:java}
>  
> {code}
>            Reporter: Eyal Allweil
>            Priority: Major
>
> Spark has an [explode|https://spark.apache.org/docs/latest/api/sql/index.html#explode]
method that divides an array into multiple rows. It is sometimes convenient to divide an array
into multiple columns.
> h5. For example, for input:
> |label|sentence_arr |
> |0.0 |[Hi, I heard, about, Spark] |
> |0.0 |[I wish, Java, could use, case classes] |
> |1.0 |[Logistic, regression, models, are neat]|
> h5. the output could be
> |label|sentence_arr |token0 |token1 |token2 |token3 |
> |0.0 |[Hi, I heard, about, Spark] |Hi |I heard |about |Spark |
> |0.0 |[I wish, Java, could use, case classes] |I wish |Java |could use|case classes|
> |1.0 |[Logistic, regression, models, are neat]|Logistic|regression|models |are neat |
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message