beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From OrielResearch Eila Arich-Landkof <e...@orielresearch.org>
Subject Re: PCollection from DataFrame
Date Wed, 08 Aug 2018 13:34:00 GMT
Hi  Jon,
thank you. will try that.
Best,
Eila


On Wed, Aug 8, 2018 at 9:00 AM, Jon Goodrum <jgoodrum@acvauctions.com>
wrote:

> Hi Eila,
>
>
> You can turn your DataFrame into a list via *df.values.tolist()* and pass
> that into *beam.Create(...)* directly:
>
> ****************************************
> import apache_beam as beam
> import pandas as pd
> from apache_beam.options.pipeline_options import PipelineOptions
>
>
> def print_item(item):
>     print(item)
>
> df = pd.DataFrame({
>     'letters' : ['a', 'b', 'c', 'd', 'e'],
>     'numbers' : [1, 2, 3, 4, 5],
> })
>
> with beam.Pipeline() as p:
>     (
>     p
>     ## You can also use df.values for a numpy.ndarray
>     | 'Read From Data Frame' >> beam.Create(df.values.tolist())
>     | 'Print Results'  >> beam.Map(print_item)
>     )
> ****************************************
>
>
> On Aug 8, 2018, at 4:01 AM, Eila Arich-Landkof <eila@orielresearch.org>
> wrote:
>
> Hello all,
>
> Is it possible to create a pcollection from a pandas dataframe
>
> Many thanks,
> Eila
>
> ———
> Eila
> www.orielresearch.org
> www.meetup.com/deep-learning-in-production
>
>
>


-- 
Eila
www.orielresearch.org
https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>p.co
<https://www.meetup.com/Deep-Learning-In-Production/>
m/Deep-Learning-In-Production/
<https://www.meetup.com/Deep-Learning-In-Production/>

Mime
View raw message