apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Redshift Output Operator
Date Tue, 21 Feb 2017 17:43:14 GMT
Chaitanya,
This is good first cut. Post this work, do take a look at loading data
before file rotation.

Thks
Amol


*Follow @amolhkekre*
*Join us at Apex Big Data World-San Jose
<http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
[image: http://www.apexbigdata.com/san-jose-register.html]
<http://www.apexbigdata.com/san-jose-register.html>

On Mon, Feb 20, 2017 at 10:56 PM, Chaitanya Chebolu <
chaitanya@datatorrent.com> wrote:

> Created JIRA for this task: APEXMALHAR-2416
>
> On Mon, Feb 13, 2017 at 4:14 PM, Chaitanya Chebolu <
> chaitanya@datatorrent.com> wrote:
>
> > Hi All,
> >
> >   I am proposing Amazon Redshift output module.
> >   Please refer below link about the Redshift: https://aws.amazon.com/
> > redshift/
> >
> >   Primary functionality of this module is load data into Redshift tables
> > from data files using copy command. Refer the below link about the copy
> > command:
> > http://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html
> >
> > Input type to this module is byte[].
> >
> >   I am proposing the below design:
> > 1) Write the tuples into EMR/S3. By default, it writes to S3.
> > 2) Once the file is rolled, upload the file into Redshift using copy
> > command.
> >
> > Please share your thoughts on design.
> >
> > Regards,
> > Chaitanya
> >
>
>
>
> --
>
> *Chaitanya*
>
> Software Engineer
>
> E: chaitanya@datatorrent.com | Twitter: @chaithu1403
>
> www.datatorrent.com  |  apex.apache.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message