airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1324) Make the Druid operator/hook more general
Date Fri, 18 Aug 2017 19:36:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133534#comment-16133534
] 

ASF subversion and git services commented on AIRFLOW-1324:
----------------------------------------------------------

Commit de99aa20f4ffaaf0757d339abcc96961172d238c in incubator-airflow's branch refs/heads/master
from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=de99aa2 ]

[AIRFLOW-1324] Generalize Druid operator and hook

Make the druid operator and hook more specific.
This allows us to
have a more flexible configuration, for example
ingest parquet.
Also get rid of the PyDruid extension since it is
more focussed on
querying druid, rather than ingesting data. Just
requests is
sufficient to submit an indexing job. Add a test
to the hive_to_druid
operator to make sure it behaves as we expect.
Furthermore cleaned
up the docstring a bit

Closes #2378 from Fokko/AIRFLOW-1324-make-more-
general-druid-hook-and-operator


> Make the Druid operator/hook more general
> -----------------------------------------
>
>                 Key: AIRFLOW-1324
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1324
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Fokko Driesprong
>             Fix For: 1.9.0
>
>
> Hi guys,
> Right now the Druid operator is quite specific with respect to the indexing spec. This
is predefined and does not fit our use case. For example, we ingest parquet files instead
of flat files. This is not possible right now and therefore a more general druid operator
would be nice.
> Right now I have changed the files, we'll check them on our own cluster the upcoming
days to make sure that they work properly.
> Cheers, Fokko



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message