airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2751) hive to druid library conflicts
Date Sun, 02 Sep 2018 17:59:03 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601327#comment-16601327
] 

Apache Spark commented on AIRFLOW-2751:
---------------------------------------

User 'happyjulie' has created a pull request for this issue:
https://github.com/apache/incubator-airflow/pull/3600

> hive to druid library conflicts
> -------------------------------
>
>                 Key: AIRFLOW-2751
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2751
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>    Affects Versions: 1.10
>            Reporter: juhwi lee
>            Assignee: juhwi lee
>            Priority: Major
>             Fix For: 2.0.0
>
>
> There is a jackson.datatype.guava library conflict problem in HiveToDruidTransfer when
hive and druid has different library version.
> In my case
>  hdp version: 2.6.3.0
>  druid version : druid-0.10.1( jackson-datatype-guava-2.4.6.jar)
> We should be able to set this property to solve that problem.
>  "mapreduce.job.user.classpath.first" : "true"
> It means "prefer loading Druid's version of a library when there is a conflict".
>  reference. [http://druid.io/docs/latest/operations/other-hadoop.html] (Tip #2)
> But Job properties is hard coded in HiveToDruidTransfer.
>  
> This is the error message I got.
>  2018-06-22T01:37:11,190 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job
- Task Id : attempt_1525761839652_85792_m_000000_0, Status : FAILED
>  Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
>  2018-06-22T01:37:15,216 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job
- Task Id : attempt_1525761839652_85792_m_000000_1, Status : FAILED
>  Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
>  2018-06-22T01:37:20,231 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job
- Task Id : attempt_1525761839652_85792_m_000000_2, Status : FAILED
>  Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
>  2018-06-22T01:37:26,248 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job
- map 100% reduce 100%
>  2018-06-22T01:37:26,252 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job
- Job job_1525761839652_85792 failed with state FAILED due to: Task failed task_1525761839652_85792_m_000000
>  Job failed as tasks failed. failedMaps:1 failedReduces:0
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message