hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16276) Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
Date Wed, 22 Mar 2017 20:43:41 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937093#comment-15937093
] 

Sahil Takiar commented on HIVE-16276:
-------------------------------------

[~appodictic] the shading of the jars should only affect the code inside the {{druid-aws-common}}
library. So any code that uses the druid-handler to access S3 will go through the newer version
of the SDK. As far as I can tell, the only other place Hive contacts S3 is via the {{hadoop-aws}}
artifact. That code path will use an older version of the sdk (the version that is hadoop
compatible).

Before this patch, it was not possible to create a Hive table on top of data on S3. The exception
in the JIRA title would be thrown. This patch fixes that.

[~jcamachorodriguez] could you take a look, it looks like you have touched some the druid-handler
shading logic.

> Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16276
>                 URL: https://issues.apache.org/jira/browse/HIVE-16276
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-16276.1.patch, HIVE-16276.2.patch, HIVE-16276.3.patch, HIVE-16276.4.patch
>
>
> The druid-handler is pulling in some com.amazonaws dependencies that conflict with the
version that Hadoop is using. This causes the above exception to be thrown when running Hive
against S3. This patch fixes the dependency issue by shading the aws dependencies in the druid
artifacts. Unfortunately, I can't find a great way to add a test for this so it doesn't happen
in the future. We will need some more robust S3-integration tests for that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message