hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Poepping (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
Date Thu, 09 Mar 2017 19:31:38 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903699#comment-15903699
] 

Thomas Poepping commented on HIVE-14848:
----------------------------------------

Good point [~spena], doesn't make sense to have this before Hadoop 2.9. Here's the issue I
was facing -- when running against "real S3" using the Tez execution engine, the credentials
passed via hive-site.xml are not passed to individual Tez workers, which fails the tests.
There are ways around this, and myself and my team have successfully worked around it.

Honestly, if nobody else has any issue, I'm happy to wait on this for Hadoop 2.9.0 as well.
I'll try to track on my end so we don't drop this issue by accident.

> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
> ------------------------------------------------------------------------
>
>                 Key: HIVE-14848
>                 URL: https://issues.apache.org/jira/browse/HIVE-14848
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 2.2.0
>            Reporter: Sergio Peña
>            Assignee: Sergio Peña
>         Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to read
> data from S3 cannot use them because S3 values are stripped from the Job configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext driverContext)
{
>   ...
>   conf.stripHiddenConfigurations(job);
>   this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR {{mapreduce.job.redacted-properties}}
that can be used to hide this list on the MR side (such as history server UI) to allow MR
run the job without issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in ql/exec/tez/DagUtils.createConfiguration(),
because this is currently broken for running hive-blobstore suite against Tez



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message