crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-372) Allow MRPipeline to retrieve delegation token if available
Date Mon, 31 Mar 2014 22:17:14 GMT


Josh Wills commented on CRUNCH-372:

Seems reasonable and harmless for non-oozie use cases, but I am not an oozie expert. Are there
any times that we'd want to disable it?

My first thought would be to put it in the DistributedPipeline class, so that Spark pipelines
could pick it up as well if/when there are Spark oozie tasks.

> Allow MRPipeline to retrieve delegation token if available
> ----------------------------------------------------------
>                 Key: CRUNCH-372
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.8.0
>            Reporter: Jeremy Beard
>            Assignee: Josh Wills
> Delegation tokens are used when executing MR from an Oozie workflow. MR in Crunch can
be manually set to reference the token:
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
>     getConf().set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> However if it would be nice if this was somewhere in the API, or perhaps automatically

This message was sent by Atlassian JIRA

View raw message