crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Bernstein (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-600) Credentials are lost in local runner mode
Date Sun, 10 Apr 2016 19:46:25 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Igor Bernstein updated CRUNCH-600:
----------------------------------
    Description: 
The credentials attached to a job are lost when CrunchOutputs builds a new Job in getJob.
For example, the following scenario fails:

{code:title=MyTarget.java}
class MyTarget implements MapReduceTarget {
  void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name)
{
    job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
    CrunchOutputs.addNamedOutput(job, name, FormatBundle.forOutput(MyOutputFormat.class),
Object.class);
  }
  //..
}
{code}

{code:title=MyOutputFormat.java}
class MyOutputFormat extends OutputFormat<Object,Object> {
  public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext context) throws
IOException, InterruptedException {
    Assert.equals("secret".getBytes(), context.getCredentials().getSecretKey("myPassword"));
  }
}
{code}

This happens because CrunchOutputs.getJob copies the baseConf using the base Configuration
class and the base Configuration class doesn't have a notion of Credentials, so the credentials
get dropped.

This issue only manifests itself when using the local runner. In an actual hadoop cluster
the credentials will be inherited from UserGroupInformation.


  was:
The credentials attached to a job are lost when CrunchOutputs builds a new Job in getJob.
For example, the following scenario fails:

{code:title=MyTarget.java}
class MyTarget implements MapReduceTarget {
  void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name)
{
    job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
    CrunchOutputs.addNamedOutput(job, name, FormatBundle.forOutput(MyOutputFormat.class),
Object.class);
  }
  //..
}
{code}

{code:title=MyOutputFormat.java}
class MyOutputFormat extends OutputFormat<Object,Object> {
  public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext context) throws
IOException, InterruptedException {
    Assert.equals("secret".getBytes(), context.getCredentials().getSecretKey("myPassword"));
  }
}
{code}

This happens because CrunchOutputs.getJob copies the baseConf using the base Configuration
class and the base Configuration class doesn't have a notion of Credentials, so the credentials
get dropped.

This is issue only manifests itself when using the local runner. In an actual hadoop cluster
the credentials will be inherited from UserGroupInformation.



> Credentials are lost in local runner mode
> -----------------------------------------
>
>                 Key: CRUNCH-600
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-600
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Igor Bernstein
>            Priority: Minor
>         Attachments: CRUNCH-600.patch
>
>
> The credentials attached to a job are lost when CrunchOutputs builds a new Job in getJob.
> For example, the following scenario fails:
> {code:title=MyTarget.java}
> class MyTarget implements MapReduceTarget {
>   void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name)
{
>     job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
>     CrunchOutputs.addNamedOutput(job, name, FormatBundle.forOutput(MyOutputFormat.class),
Object.class);
>   }
>   //..
> }
> {code}
> {code:title=MyOutputFormat.java}
> class MyOutputFormat extends OutputFormat<Object,Object> {
>   public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext context)
throws IOException, InterruptedException {
>     Assert.equals("secret".getBytes(), context.getCredentials().getSecretKey("myPassword"));
>   }
> }
> {code}
> This happens because CrunchOutputs.getJob copies the baseConf using the base Configuration
class and the base Configuration class doesn't have a notion of Credentials, so the credentials
get dropped.
> This issue only manifests itself when using the local runner. In an actual hadoop cluster
the credentials will be inherited from UserGroupInformation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message