pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-3038) Support for Credentials for UDF,Loader and Storer
Date Wed, 07 Nov 2012 18:12:13 GMT

     [ https://issues.apache.org/jira/browse/PIG-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Rohini Palaniswamy updated PIG-3038:

    Summary: Support for Credentials for UDF,Loader and Storer  (was: Support for Credentials
for UDF/Loader/Storer)

Currently it is possible to add Credentials in LoadFunc (in the Job passed to setLocation)
and StoreFunc (in the Job passed to setStoreLocation). In case of StoreFunc, the Job passed
is different everytime for the 3 calls to setStoreLocation made in front end and only the
Credentials set during the call to setStoreLocation from PigOutputFormat.checkOutputSpecs
take effect. That is very obscure and needs people to know the internals of the working of
pig to get it working.

  UDFs(EvalFunc) on the other had have no API in them where the Job is passed and Credentials
can be added. Users will have to serialize and store the credentials in UDFProperties and
retrieve from there which is not secure.

  Add a getCredentials() API to UDFContext similar to JobConf.getCredentials(). Users can
add their tokens to the Credentials object returned. On the backend the API will return the
Credentials the job was launched with. The approach would be cleaner and backward compatible.

  Alternative is to have credential related APIs added to EvalFunc, LoadFunc and StoreFunc.
To deal with backward compatibility in that case, one will have to do reflection and determine
whether the method is implemented or not and then call it which is not that good. 
Note: However we do it, adding a API with Credentials will break backward compatibility with
Hadoop 0.20.2. We will have to decide whether we we plan to continue support for it even in
pig 0.12. 
> Support for Credentials for UDF,Loader and Storer
> -------------------------------------------------
>                 Key: PIG-3038
>                 URL: https://issues.apache.org/jira/browse/PIG-3038
>             Project: Pig
>          Issue Type: New Feature
>    Affects Versions: 0.10.0
>            Reporter: Rohini Palaniswamy
>             Fix For: 0.12
>   Pig does not have a clean way (APIs) to support adding Credentials (hbase token, hcat/hive
metastore token) to Job and retrieving it.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message