pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prashant Kommireddi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3135) HExecutionEngine should look for resources in user passed Properties
Date Wed, 20 Feb 2013 01:06:12 GMT

    [ https://issues.apache.org/jira/browse/PIG-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581825#comment-13581825

Prashant Kommireddi commented on PIG-3135:

Hey [~cheolsoo], I think the issue is with the fact that MiniCluster.buildCluster() generates
a hadoop-site.xml file under build/classes. However, MiniCluster.shutDown() does not delete
this file. Ideally, hadoop-site.xml should be deleted on cluster shutdown. 

With respect to TestHExecutionEngine, the issue is with the fact a previous test run that
uses mini-cluster generates hadoop-site.xml and does not delete it. At the time TestHExecutionEngine
runs this file is present on the classpath but mini-dfs and mini-mr were shutdown at the completion
of previous test. pigContext.connect() in TestHExecutionEngine tries to establish a connection
with mini-cluster which is no longer up.

The fix would be to:

1. Delete hadoop-site-xml on mini-cluster shutdown (to be done in another JIRA)
2. Check if this patch stabilizes (and nothing else breaks)

Does that make sense?
> HExecutionEngine should look for resources in user passed Properties
> --------------------------------------------------------------------
>                 Key: PIG-3135
>                 URL: https://issues.apache.org/jira/browse/PIG-3135
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.10.0
>            Reporter: Prashant Kommireddi
>            Assignee: Prashant Kommireddi
>             Fix For: 0.12
>         Attachments: PIG-3135_1.patch, PIG-3135.patch
> Looking at this snippet:
> {code}
>     private void init(Properties properties) throws ExecException {
>           .
>           .
>           .
>             // Check existence of hadoop-site.xml or core-site.xml
>             Configuration testConf = new Configuration();
>             ClassLoader cl = testConf.getClassLoader();
>             URL hadoop_site = cl.getResource( HADOOP_SITE );
>             URL core_site = cl.getResource( CORE_SITE );
>             if( hadoop_site == null && core_site == null ) {
>                 throw new ExecException("Cannot find hadoop configurations in classpath
(neither hadoop-site.xml nor core-site.xml was found in the classpath)." +
>                         " If you plan to use local mode, please put -x local option in
command line",
>                         4010);
>             }
> {code}
> This assumes the resources (*-site.xml) are set on the classpath, but this will not always
be the case when run with Pig's Java APIs. One could want to programatically set the resources
and the code here should additionally check if they are available in there. 
> Example: When a Configuration object is created and resources are added before passing
it on to Pig.
> {code}
> Configuration conf = new Configuration(false);
> conf.addResource("foo/core-site.xml");
> conf.addResource("bar/hadoop-site.xml");
> PigServer pServer = new PigServer(ExecType.MAPREDUCE, conf);
> {code}
> The above conf is not used right now to obtain resources.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message