cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miguel Angel Martin junquera <mianmarjun.mailingl...@gmail.com>
Subject Re: setting PIG_INPUT_INITIAL_ADDRESS environment . variable in Oozie for cassandra ...¿?
Date Thu, 12 Dec 2013 08:04:38 GMT
¡Eureka!

At last !!!

the trick??

As well as putting the jar libraries dependecies in the sharelib folder in
hdfs ...

I had define this and the other environment variables in the bash_profile
 and works fine if I launch pig scripts from command line shell.

I have to define these
variables: PIG_INITIAL_ADDRESS, PIG_CONF_DIR, PIG_RPC_PORT, PIG_PARTITIONER
.....
in the  hadoop_env.sh and works fine.

The others exceptions were like null exceptions in cassandra's class, etc


Now, It is  working with  pig 0.10

...

Run pig script using PigRunner.run() for Pig version 0.8+
Apache Pig version 0.10.0 (r1328203)

...


Although I configue to use pig 0.11   and put this pig jar  version in the
oozie shrelib, pom , classpath, etc


and , I can not  run pig script by shell actions in oozie, but this another
song!!!




Regards


2013/12/12 Aaron Morton <aaron@thelastpickle.com>

> > Caused by: java.io.IOException: PIG_INPUT_INITIAL_ADDRESS or
> PIG_INITIAL_ADDRESS environment variable not set
> >       at
> org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(CassandraStorage.java:314)
> >       at
> org.apache.cassandra.hadoop.pig.CassandraStorage.getSchema(CassandraStorage.java:358)
> >       at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
> >       ... 35 more
>
> Have you checked these are set ?
>
> Cheers
>
> -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 11/12/2013, at 4:00 am, Miguel Angel Martin junquera <
> mianmarjun.mailinglist@gmail.com> wrote:
>
> > Hi,
> >
> > I have an error with pig action in oozie 4.0.0  using cassandraStorage.
> (cassandra 1.2.10)
> >
> > I can run pig scripts right  with cassandra. but whe I try to use
> cassandraStorage to load data I have this error:
> >
> >
> > Run pig script using PigRunner.run() for Pig version 0.8+
> > Apache Pig version 0.10.0 (r1328203)
> > compiled Apr 20 2012, 00:33:25
> >
> > Run pig script using PigRunner.run() for Pig version 0.8+
> > 2013-12-10 12:24:39,084 [main] INFO  org.apache.pig.Main  - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25
> > 2013-12-10 12:24:39,084 [main] INFO  org.apache.pig.Main  - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25
> > 2013-12-10 12:24:39,095 [main] INFO  org.apache.pig.Main  - Logging
> error messages to:
> /tmp/hadoop-ec2-user/mapred/local/taskTracker/ec2-user/jobcache/job_201312100858_0007/attempt_201312100858_0007_m_000000_0/work/pig-job_201312100858_0007.log
> > 2013-12-10 12:24:39,095 [main] INFO  org.apache.pig.Main  - Logging
> error messages to:
> /tmp/hadoop-ec2-user/mapred/local/taskTracker/ec2-user/jobcache/job_201312100858_0007/attempt_201312100858_0007_m_000000_0/work/pig-job_201312100858_0007.log
> > 2013-12-10 12:24:39,501 [main] INFO
>  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  -
> Connecting to hadoop file system at: hdfs://10.228.243.18:9000
> > 2013-12-10 12:24:39,501 [main] INFO
>  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  -
> Connecting to hadoop file system at: hdfs://10.228.243.18:9000
> > 2013-12-10 12:24:39,510 [main] INFO
>  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  -
> Connecting to map-reduce job tracker at: 10.228.243.18:9001
> > 2013-12-10 12:24:39,510 [main] INFO
>  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  -
> Connecting to map-reduce job tracker at: 10.228.243.18:9001
> > 2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt  -
> ERROR 2245:
> > <file testCassandra.pig, line 7, column 7> Cannot get schema from
> loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage
> > 2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt  -
> ERROR 2245:
> > <file testCassandra.pig, line 7, column 7> Cannot get schema from
> loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage
> > 2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt  -
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> > <file testCassandra.pig, line 7, column 7> Cannot get schema from
> loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage
> >       at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:155)
> >       at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:110)
> >       at
> org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> >       at
> org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> >       at
> org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> >       at
> org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> >       at
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> >       at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> >       at org.apache.pig.PigServer$Graph.compile(PigServer.java:1617)
> >       at org.apache.pig.PigServer$Graph.compile(PigServer.java:1611)
> >       at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1334)
> >       at org.apache.pig.PigServer.execute(PigServer.java:1239)
> >       at org.apache.pig.PigServer.executeBatch(PigServer.java:362)
> >       at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)
> >       at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> >       at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> >       at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> >       at org.apache.pig.Main.run(Main.java:430)
> >       at org.apache.pig.PigRunner.run(PigRunner.java:49)
> >       at
> org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> >       at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> >       at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> >       at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:601)
> >       at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> >       at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >       at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> >       at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >       at java.security.AccessController.doPrivileged(Native Method)
> >       at javax.security.auth.Subject.doAs(Subject.java:415)
> >       at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> >       at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > Caused by: java.io.IOException: PIG_INPUT_INITIAL_ADDRESS or
> PIG_INITIAL_ADDRESS environment variable not set
> >       at
> org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(CassandraStorage.java:314)
> >       at
> org.apache.cassandra.hadoop.pig.CassandraStorage.getSchema(CassandraStorage.java:358)
> >       at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
> >       ... 35 more
> >
> > 2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt  -
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> > <file testCassandra.pig, line 7, column 7> Cannot get schema from
> loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage
> >       at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:155)
> >       at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:110)
> >       at
> org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> >       at
> org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> >       at
> org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> >       at
> org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> >       at
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> >       at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> >       at org.apache.pig.PigServer$Graph.compile(PigServer.java:1617)
> >       at org.apache.pig.PigServer$Graph.compile(PigServer.java:1611)
> >       at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1334)
> >       at org.apache.pig.PigServer.execute(PigServer.java:1239)
> >       at org.apache.pig.PigServer.executeBatch(PigServer.java:362)
> >       at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)
> >       at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> >       at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> >       at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> >       at org.apache.pig.Main.run(Main.java:430)
> >       at org.apache.pig.PigRunner.run(PigRunner.java:49)
> >       at
> org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> >       at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> >       at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> >       at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:601)
> >       at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> >       at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >       at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> >       at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >       at java.security.AccessController.doPrivileged(Native Method)
> >       at javax.security.auth.Subject.doAs(Subject.java:415)
> >       at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> >       at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > Caused by: java.io.IOException: PIG_INPUT_INITIAL_ADDRESS or
> PIG_INITIAL_ADDRESS environment variable not set
> >       at
> org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(CassandraStorage.java:314)
> >       at
> org.apache.cassandra.hadoop.pig.CassandraStorage.getSchema(CassandraStorage.java:358)
> >       at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
> >       ... 35 more
> >
> > <<< Invocation of Main class completed <<<
> >
> > Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.PigMain], exit code [2]
> >
> > Oozie Launcher failed, finishing Hadoop job gracefully
> > -----------------------------------------------------------------------
> >
> >
> > I, m using Ec2 instances and i have a hadoop cluster with cassandra in
> all nodes and I can run right  pig_cassandra scriptand I conigured oozie in
> other instance with namenode..
> >
> >
> > I set the var in .bash_profile file like:
> > ..
> > export PIG_INITIAL_ADDRESS=${seed}
> > ...
> >
> >
> > How can I set this env variable in Oozie or in Pig with Oozie
> >
> > Another  cuestion is:
> >
> > 2.- I configured oozie installation to use pig 0.11.1 in the pom.xml
> like the Oozie instructions installation   but I see in the logs that Oozie
> is using
> > Pig 0.10.
> >
> > ...
> > Run pig script using PigRunner.run() for Pig version 0.8+
> > Apache Pig version 0.10.0 (r1328203)
> > compiled Apr 20 2012, 00:33:25
> > ...
> > I upload to hdfs share lib and check pig library is 0.11.1 version
> > How can i change or configure this issue?
> >
> >
> >
> >
> > thanks in advance
> > And any helps wil be appreciated.
> >
> > Regards
>
>

Mime
View raw message