accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob.Thor...@l-3com.com
Subject RE: AcculumoFileOutputFormat class cannot be found by child jvm
Date Tue, 22 May 2012 20:57:32 GMT
You may be right about the tool.sh -libjars.  It doesn't copy the jar
files to the ${mapred.local.dir}/taskTracker/archive/ directory.
Neither does adding them to the HADOOP_CLASSPATH.

 

So to fix this problem I had to set up a DistributedCache and explicitly
add them to class path as follows:

 

DistributedCache.addFileToClassPath(new
Path("/usr/lib/accumulo-core-1.4.0.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/kxml2-2.3.0.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/xmlpull-1.1.3.1.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/xstream-1.4.1.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/zookeeper-3.4.3.jar"), conf);

                DistributedCache.addFileToClassPath(new
Path("/usr/lib/commons-collections-3.2.jar"), conf);

 

Although this works, it doesn't feel like I have something configured
correctly with my accumulo 1.4.0 upgrade.  Is there a better way to
configure the resources for the child jvm's?

 

From: John Vines [mailto:john.w.vines@ugov.gov] 
Sent: Tuesday, May 22, 2012 13:20
To: user@accumulo.apache.org
Subject: Re: AcculumoFileOutputFormat class cannot be found by child jvm

 

I'm wondering if our tool.sh plays nice with an explicit -libjars.

John

On Tue, May 22, 2012 at 2:07 PM, William Slacum <wslacum@gmail.com>
wrote:

Make sure that the paths you pass to `-libjars` are correct. I have
noticed behavior with Hadoop where it will silently drop the end of a
libjars string if the paths do not exist.


On Tue, May 22, 2012 at 11:03 AM, Keith Turner <keith@deenlo.com> wrote:
> I think it should be in the job.xml file.  The way the class path for
> the java program that starts map reduce is configured is different
> from how the classpath for the remote task are configured.   So your
> program that starts the map reduce job has AccumuloFileOutputFormat on
> the class path because you use a static method on it and it runs.  So
> I am thinking its not in --libjars for some reason or maybe you are
> not using Hadoop Tool?
>
> On Tue, May 22, 2012 at 1:54 PM,  <Bob.Thorman@l-3com.com> wrote:
>> No, it's not in the job.xml file.  A reference to
AccumuloFileOutputFormat is but not the accumulo-core-1.4.0.jar.  The
Job*.jar file is referenced there.  I was under the impression that the
child jvm inherits the parent jvm attributes which makes the job*.jar
file.  Is that not correct?
>>
>> -----Original Message-----
>> From: Keith Turner [mailto:keith@deenlo.com]
>> Sent: Tuesday, May 22, 2012 12:07
>> To: user@accumulo.apache.org
>> Subject: Re: AcculumoFileOutputFormat class cannot be found by child
jvm
>>
>> If you look at Job.xml, do you see accumulo-core there?  There should
be a link to this file on the jobs page on the tasktracker.
>>
>> On Tue, May 22, 2012 at 10:40 AM,  <Bob.Thorman@l-3com.com> wrote:
>>> I upgrade to accumulo-1.4.0 and updated my map/reduce jobs and now
>>> they don't run.  The parent class path has the
accumulo-core-1.4.0.jar
>>> file included.  Do the accumulo jar files have to be manually put on
a
>>> distribute cache?  Any help is appreciated.
>>>
>>> [hadoop@redhat-cloudbase1 placemarks]$ ./runPlacemarkIngester.sh
Found
>>> 5 items drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:13
>>> /accumulo drwxrwxr-x   - hadoop hadoop          0 2012-05-21 15:06
>>> /data drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58
/input
>>> drwxr-xr-x   - hadoop hadoop          0 2012-05-22 08:58 /output
>>> drwxrwxr-x   - hadoop hadoop          0 2012-05-21 14:34 /usr
Deleted
>>> hdfs://redhat-cloudbase1:9000/output
>>> Deleted hdfs://redhat-cloudbase1:9000/input
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: Invoking
>>> ToolRunner.run
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: zooKeeper is
>>> redhat-cloudbase1:2181
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: instanceName is
>>> NCCT-Cloudbase
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: timeTableName
is
>>> NCCTServicesTimes
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: geoTableName is
>>> NCCTServicesGeos
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester:
metadataTableName
>>> is NCCTServicesMetadata
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: edgeTableName
is
>>> NCCTEdgeTable
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: userName is
>>> NCCT.Services.Client
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: password is
>>> *********
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: visibility is
>>> public,BD2,UNCLASSIFIED-NO_COMPARTMENT-UNRESTRICTED
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: inputDir is
>>> /input
>>> 12/05/22 09:05:29 INFO placemarks.PlacemarkIngester: outputDir is
>>> /output
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012
10:48
>>> GMT
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:host.name=redhat-cloudbase1
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.version=1.6.0_32
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.vendor=Sun Microsystems Inc.
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.home=/usr/java/jdk1.6.0_32/jre
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>>
environment:java.class.path=/cloudbase/hadoop-0.20.2/bin/../conf:/usr/
>>> ja
>>>
va/jdk1.6.0_32/lib/tools.jar:/cloudbase/hadoop-0.20.2/bin/..:/cloudbas
>>> e/
>>>
hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/cloudbase/hadoop-0.20.2/b
>>> in
>>>
/../lib/commons-cli-1.2.jar:/cloudbase/hadoop-0.20.2/bin/../lib/common
>>> s-
>>>
codec-1.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/
>>> cl
>>>
oudbase/hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/cloudba
>>> se
>>>
/hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/cloudbase/hadoop-0.
>>>
20.2/bin/../lib/commons-logging-api-1.0.4.jar:/cloudbase/hadoop-0.20.2
>>> /b
>>>
in/../lib/commons-net-1.4.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/co
>>> re
>>>
-3.1.1.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-ant.jar:/
>>> cl
>>>
oudbase/hadoop-0.20.2/bin/../lib/hadoop-0.20.2-core.jar:/cloudbase/had
>>> oo
>>>
p-0.20.2/bin/../lib/hadoop-0.20.2-examples.jar:/cloudbase/hadoop-0.20.
>>> 2/
>>>
bin/../lib/hadoop-0.20.2-test.jar:/cloudbase/hadoop-0.20.2/bin/../lib/
>>> ha
>>>
doop-0.20.2-tools.jar:/cloudbase/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0
>>> .1
>>>
0.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jasper-compiler-5.5.12.jar:/
>>> cl
>>>
oudbase/hadoop-0.20.2/bin/../lib/jasper-runtime-5.5.12.jar:/cloudbase/
>>> ha
>>>
doop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/cloudbase/hadoop-0.20.2/bin/.
>>> ./
>>>
lib/jetty-6.1.14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/jetty-util-6.1.
>>>
14.jar:/cloudbase/hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/cloudbase/
>>> ha
>>>
doop-0.20.2/bin/../lib/kfs-0.2.2.jar:/cloudbase/hadoop-0.20.2/bin/../l
>>> ib
>>>
/log4j-1.2.15.jar:/cloudbase/hadoop-0.20.2/bin/../lib/mockito-all-1.8.0.
>>>
jar:/cloudbase/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/cloudbase/hadoo
>>> p-
>>>
0.20.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/cloudbase/hadoop-0.20.2/
>>> bi
>>>
n/../lib/slf4j-api-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/slf4j
>>> -l
>>>
og4j12-1.4.3.jar:/cloudbase/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/
>>> cl
>>>
oudbase/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/cloudbase/hadoop
>>> -0
>>>
.20.2/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/cloudbase/accumulo-1.4.0/lib
>>> /l
>>>
ibthrift-0.6.1.jar:/cloudbase/accumulo-1.4.0/lib/accumulo-core-1.4.0.j
>>> ar
>>>
:/cloudbase/zookeeper-3.4.3/zookeeper-3.4.3.jar:/cloudbase/accumulo-1.4.
>>>
0/lib/cloudtrace-1.4.0.jar:/usr/lib/ncct/kxml2-2.3.0.jar:/usr/lib/ncct
>>> /x
>>>
mlpull-1.1.3.1.jar:/usr/lib/ncct/xstream-1.4.1.jar:/cloudbase/accumulo
>>> -1
>>>
.4.0/lib/commons-collections-3.2.jar:/cloudbase/accumulo-1.4.0/lib/com
>>> mo
>>>
ns-configuration-1.5.jar:/cloudbase/accumulo-1.4.0/lib/commons-io-1.4.
>>> ja
>>>
r:/cloudbase/accumulo-1.4.0/lib/commons-jci-core-1.0.jar:/cloudbase/ac
>>> cu
>>>
mulo-1.4.0/lib/commons-jci-fam-1.0.jar:/cloudbase/accumulo-1.4.0/lib/c
>>> om
>>>
mons-lang-2.4.jar:/cloudbase/accumulo-1.4.0/lib/commons-logging-1.0.4.
>>> ja
>>> r:/cloudbase/accumulo-1.4.0/lib/commons-logging-api-1.0.4.jar:
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>>
environment:java.library.path=/cloudbase/hadoop-0.20.2/bin/../lib/nati
>>> ve
>>> /Linux-amd64-64
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.io.tmpdir=/tmp
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:java.compiler=<NA>
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:os.name=Linux
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:os.arch=amd64
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:os.version=2.6.32-131.0.15.el6.x86_64
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:user.name=hadoop
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>> environment:user.home=/home/hadoop
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Client
>>>
environment:user.dir=/mnt/hgfs/CSI.Cloudbase/Java/CloudbaseServices/Cl
>>> ou dbaseIngesters/src/com/comcept/cloudbase/ingesters/placemarks
>>> 12/05/22 09:05:29 INFO zookeeper.ZooKeeper: Initiating client
>>> connection, connectString=redhat-cloudbase1:2181
sessionTimeout=30000
>>>
watcher=org.apache.accumulo.core.zookeeper.ZooSession$AccumuloWatcher@
>>> 4f
>>> 4db0e3
>>> 12/05/22 09:05:29 INFO zookeeper.ClientCnxn: Opening socket
connection
>>> to server /192.168.136.2:2181
>>> 12/05/22 09:05:29 WARN client.ZooKeeperSaslClient:
SecurityException:
>>> java.lang.SecurityException: Unable to locate a login configuration
>>> occurred when trying to find JAAS configuration.
>>> 12/05/22 09:05:29 INFO client.ZooKeeperSaslClient: Client will not
>>> SASL-authenticate because the default JAAS configuration section
>>> 'Client' could not be found. If you are not using SASL, you may
ignore
>>> this. On the other hand, if you expected SASL to work, please fix
your
>>> JAAS configuration.
>>> 12/05/22 09:05:29 INFO zookeeper.ClientCnxn: Socket connection
>>> established to redhat-cloudbase1/192.168.136.2:2181, initiating
>>> session
>>> 12/05/22 09:05:29 INFO zookeeper.ClientCnxn: Session establishment
>>> complete on server redhat-cloudbase1/192.168.136.2:2181, sessionid =
>>> 0x1377101615a323f, negotiated timeout = 30000
>>> 12/05/22 09:05:30 INFO input.FileInputFormat: Total input paths to
>>> process : 3
>>> 12/05/22 09:05:30 INFO mapred.JobClient: Running job:
>>> job_201205211505_0012
>>> 12/05/22 09:05:31 INFO mapred.JobClient:  map 0% reduce 0%
>>> 12/05/22 09:05:38 INFO mapred.JobClient: Task Id :
>>> attempt_201205211505_0012_m_000004_0, Status : FAILED
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat
>>>        at
>>>
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>>>        at
>>>
org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext
>>> .j
>>> ava:193)
>>>        at org.apache.hadoop.mapred.Task.initialize(Task.java:413)
>>>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:288)
>>>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat
>>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>        at
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>        at java.lang.Class.forName0(Native Method)
>>>        at java.lang.Class.forName(Class.java:247)
>>>        at
>>>
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java
>>> :7
>>> 62)
>>>        at
>>>
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
>>>        ... 4 more
>>>
>>> Bob Thorman
>>> Engineering Fellow
>>> L-3 Communications, ComCept
>>> 1700 Science Place
>>> Rockwall, TX 75032
>>> (972) 772-7501 <tel:%28972%29%20772-7501>  work
>>> Bob.Thorman@ncct.af.smil.mil
>>> rdthorm@nsa.ic.gov
>>>
>>>

 


Mime
View raw message