hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: hbase export in cluster mode
Date Fri, 09 Apr 2010 04:46:45 GMT
Our doc. on this topic can be found here:
http://hadoop.apache.org/hbase/docs/r0.20.3/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description

St.Ack

On Thu, Apr 8, 2010 at 9:40 PM, Todd Lipcon <todd@cloudera.com> wrote:
> On Thu, Apr 8, 2010 at 4:39 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> Ted, please make sure you keep the conversation on the mailing list,
>> private answers have no value for an open source community.
>>
>> About JT, it would be awesome not to have to restart it but I don't
>> know enough about its internals to comment on how hard it would be to
>> modify. Maybe there's even an open jira.
>>
>> Or maybe someone else who listens could comment? Cloudera people?
>>
>>
> Rather than adding the jars to lib/ or HADOOP_CLASSPATH, you can also insert
> them directly into a lib/ directory inside your job jar. Then it will be
> added to the classpath of your MR Task child JVM when your job is localized
> on the task trackers. I think this is best practice for any jars that you
> anticipate will change on even a semiregular basis.
>
> -Todd
>
>
>> J-D
>>
>> On Thu, Apr 8, 2010 at 4:33 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> > J-D:
>> > Do you think it makes for job tracker to dynamically refresh classpath ?
>> > I can see this need because HBase, HIVE, etc may initiate request in
>> cluster
>> > mode and we don't want to restart job tracker in production often.
>> >
>> > Cheers
>> >
>> > On Thu, Apr 8, 2010 at 3:52 PM, Jean-Daniel Cryans <jdcryans@apache.org>
>> > wrote:
>> >>
>> >> Actually the preferred method is putting the hbase and zookeeper jar
>> >> on the HADOOP_CLASSPATH of conf/hadoop-env.sh (putting the hbase conf
>> >> folder there is also good).
>> >>
>> >> Yeah have to restart JT, but AFAIK it's not HBase's fault right? ;)
>> >>
>> >> J-D
>> >>
>> >> On Thu, Apr 8, 2010 at 3:47 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >> > I copied zookeeper-3.2.1.jar after seeing the exception below.
>> >> > Turns out that I have to restart job tracker to refresh classpath :-(
>> >> >
>> >> > On Thu, Apr 8, 2010 at 3:03 PM, Jean-Daniel Cryans
>> >> > <jdcryans@apache.org>wrote:
>> >> >
>> >> >> Zookeeper jar isn't in the path of hadoop?
>> >> >>
>> >> >> btw https://issues.apache.org/jira/browse/HBASE-2335
>> >> >>
>> >> >> J-D
>> >> >>
>> >> >> On Thu, Apr 8, 2010 at 2:58 PM, Ted Yu <yuzhihong@gmail.com>
wrote:
>> >> >> > I ran hbase export in cluster mode.
>> >> >> >
>> >> >> > From Saturn:
>> >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: zookeeper.
>> >> >> > disableAutoWatchReset is false
>> >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: Attempting connection
>> to
>> >> >> server
>> >> >> > uranus/10.10.31.18:2181
>> >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: Priming connection
to
>> >> >> > java.nio.channels.SocketChannel[connected local=/10.10.31.16:34170
>> >> >> remote=uranus/
>> >> >> > 10.10.31.18:2181]
>> >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: Server connection
>> >> >> > successful
>> >> >> > 10/04/07 14:18:22 INFO mapreduce.TableInputFormatBase: split:
>> >> >> 0*->Neptune*:,
>> >> >> > 10/04/07 14:18:23 INFO mapred.JobClient: Running job:
>> >> >> job_201004051529_0015
>> >> >> > 10/04/07 14:18:24 INFO mapred.JobClient:  map 0% reduce 0%
>> >> >> > 10/04/07 14:18:36 INFO mapred.JobClient: Task Id :
>> >> >> > attempt_201004051529_0015_m_000000_0, Status : FAILED
>> >> >> > Error: java.lang.ClassNotFoundException:
>> org.apache.zookeeper.Watcher
>> >> >> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >> >> >        at java.security.AccessController.doPrivileged(Native
>> Method)
>> >> >> >        at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >> >> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >> >> >        at
>> >> >> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >> >> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> >> >> >        at
>> >> >> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> >> >> >        at java.lang.ClassLoader.defineClass1(Native Method)
>> >> >> >        at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
>> >> >> >        at
>> >> >> > java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
>> >> >> >        at
>> java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>> >> >> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>> >> >> >        at java.security.AccessController.doPrivileged(Native
>> Method)
>> >> >> >        at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >> >> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >> >> >        at
>> >> >> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >> >> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> >> >> >        at
>> >> >> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager.getClientZooKeeperWatcher(HConnectionManager.java:151)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getZooKeeperWrapper(HConnectionManager.java:885)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:901)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:580)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:556)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:630)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:589)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:556)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:630)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:593)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:556)
>> >> >> >        at
>> >> >> > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:127)
>> >> >> >        at
>> >> >> > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:105)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:73)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>> >> >> >        at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> >> >> >        at
>> >> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:536)
>> >> >> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >> >> >        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >> >> >
>> >> >> > I wonder why the above exception was thrown because we have
same
>> >> >> > hbase
>> >> >> > deployment on all the servers:
>> >> >> > [root@neptune software]# ls -l lib/zookeeper-3.2.1.jar
>> >> >> > -rwxr-xr-x 1 hbaseadmin hbaseadmin 913093 Oct  7 11:55
>> >> >> > lib/zookeeper-3.2.1.jar
>> >> >> >
>> >> >> > Can someone shed some light on how to get over the above exception
>> ?
>> >> >> >
>> >> >> > Thanks
>> >> >> >
>> >> >>
>> >> >
>> >
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message