hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: Connection problem during data import into hbase
Date Sat, 21 Feb 2009 10:21:20 GMT
I have 1 master + 2 slaves. I did set the timout to zero. I'll set the
xceivers to 2047 and try again. Can this be done in the job config or does
the site.xml need to be changed and the cluster restarted?

Amandeep


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz


On Sat, Feb 21, 2009 at 2:16 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> So the usual suspects are:
>
> - xcievers (i hvae mine set to 2047)
> - timeout (i have mine set to 0)
>
> I can import a few hundred million records with these settings.
>
> how many nodes do you have again?
>
> On Sat, Feb 21, 2009 at 2:14 AM, Amandeep Khurana <amansk@gmail.com>
> wrote:
>
> > Yes, I noticed it this time. The regionserver gets slow or stops
> responding
> > and then this error comes. How do I get this to work? Is there a way of
> > limiting the resources that the map red job should take?
> >
> > I did make the changes in the config site similar to Larry Comton's
> config.
> > It only made the job go from dying at 7% to 12% this time.
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sat, Feb 21, 2009 at 1:14 AM, stack <stack@duboce.net> wrote:
> >
> > > It looks like regionserver hosting root crashed:
> > >
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > >
> > > How many servers you running?
> > >
> > > You made similar config. to that reported by Larry Compton in a mail
> from
> > > earlier today?  (See FAQ and Troubleshooting page for more on his
> listed
> > > configs.)
> > >
> > > St.Ack
> > >
> > >
> > > On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana <amansk@gmail.com>
> > > wrote:
> > >
> > > > Yes, the table exists before I start the job.
> > > >
> > > > I am not using TableOutputFormat. I picked up the sample code from
> the
> > > docs
> > > > and am using it.
> > > >
> > > > Here's the job conf:
> > > >
> > > > JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
> > > >        FileInputFormat.setInputPaths(conf, new Path("import_data"));
> > > >        conf.setMapperClass(MapClass.class);
> > > >        conf.setNumReduceTasks(0);
> > > >        conf.setOutputFormat(NullOutputFormat.class);
> > > >        JobClient.runJob(conf);
> > > >
> > > > Interestingly, the hbase shell isnt working now either. Its giving
> > errors
> > > > even when I give the command "list"...
> > > >
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sat, Feb 21, 2009 at 12:10 AM, stack <stack@duboce.net> wrote:
> > > >
> > > > > The table exists before you start the MR job?
> > > > >
> > > > > When you say 'midway through the job', are you using
> > tableoutputformat
> > > to
> > > > > insert into your table?
> > > > >
> > > > > Which version of hbase?
> > > > >
> > > > > St.Ack
> > > > >
> > > > > On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I dont know if this is related or not, but it seems to be. After
> > this
> > > > map
> > > > > > reduce job, I tried to count the number of entries in the table
> in
> > > > hbase
> > > > > > through the shell. It failed with the following error:
> > > > > >
> > > > > > hbase(main):002:0> count 'in_table'
> > > > > > NativeException: java.lang.NullPointerException: null
> > > > > >    from java.lang.String:-1:in `<init>'
> > > > > >    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
> > > > > >    from
> > > > > org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
> > > > > > `getMessage'
> > > > > >    from
> > > > > org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
> > > > > > `<init>'
> > > > > >    from
> > org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
> > > > > > `getRegionServerWithRetries'
> > > > > >    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in
> > > > `metaScan'
> > > > > >    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in
> > > > `metaScan'
> > > > > >    from
> > org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
> > > > > > `getHTableDescriptor'
> > > > > >    from org/apache/hadoop/hbase/client/HTable.java:219:in
> > > > > > `getTableDescriptor'
> > > > > >    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
> > > > > >    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
> > > > > >    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
> > > > > >    from java.lang.reflect.Method:-1:in `invoke'
> > > > > >    from org/jruby/javasupport/JavaMethod.java:250:in
> > > > > > `invokeWithExceptionHandling'
> > > > > >    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> > > > > >    from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > > > > > ... 145 levels...
> > > > > >    from
> org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> > > > > `call'
> > > > > >    from
> > org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> > > > > `call'
> > > > > >    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > > > > >    from org/jruby/runtime/CallSite.java:298:in `call'
> > > > > >    from
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
> > > > > > `__file__'
> > > > > >    from
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
> > > > > > `__file__'
> > > > > >    from
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
> > > > > > `load'
> > > > > >    from org/jruby/Ruby.java:512:in `runScript'
> > > > > >    from org/jruby/Ruby.java:432:in `runNormally'
> > > > > >    from org/jruby/Ruby.java:312:in `runFromMain'
> > > > > >    from org/jruby/Main.java:144:in `run'
> > > > > >    from org/jruby/Main.java:89:in `run'
> > > > > >    from org/jruby/Main.java:80:in `main'
> > > > > >    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
> > > > > >    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
> > > > > >    from (hbase):3:in `binding'
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <
> > amansk@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Here's what it throws on the console:
> > > > > > >
> > > > > > > 09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
> > > > > > > attempt_200902201300_0019_m_000006_0, Status : FAILED
> > > > > > > java.io.IOException: table is null
> > > > > > >         at
> IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
> > > > > > >         at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
> > > > > > >         at
> > > org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > > > > > >         at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> > > > > > >         at org.apache.hadoop.mapred.Child.main(Child.java:155)
> > > > > > >
> > > > > > > attempt_200902201300_0019_m_000006_0:
> > > > > > > org.apache.hadoop.hbase.client.NoServerForRegionException:
> Timed
> > > out
> > > > > > trying
> > > > > > > to locate root region
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
> > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > org.apache.hadoop.mapred.Child.main(Child.java:155)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Amandeep Khurana
> > > > > > > Computer Science Graduate Student
> > > > > > > University of California, Santa Cruz
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <
> > > amansk@gmail.com
> > > > > > >wrote:
> > > > > > >
> > > > > > >> I am trying to import data from a flat file into Hbase
using a
> > Map
> > > > > > Reduce
> > > > > > >> job. There are close to 2 million rows. Mid way into
the job,
> it
> > > > > starts
> > > > > > >> giving me connection problems and eventually kills
the job.
> When
> > > the
> > > > > > error
> > > > > > >> comes, the hbase shell also stops working.
> > > > > > >>
> > > > > > >> This is what I get:
> > > > > > >>
> > > > > > >> 2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
> > > > > Retrying
> > > > > > connect to server: /171.69.102.52:60020. Already tried 0
> time(s).
> > > > > > >>
> > > > > > >> What could be going wrong?
> > > > > > >>
> > > > > > >> Amandeep
> > > > > > >>
> > > > > > >>
> > > > > > >> Amandeep Khurana
> > > > > > >> Computer Science Graduate Student
> > > > > > >> University of California, Santa Cruz
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message