hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taylor, Ronald C" <ronald.tay...@pnl.gov>
Subject RE: Still need help with data upload into HBase
Date Wed, 08 Apr 2009 21:49:39 GMT
 
Hi Ryan,

Thanks for the suggestion on checking whether the number of file handles
allowed actually gets increased after I make the change the
/etc/security/limits.conf.

Turns out it was not. I had to check with one of our sysadmins so that
the new 32K number of handles setting actually gets used on my Red Hat
box.

With that, and with one other change which I'll get to in a moment, I
finally was able to read in all the rows that I wanted, instead of the
program breaking before finishing. Checked the table by scanning it -
looks OK. So - it looks like things are working as they should. 

Thank you very much for the help.

Now as to the other parameter that needed changing: I found that the
xceivers (xcievers?) limit was not being bumped up - I was crashing on
that. I went to add what Ryan suggested in hadoop-site.xml, i.e.,

<property>
<name>dfs.datanode.max.xcievers</name>
<value>2047</value>
</property>

and discovered that I did not know whether to use
"dfs.datanode.max.xcievers" or "dfs.datanode.max.xceivers", where the
"i" and "e" switch. I was getting error msgs in the log files with

  "xceiverCount 257 exceeds the limit of concurrent xcievers 256"

 with BOTH spelling variants employed within the same error msg. Very
confusing. So I added property entries for both spellings in the
hadoop-site.xml file. Figured one of them would take effect. That
appears to work fine. But I would like get the correct spelling. Did a
Google search and the spelling keeps popping up both ways, so I remain
confused.

I think the Hbase getting started documentation could use some
enhancement on file handle settings, xceiver (xciever?) settings, and
datanode handler count settings.
 Ron

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, MSIN K7-90
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov
www.pnl.gov

-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com] 
Sent: Monday, April 06, 2009 6:47 PM
To: Taylor, Ronald C
Cc: hbase-user@hadoop.apache.org
Subject: Re: Still need help with data upload into HBase

I ran into a problem on ubuntu where /etc/security/limits.conf wasnt
being honored due to a missing line in /etc/pam.d/common-session:
"session required        pam_limits.so"

this prevented the ulimits from being run.

can you sudo to the hadoop/hbase user and verify with ulimit -a ?



On Mon, Apr 6, 2009 at 5:07 PM, Taylor, Ronald C
<ronald.taylor@pnl.gov>wrote:

>  Hello Ryan and the list,
>
> Well, I am still stuck. In addition to making the changes recommended 
> by Ryan to my hadoop-site.xml file (see below), I also added a line 
> for HBase to /etc/security/limits.conf and had the fs.file-max hugely 
> increased, to hopefully handle any file handle limit problem. Still no

> luck with my upload program. It fails about where it did before, 
> around the loading of  the 160,000th row into the one table that I 
> create in Hbase. Didn't  the "too many file open" msg, but did get 
> "handleConnectionFailure"  in the same place in the upload.
>
> I then tried a complete reinstall of Hbase and Hadoop, upgrading from 
> 0.19.0 to 0.19.1. Used the same config parameters as before, and reran

> the program. It fails again, at about the same number of rows uploaded

> - and I'm back to getting "too many files open" as what I think is the

> principal error msg.
>
> So - does anybody have any suggestions? I am running a
"pseudo-distributed"
> installation of Hadoop on one Red Hat Linux machine with about ~3Gb of
RAM.
> Are there any known problems with bulk uploads when running 
> "pseudo-distributed" on on a single box, rather than a true cluster? 
> Is there anything else I can try?
>  Ron
>
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group Pacific Northwest 
> National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, MSIN K7-90
> Richland, WA  99352 USA
> Office:  509-372-6568
> Email: ronald.taylor@pnl.gov
> www.pnl.gov
>
>
>  ------------------------------
> *From:* Ryan Rawson [mailto:ryanobjc@gmail.com]
> *Sent:* Friday, April 03, 2009 5:56 PM
> *To:* Taylor, Ronald C
> *Subject:* Re: FW: Still need help with data upload into HBase
>
> Welcome to hbase :-)
>
> This is pretty much how it goes for nearly every new user.
>
> We might want to review our docs...
>
> On Fri, Apr 3, 2009 at 5:54 PM, Taylor, Ronald C
<ronald.taylor@pnl.gov>wrote:
>
>>
>>
>> Thanks. I'll make those settings, too, in addition to bumping up the 
>> file handle limit, and give it another go.
>> Ron
>>
>> -----Original Message-----
>> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
>> Sent: Friday, April 03, 2009 5:48 PM
>> To: hbase-user@hadoop.apache.org
>>  Subject: Re: Still need help with data upload into HBase
>>
>> Hey,
>>
>> File handle - yes... there was a FAQ and/or getting started which 
>> talks about upping lots of limits.
>>
>> I have these set in my hadoop-site.xml (that is read by datanode):
>> <property>
>> <name>dfs.datanode.max.xcievers</name>
>> <value>2047</value>
>> </property>
>>
>> <property>
>> <name>dfs.datanode.handler.count</name>
>> <value>10</value>
>> </property>
>>
>> I should probably set the datanode.handler.count higher.
>>
>> Don't forget to toss a reasonable amount of ram at hdfs... not sure 
>> what that is exactly, but -Xmx1000m wouldn't hurt.
>>
>> On Fri, Apr 3, 2009 at 5:44 PM, Taylor, Ronald C
>> <ronald.taylor@pnl.gov>wrote:
>>
>> >
>> > Hi Ryan,
>> >
>> > Thanks for the info. Re checking the Hadoop datanode log file: I 
>> > just did so, and found a "too many open files" error. Checking the 
>> > Hbase
>> FAQ,
>> > I see that I should drastically bump up the file handle limit. So I
>> will
>> > give that a try.
>> >
>> > Question: what does the xciver variable do? My hadoop-site.xml file
>> does
>> > not contain any entry for such a var. (Nothing reported in the 
>> > datalog file either with the word "xciver".)
>> >
>> > Re using the local file system: well, as soon as I load a nice data
>> set
>> > loaded in, I'm starting a demo project manipulating it for our Env 
>> > Molecular Sciences Lab (EMSL), a DOE Nat User Facility. And I'm
>> supposed
>> > to be doing the manipulating using MapReduce programs, to show the 
>> > usefulness of such an approach. So I need Hadoop and the HDFS. And 
>> > so
>> I
>> > would prefer to keep using Hbase on top of Hadoop, rather than the
>> local
>> > Linux file system. Hopefully the "small HDFS clusters" issues you 
>> > mention are survivable. Eventually, some of this programming might
>> wind
>> > up on Chinook, our 160 Teraflop supercomputer cluster, but that's a
>> ways
>> > down the road. I'm starting on my Linux desktop.
>> >
>> > I'll try bumping up the file handle limit, restart Hadoop and 
>> > Hbase,
>> and
>> > see what happens.
>> > Ron
>> >
>> > ___________________________________________
>> > Ronald Taylor, Ph.D.
>> > Computational Biology & Bioinformatics Group Pacific Northwest 
>> > National Laboratory
>> > 902 Battelle Boulevard
>> > P.O. Box 999, MSIN K7-90
>> > Richland, WA  99352 USA
>> > Office:  509-372-6568
>> > Email: ronald.taylor@pnl.gov
>> > www.pnl.gov
>> >
>> >
>> > -----Original Message-----
>> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
>> > Sent: Friday, April 03, 2009 5:08 PM
>> > To: hbase-user@hadoop.apache.org
>> > Subject: Re: Still need help with data upload into HBase
>> >
>> > Hey,
>> >
>> > Can you check the datanode logs?  You might be running into the
>> dreaded
>> > xciver limit :-(
>> >
>> > try upping the xciver in hadoop-site.xml... i run at 2048.
>> >
>> > -ryan
>> >
>> > -----Original Message-----
>> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
>> > Sent: Friday, April 03, 2009 5:13 PM
>> > To: hbase-user@hadoop.apache.org
>> > Subject: Re: Still need help with data upload into HBase
>> >
>> > Non replicated yet is probably what you think - HDFS hasnt place
>> blocks
>> > on
>> > more nodes yet.  This could be due to the pseudo distributed nature

>> > of your set-up.  I'm not familiar with that configuration, so I 
>> > can't really
>> say
>> > more.
>> >
>> > If you only have 1 machine, you might as well just go with local
>> files.
>> > The
>> > HDFS gets you distributed replication, but until you have many
>> machines,
>> > it
>> > won't buy you anything and only cause problems, since small HDFS 
>> > clusters are known to have issues.
>> >
>> > Good luck (again!)
>> > -ryan
>> >
>> > On Fri, Apr 3, 2009 at 5:07 PM, Ryan Rawson <ryanobjc@gmail.com>
>> wrote:
>> >
>> > > Hey,
>> > >
>> > > Can you check the datanode logs?  You might be running into the
>> > dreaded
>> > > xciver limit :-(
>> > >
>> > > try upping the xciver in hadoop-site.xml... i run at 2048.
>> > >
>> > > -ryan
>> > >
>> > >
>> > > On Fri, Apr 3, 2009 at 4:35 PM, Taylor, Ronald C
>> > <ronald.taylor@pnl.gov>wrote:
>> > >
>> > >>
>> > >> Hello folks,
>> > >>
>> > >> I have just tried using Ryan's doCommit() method for my bulk 
>> > >> upload
>> > into
>> > >> one Hbase table. No luck. I still start to get errors around row

>> > >> 160,000. On-screen, the program starts to generate error msgs 
>> > >> like
>> > so:
>> > >> ...
>> > >> INFO: Retrying connect to server: /127.0.0.1:60383. Already 
>> > >> tried 8 time(s).
>> > >> Apr 3, 2009 2:39:52 PM
>> > >> org.apache.hadoop.hbase.ipc.HBaseClient$Connection
>> > >> handleConnectionFailure
>> > >> INFO: Retrying connect to server: /127.0.0.1:60383. Already 
>> > >> tried 9 time(s).
>> > >> Apr 3, 2009 2:39:57 PM
>> > >> org.apache.hadoop.hbase.ipc.HBaseClient$Connection
>> > >> handleConnectionFailure
>> > >> INFO: Retrying connect to server: /127.0.0.1:60383. Already 
>> > >> tried 0 time(s).
>> > >> Apr 3, 2009 2:39:58 PM
>> > >> org.apache.hadoop.hbase.ipc.HBaseClient$Connection
>> > >> handleConnectionFailure
>> > >> INFO: Retrying connect to server: /127.0.0.1:60383. Already 
>> > >> tried 1 time(s).
>> > >> ...
>> > >> In regard to log file information, I have appended at bottom 
>> > >> some
>> of
>> > the
>> > >> output from my hbase-<user>-master-<machine>.log file,
at the 
>> > >> place where it looks to me like things might have started to go
wrong.
>> > Several
>> > >> questions:
>> > >>
>> > >> 1)  Is there any readily apparent cause for such a 
>> > >> HBaseClient$Connection handleConnectionFailure to occur in a 
>> > >> Hbase installation configured on a Linux desktop to work in the 
>> > >> pseudo-distributed operation mode? From my understanding, even
>> > importing
>> > >> ~200,000 rows (each row being filled with info for ten columns) 
>> > >> is
>> a
>> > >> minimal data set for Hbase, and upload should not be failing 
>> > >> like
>> > this.
>> > >>
>> > >> FYI - minimal changes were made to the Hbase default settings in
>> the
>> > >> Hbase ../conf/ config files when I installed Hbase 0.19.0. I 
>> > >> have
>> one
>> > >> entry in hbase-env.sh, to set JAVA_HOME, and one property entry 
>> > >> in hbase-site.xml, to set the hbase.rootdir.
>> > >>
>> > >> 2) My Linux box has about 3 Gb of memory. I left the HADOOP_HEAP
>> and
>> > >> HBASE_HEAP sizes at their default values, which I understand are
>> > 1000Mb
>> > >> each. Should I have changed either value?
>> > >>
>> > >> 3) I left the dfs.replication value at the default of  "3" in 
>> > >> the hadoop-site.xml file, for my test of pseudo-distributed
operation.
>> > >> Should I have changed that to "1", for operation on my single
>> > machine?
>> > >> Downsizing to "1" would appear to me to negate trying out Hadoop

>> > >> in
>> > the
>> > >> pseudo-distributed operation mode, so I left the value "as is", 
>> > >> but
>> > did
>> > >> I get this wrong?
>> > >>
>> > >> 4) In the log output below, you can see that Hbase starts to 
>> > >> block
>> > and
>> > >> then unblock updates to my one Hbase table (called the 
>> > >> "ppInteractionTable", for protein-protein interaction table). A
>> > little
>> > >> later, a msg says that the ppInteractionTable has been closed. 
>> > >> At
>> > this
>> > >> point, my program has *not* issued a command to close the table 
>> > >> -
>> > that
>> > >> only happens at the end of the program. So - why is this
happening?
>> > >>
>> > >> Also, near the end of my log extract, I get a different error
msg:
>> > >> NotReplicatedYetException. I have no idea what that means.
>> Actually,
>> > I
>> > >> don't really have a grasp yet on what any of these error msgs is

>> > >> supposed to tell us. So - once again, any help would be much 
>> > >> appreciated.
>> > >>
>> > >>  Ron
>> > >>
>> > >> ___________________________________________
>> > >> Ronald Taylor, Ph.D.
>> > >> Computational Biology & Bioinformatics Group Pacific Northwest

>> > >> National Laboratory
>> > >> 902 Battelle Boulevard
>> > >> P.O. Box 999, MSIN K7-90
>> > >> Richland, WA  99352 USA
>> > >> Office:  509-372-6568
>> > >> Email: ronald.taylor@pnl.gov
>> > >> www.pnl.gov
>> > >>
>> > >>
>> > >>
>> > >> -----Original Message-----
>> > >> From: Taylor, Ronald C
>> > >> Sent: Tuesday, March 31, 2009 5:48 PM
>> > >> To: 'hbase-user@hadoop.apache.org'
>> > >> Cc: Taylor, Ronald C
>> > >> Subject: Novice Hbase user needs help with data upload - gets a 
>> > >> RetriesExhaustedException, followed by 
>> > >> NoServerForRegionException
>> > >>
>> > >>
>> > >> Hello folks,
>> > >>
>> > >> This is my first msg to the list - I just joined today, and I am

>> > >> a novice Hadoop/HBase programmer. I have a question:
>> > >>
>> > >> I have written a Java program to create an HBase table and then
>> enter
>> > a
>> > >> number of rows into the table. The only way I have found so far 
>> > >> to
>> do
>> > >> this is to enter each row one-by-one, creating a new BatchUpdate

>> > >> updateObj for each row, doing about ten updateObj.put()'s to add
>> the
>> > >> column data, and then doing a tableObj.commit(updateObj). 
>> > >> There's probably a more efficient way (happy to hear, if so!), 
>> > >> but this is
>> > what
>> > >> I'm starting with.
>> > >>
>> > >> When I do this on input that creates 3000 rows, the program 
>> > >> works
>> > fine.
>> > >> When I try this on input that would create 300,000 rows (still 
>> > >> relatively small for an HBase table, I would think), the program

>> > >> terminates around row 160,000 or so, generating first an 
>> > >> RetriesExhaustedException, followed by
NoServerForRegionException.
>> > The
>> > >> HBase server crashes, and I have to restart it. The Hadoop 
>> > >> server appears to remain OK and does not need restarting.
>> > >>
>> > >> Can anybody give me any guidance? I presume that I might need to
>> > adjust
>> > >> some setting for larger input in the HBase and/or Hadoop config
>> > files.
>> > >> At present, I am using default settings. I have installed Hadoop
>> > 0.19.0
>> > >> and HBase 0.19.0 in the "pseudo" cluster mode on a single 
>> > >> machine,
>> my
>> > >> Red Hat Linux desktop, which has 3 Gb RAM.
>> > >>
>> > >> Any help / suggestions would be much appreciated.
>> > >>
>> > >>  Cheers,
>> > >>   Ron Taylor
>>
>

Mime
View raw message