hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu, Raymond" <raymond....@intel.com>
Subject RE: How is DataXceiver been used?
Date Fri, 18 Jan 2013 01:15:13 GMT
Hi Tariq

Thanks, this blog is very informational!

I roughly figure out the usage of xceiver. While the weird thing is even according to this
blog, I should have far enough xceiever in my full table scan job. Especially if it count
the number of storefile instead of real blk_xxxx tfile on nativeFS. Then I only have around
24 storefiles for 24 region since I do major compact them. And then it exceeding 1024 limit
during full table scan.... 

Might need more investigation into my case to check out what wrong with it.


> 
> Hello Raymond,
> 
>             You might find this
> link<http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/>helpful.
> It explains the problem
> and solution in great detail.
> 
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
> 
> 
> On Thu, Jan 17, 2013 at 12:14 PM, Liu, Raymond
> <raymond.liu@intel.com>wrote:
> 
> > Hi
> >
> >         I have a table with about 24 region on the one regionserver,
> > and each region have about 20 block files on hdfs.
> >         The xceiverCount is set to 1024, I have thought that this is
> > quite enough since at most 480 blocks will be opened.
> >         While when I do a MR job to scan the table, with 24 map task
> > each open and scan a different region at the same time, it turn out
> > that the DataXceiver is run out...
> >
> >         I am a little bit puzzled, those blocks will only be read by
> > one task, then shouldn't region server scan blocks one by one? And
> > since there are 480 blocks at most, how can it use up dataXceiver?
> >
> > Best Regards,
> > Raymond Liu
> >
> >

Mime
View raw message