hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Baranau <alex.barano...@gmail.com>
Subject Re: HBase Block locality always 0
Date Mon, 18 May 2015 23:18:07 GMT
Sorry if I'm asking a silly question... Are you sure your RSs and Datanodes
are all up and running? Are you sure they are collocated?

> Datanode on l-hbase[26-31].data.cn8 and regionserver on
> l-hbase[25-31].data.cn8,

Could be that your only live RS is on l-hbase25.data.cn8, which would cause
that behavior... Btw, why 25th is not collocated with datanode?

Alex Baranau
--
http://cdap.io - open source framework to build and run data applications
on Hadoop & HBase

On Fri, May 15, 2015 at 8:12 PM, Louis Hust <louis.hust@gmail.com> wrote:

> Hi, Esteban,
>
> Hadoop Version 2.2.0, r1537062.
> So i do not know why it always write other datanode instead of local
> datanode,
> If there is some log for the hdfs write policy? And now the cluster is
> working not healthy,
> with heavy networking.
>
> 2015-05-15 1:28 GMT+08:00 Esteban Gutierrez <esteban@cloudera.com>:
>
> > Hi Louis,
> >
> > Locality 0 is not right for a cluster of that size and having 3 replicas
> > per block unless all RS cannot connect to the local DN and somehow the
> > local DN to the RS is always excluded from the pipeline. In Hadoop
> > 2.0-alpha there was a bug (HDFS-3224) that caused the NN to report a DN
> as
> > live and dead if the storage ID was changed in a single volume (e.g.
> after
> > replacing one drive) and that caused fs.getFileBlockLocations() to report
> > less blocks for calculating the HDFS locality index. Unless your cluster
> is
> > using Hadoop 2.0-alpha I won't worry too much about that.
> >
> > Regarding the logs its odd that the JN is taking about 1.5 seconds just
> to
> > send less than 200 bytes. Perhaps some IO contention issue is going on in
> > your cluster?
> >
> > thanks,
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Thu, May 14, 2015 at 5:48 AM, Louis Hust <louis.hust@gmail.com>
> wrote:
> >
> > > Hi, Esteban
> > >
> > > Each region server has about 122 regions, data is large. HDFS replica
> is
> > > defined as default 3,  and namenode have some WARN like below.
> > >
> > > {log}
> > > 2015-05-14 20:45:37,463 WARN
> > > org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took
> 1503ms
> > to
> > > send a batch of 3 edits (179 bytes) to remote journal
> 192.168.44.29:8485
> > > {/log}
> > >
> > > Regionserver's log seems normal:
> > >
> > > {log}
> > > 2015-05-14 20:46:59,890 INFO  [Thread-15] regionserver.HRegion:
> Finished
> > > memstore flush of ~44.4 M/46586984, currentsize=0/0 for region
> > >
> > >
> >
> qmq_backup,0066485937885860620cb396a3e65c6c9de92cae9aa29,1412429632233.65684ef65f58cb3e27986ca38d397bee.
> > > in 3141ms, sequenceid=7493455453, compaction requested=true
> > > 2015-05-14 20:46:59,890 INFO
> > >  [regionserver60020-smallCompactions-1431462564717]
> regionserver.HRegion:
> > > Starting compaction on m in region
> > >
> > >
> >
> qmq_backup,0066485937885860620cb396a3e65c6c9de92cae9aa29,1412429632233.65684ef65f58cb3e27986ca38d397bee.
> > > {/log}
> > >
> > > Any idea?
> > >
> > >
> > >
> > > 2015-05-13 1:26 GMT+08:00 Esteban Gutierrez <esteban@cloudera.com>:
> > >
> > > > Hi,
> > > >
> > > > How many regions you per RS? one possibility is that you have very
> > little
> > > > data in your cluster and regions have moved around and there are no
> > > blocks
> > > > in the local DN to the RS. Another possibility is that you have one
> > > replica
> > > > configured and regions moved too so that makes even harder to have
> some
> > > > local blocks in the DN to the RS. Lastly it could be some other
> problem
> > > > where the HDFS pipeline has excluded the local DN. Have you seen any
> > > > exception in the RSs or the NameNode that might be interesting?
> > > >
> > > > thanks,
> > > > esteban.
> > > >
> > > >
> > > >
> > > > --
> > > > Cloudera, Inc.
> > > >
> > > >
> > > > On Tue, May 12, 2015 at 2:59 AM, 娄帅 <louis.hust.ml@gmail.com>
wrote:
> > > >
> > > > > Hi, all,
> > > > >
> > > > > I am maintaining an hbase 0.96.0 cluster, but from the web ui of
> > HBase
> > > > > regionserver,
> > > > > i saw Block locality is 0 for all regionserver.
> > > > >
> > > > > Datanode on l-hbase[26-31].data.cn8 and regionserver on
> > > > > l-hbase[25-31].data.cn8,
> > > > >
> > > > > Any idea?
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message