hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Louis Hust <louis.h...@gmail.com>
Subject Re: HBase Block locality always 0
Date Sat, 16 May 2015 03:12:30 GMT
Hi, Esteban,

Hadoop Version 2.2.0, r1537062.
So i do not know why it always write other datanode instead of local
datanode,
If there is some log for the hdfs write policy? And now the cluster is
working not healthy,
with heavy networking.

2015-05-15 1:28 GMT+08:00 Esteban Gutierrez <esteban@cloudera.com>:

> Hi Louis,
>
> Locality 0 is not right for a cluster of that size and having 3 replicas
> per block unless all RS cannot connect to the local DN and somehow the
> local DN to the RS is always excluded from the pipeline. In Hadoop
> 2.0-alpha there was a bug (HDFS-3224) that caused the NN to report a DN as
> live and dead if the storage ID was changed in a single volume (e.g. after
> replacing one drive) and that caused fs.getFileBlockLocations() to report
> less blocks for calculating the HDFS locality index. Unless your cluster is
> using Hadoop 2.0-alpha I won't worry too much about that.
>
> Regarding the logs its odd that the JN is taking about 1.5 seconds just to
> send less than 200 bytes. Perhaps some IO contention issue is going on in
> your cluster?
>
> thanks,
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Thu, May 14, 2015 at 5:48 AM, Louis Hust <louis.hust@gmail.com> wrote:
>
> > Hi, Esteban
> >
> > Each region server has about 122 regions, data is large. HDFS replica is
> > defined as default 3,  and namenode have some WARN like below.
> >
> > {log}
> > 2015-05-14 20:45:37,463 WARN
> > org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took 1503ms
> to
> > send a batch of 3 edits (179 bytes) to remote journal 192.168.44.29:8485
> > {/log}
> >
> > Regionserver's log seems normal:
> >
> > {log}
> > 2015-05-14 20:46:59,890 INFO  [Thread-15] regionserver.HRegion: Finished
> > memstore flush of ~44.4 M/46586984, currentsize=0/0 for region
> >
> >
> qmq_backup,0066485937885860620cb396a3e65c6c9de92cae9aa29,1412429632233.65684ef65f58cb3e27986ca38d397bee.
> > in 3141ms, sequenceid=7493455453, compaction requested=true
> > 2015-05-14 20:46:59,890 INFO
> >  [regionserver60020-smallCompactions-1431462564717] regionserver.HRegion:
> > Starting compaction on m in region
> >
> >
> qmq_backup,0066485937885860620cb396a3e65c6c9de92cae9aa29,1412429632233.65684ef65f58cb3e27986ca38d397bee.
> > {/log}
> >
> > Any idea?
> >
> >
> >
> > 2015-05-13 1:26 GMT+08:00 Esteban Gutierrez <esteban@cloudera.com>:
> >
> > > Hi,
> > >
> > > How many regions you per RS? one possibility is that you have very
> little
> > > data in your cluster and regions have moved around and there are no
> > blocks
> > > in the local DN to the RS. Another possibility is that you have one
> > replica
> > > configured and regions moved too so that makes even harder to have some
> > > local blocks in the DN to the RS. Lastly it could be some other problem
> > > where the HDFS pipeline has excluded the local DN. Have you seen any
> > > exception in the RSs or the NameNode that might be interesting?
> > >
> > > thanks,
> > > esteban.
> > >
> > >
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > > On Tue, May 12, 2015 at 2:59 AM, 娄帅 <louis.hust.ml@gmail.com> wrote:
> > >
> > > > Hi, all,
> > > >
> > > > I am maintaining an hbase 0.96.0 cluster, but from the web ui of
> HBase
> > > > regionserver,
> > > > i saw Block locality is 0 for all regionserver.
> > > >
> > > > Datanode on l-hbase[26-31].data.cn8 and regionserver on
> > > > l-hbase[25-31].data.cn8,
> > > >
> > > > Any idea?
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message