hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeesoo Shin <bsh...@gmail.com>
Subject Re: wal.FSHLog: Slow sync cost: 143 ms, current pipeline:
Date Wed, 21 Oct 2015 14:32:40 GMT
FSHLog: Slow sync cost:143 ms, current ...
I have seen this messages at recent performance test.
This showed up when hdfs had hard time catching up.
(I did a lot of put request. )
check read/write byte of hdfs jmx to confirm.

Maybe when you added new RS server,
loadbalance kicked in and many region got moved.
when new assigned region compacted,
data got moved a lot for locality and caused this message.

regard,
2015. 10. 18. 오전 3:24에 "Artem Ervits" <artemervits@gmail.com>님이 작성:

> The slow sync messages appeared across all servers as soon new RS were
> added. Would it make a difference if I said it was on Azure using standard
> storage? Should I look at GC settings across DN, RS and NN?
> On Oct 17, 2015 12:23 PM, "Ted Yu" <yuzhihong@gmail.com> wrote:
>
> > From your first post, I got the impression that 'Slow sync' appeared in
> new
> > region server logs.
> >
> > Can you monitor disk activities across the 6 datanodes ?
> > Take into consideration of YARN usage.
> >
> > Cheers
> >
> > On Sat, Oct 17, 2015 at 9:07 AM, Artem Ervits <artemervits@gmail.com>
> > wrote:
> >
> > > The initial three RS were on Kafka brokers with two disks shared
> between
> > > Kafka logs and HDFS. HBase 1.1.1. The new RS servers are 3 disks each,
> > with
> > > RS, DN and NodeManagers on them.
> > > On Oct 17, 2015 11:36 AM, "Ted Yu" <yuzhihong@gmail.com> wrote:
> > >
> > > bq. once I added more regionservers
> > >
> > > Were any of the new regionservers on the Kafka broker nodes ?
> > >
> > > Which release of hbase are you using ? Looks like 1.x since the log was
> > > added by HBASE-11240
> > >
> > > Thanks
> > >
> > > On Sat, Oct 17, 2015 at 8:22 AM, Artem Ervits <artemervits@gmail.com>
> > > wrote:
> > >
> > > > Hello all, trying to address a sudden change in performance,
> processing
> > > > Kafka, Storm, HBase pipeline. I'm seeing error wal.FSHLog: Slow sync
> > > cost:
> > > > 143 ms, current pipeline:, this started appearing once I added more
> > > > regionservers. Is there a problem with small Xmx value for datanode?
> > > That's
> > > > what I've found so far in searches. Table is presplit with no
> > hotspotting
> > > > to be seen. I See the message on all RS. I do have 3 RS servers
> > colocated
> > > > with Kafka brokers and Kafka disks share storage with datanode but
> > > > performance was never an issue before.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message