hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: 0.96.0 keep logging "Memstore is above high water mark"
Date Tue, 03 Sep 2013 18:38:24 GMT
The reason I was asking is if you used --nomapred, all your clients would
share the same connection so only one stack trace should be printed (when
the connection is closed).

Since you're running with MR, how many stack traces do you see? Still one?
If so, is it a local job runner? I'm trying to understand how many
connections you are using.

In any case, it's a normal-ish stack trace.

J-D


On Tue, Sep 3, 2013 at 11:14 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> I was running a MR with 10 clients, so it's a MapRed one.
>
> Just retried with only 1 thread and got the same exception.
>
> So it's not making any difference.
>
>
> 2013/9/3 Jean-Daniel Cryans <jdcryans@apache.org>
>
> > Alright so your ZK is running inside the master and that's where you're
> > seeing those lines, and they are normal if each thread has a different
> > connection... Are you doing a --nomapred PE?
> >
> >
> > On Tue, Sep 3, 2013 at 10:50 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > Oh, ok ;) I just un-packed the jar and ran it. So my config file is:
> > > <configuration>
> > > </configuration>
> > >
> > >
> > >
> > > 2013/9/3 Jean-Daniel Cryans <jdcryans@apache.org>
> > >
> > > > On Tue, Sep 3, 2013 at 10:35 AM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org> wrote:
> > > >
> > > > > > Your 10 clients are disconnecting from ZK, you're letting HBase
> > > manage
> > > > > it?
> > > > > Yep, I start PE from the command line, I don't expect to have to
do
> > > > > anything after that. So issue is on PE side?
> > > > >
> > > >
> > > > Not really what I asked, I wanted to know if you set HBASE_MANAGES_ZK
> > at
> > > > all in hbase-env.
> > > >
> > > >
> > > > >
> > > > > > Please don't remove the warn. It is important for troubleshooting
> > and
> > > > > sizing.
> > > > > Can you please tell more on how it helps to do sizing? Interested.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > JM
> > > > >
> > > > >
> > > > > 2013/9/3 Kevin O'dell <kevin.odell@cloudera.com>
> > > > >
> > > > > > Please don't remove the warn. It is important for troubleshooting
> > and
> > > > > > sizing.
> > > > > > On Sep 3, 2013 1:29 PM, "Jean-Daniel Cryans" <
> jdcryans@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > On Mon, Sep 2, 2013 at 12:04 PM, Jean-Marc Spaggiari <
> > > > > > > jean-marc@spaggiari.org> wrote:
> > > > > > >
> > > > > > > > While running PE with 10 clients, server keep logging:
> > > > > > > > 2013-09-02 14:56:59,919 WARN
>  [RpcServer.handler=5,port=44439]
> > > > > > > > regionserver.MemStoreFlusher: Memstore is above high
water
> mark
> > > and
> > > > > > > > block 2362ms
> > > > > > > > 2013-09-02 14:56:59,919 WARN
>  [RpcServer.handler=18,port=44439]
> > > > > > > > regionserver.MemStoreFlusher: Memstore is above high
water
> mark
> > > and
> > > > > > > > block 2363ms
> > > > > > > >
> > > > > > >
> > > > > > > Yeah that was added in HBASE-6466, it helps tracing when
the
> > > handlers
> > > > > are
> > > > > > > blocked on the memstores, else you have to match the "Blocking
> > > > updates"
> > > > > > > with the "Unblocking updates" lines. I'd just be fine adding
> the
> > > time
> > > > > > into
> > > > > > > the "Unblocking updates" line and remove that WARN.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Then when test is over:
> > > > > > > > 2013-09-02 14:57:02,280 WARN
> > > > > > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> > > server.NIOServerCnxn:
> > > > > > > > caught end of stream exception
> > > > > > > > EndOfStreamException: Unable to read additional data
from
> > client
> > > > > > > > sessionid 0x140dfdfc6270044, likely client has closed
socket
> > > > > > > >     at
> > > > > > > >
> > > > >
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> > > > > > > >     at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> > > > > > > >     at java.lang.Thread.run(Thread.java:662)
> > > > > > > >
> > > > > > >
> > > > > > > Your 10 clients are disconnecting from ZK, you're letting
HBase
> > > > manage
> > > > > > it?
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > /hbase-0.96.0$ cat logs/hbase-jmspaggiari-master-t430s.log
 |
> > > grep
> > > > > > > > "Memstore is above high water mark and block" | wc
> > > > > > > >    3555   49770  514558
> > > > > > > >
> > > > > > > >
> > > > > > > > /hbase-0.96.0$ cat logs/hbase-jmspaggiari-master-t430s.log
 |
> > > grep
> > > > > > > > "Unable to read additional data from client sessionid"
| wc
> > > > > > > >     102    1530   12852
> > > > > > > >
> > > > > > > > I guess it's only because of PE, but is this something
which
> > need
> > > > to
> > > > > > > > be looked at?
> > > > > > > >
> > > > > > > > JM
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message