incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron McCurry (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BLUR-160) Create a back pressure feature in Blur when GC is too high.
Date Thu, 27 Jun 2013 11:12:20 GMT

    [ https://issues.apache.org/jira/browse/BLUR-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13694621#comment-13694621
] 

Aaron McCurry commented on BLUR-160:
------------------------------------

Yeah, this is a good one.  So in a system that I run in production we are seeing under high
load some bad queries.  These are either terrible wildcard or fuzzy or both (we are using
0.1.4 which uses lucene 3.6 which makes the problem worse than in lucene 4.0).  The front
end application that sends these queries to Blur does it's best to filter out bad queries
but some get through and some are valid queries.  And most of the time Blur can get through
them but sometimes there is a perfect storm of bad queries at the same time.  This has causes
us to has some slow nodes because of GC (and even OOM) that in turn cause ZK timeouts and
the nodes dies.  BLur then takes action and brings the shards that are down on another shard
server.  The controller reissues the queries to this new shard server, and it too goes down.
 Usually it stops there because the query times out.

So this task is here to try and detect an almost out of memory heap and do something about
it before it's too late.  I have created a new class that detects the situation and will take
action.  Now we need to have it clear out the jvm of queries (maybe more types of requests).
 The class is called GCWatcher and it has static methods to interact with.  Let me know what
you think.

Aaron
                
> Create a back pressure feature in Blur when GC is too high.
> -----------------------------------------------------------
>
>                 Key: BLUR-160
>                 URL: https://issues.apache.org/jira/browse/BLUR-160
>             Project: Apache Blur
>          Issue Type: Improvement
>          Components: Blur
>    Affects Versions: 0.2.0
>            Reporter: Aaron McCurry
>            Priority: Blocker
>             Fix For: 0.2.0
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message