Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Wed, 8 Jan 2014 19:50:50 +0000 (UTC)
From: "Alex Moundalexis (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12684569.1386973808859.63544.1389210650897@arcas>
In-Reply-To: <JIRA.12684569.1386973808859@arcas>
References: <JIRA.12684569.1386973808859@arcas>
Subject: [jira] [Commented] (ACCUMULO-2014) Feedback loop on Tracer table
 given small in-memory map on server
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ACCUMULO-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13865799#comment-13865799 ] 

Alex Moundalexis commented on ACCUMULO-2014:
--------------------------------------------

Experienced the same issue on a new (and idle) 3 node system, but saw a lower rate of ingest (~150 entries/second). Trace log was empty except for startup message and configuration dump.

Only two tables in use, after 3 hours idle uptime:
38 entries in !METADATA
1.08M entries in trace

Did not increase in-memory-map size, but restarting the cluster didn't have any impact, but restarting the tracer seemed to work. Can provide screenshots and/or logs if desired.


> Feedback loop on Tracer table given small in-memory map on server
> -----------------------------------------------------------------
>
>                 Key: ACCUMULO-2014
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2014
>             Project: Accumulo
>          Issue Type: Bug
>          Components: trace, tserver
>    Affects Versions: 1.4.3
>            Reporter: Sean Busbey
>            Assignee: Eric Newton
>            Priority: Minor
>         Attachments: sanitized.tar.gz
>
>
> Under the right conditions the tracer table can get into a feedback loop where messages about flushes to a tracer table tablet keep arriving and triggering subsequent flush operations.
> This was observed on a 1.4.3 cluster initially configured for 256MB in-memory maps.
> This manifests as a high rate of ingest to the tracer table, even in the absence of client activity or any client created tables. i.e. on a 3  worker node cluster with no external activity 500 entries/second into the tracer table.
> Viewing the tserver logs:
> {noformat}
> tserver.log (less AUDIT):
> $ tail -50 tserver-3-example.com.log | grep -v AUDIT
> 2013-12-10 19:08:22,433 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent 1<;b99f as alias 5
> 2013-12-10 19:08:22,627 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent !0<;~ as alias 2
> 2013-12-10 19:08:23,431 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent 1<;b99f as alias 5
> 2013-12-10 19:08:23,735 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent !0<;~ as alias 2
> 2013-12-10 19:08:24,431 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent 1<;b99f as alias 5
> 2013-12-10 19:08:24,875 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent !0<;~ as alias 2
> 2013-12-10 19:08:26,431 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent 1<;b99f as alias 5
> 2013-12-10 19:08:26,486 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent !0<;~ as alias 2
> tserver.log (full)
> 2013-12-10 19:07:44,479 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials root: authenticated
> 2013-12-10 19:07:44,480 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table 1 for root
> 2013-12-10 19:07:44,561 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
> 2013-12-10 19:07:45,379 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission READ on table !0 for !SYSTEM
> 2013-12-10 19:07:45,379 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: got authorizations for !SYSTEM
> 2013-12-10 19:07:45,479 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials root: authenticated
> 2013-12-10 19:07:45,480 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table 1 for root
> 2013-12-10 19:07:45,481 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent 1<;b99f as alias 5
> 2013-12-10 19:07:45,554 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
> 2013-12-10 19:07:45,555 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent !0<;~ as alias 2
> 2013-12-10 19:07:45,731 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
> 2013-12-10 19:07:46,479 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials root: authenticated
> 2013-12-10 19:07:46,481 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table 1 for root
> 2013-12-10 19:07:46,482 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent 1<;b99f as alias 5
> 2013-12-10 19:07:46,574 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
> 2013-12-10 19:07:46,575 INFO org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent !0<;~ as alias 2
> 2013-12-10 19:07:46,743 AUDIT org.apache.accumulo.server.security.Auditor: Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
> {noformat}
> note that this example relies on the default configuration, using "root" as the user for the tracer role. If your tracer user is set to something else, that is the user that would show up in the credential checks above.
> Current work around is to increase the memory map size.
> We should attempt to detect this situation and at least issue a clear log message. It would be nicer if we could detect it and turn off tracing messages related to the tracing table.


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)