Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of josh.elser@gmail.com designates
 209.85.216.48 as permitted sender)
Message-ID: <501C7406.70803@gmail.com>
Date: Fri, 03 Aug 2012 20:59:50 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:14.0) Gecko/20120713 Thunderbird/14.0
MIME-Version: 1.0
To: user@accumulo.apache.org
Subject: Re: Accumulo Caching for benchmarking
References: 
 <CAPhC5mq0RUJgdDTJmC6s8yLNCi6e80GT20jf-O5Z-waJY1iRFg@mail.gmail.com>
 <CAMz+DuvYQLBrwWh5nFCJYhHBnrsY6qw8vZx1BUABE2GFOMq04w@mail.gmail.com>
In-Reply-To: 
 <CAMz+DuvYQLBrwWh5nFCJYhHBnrsY6qw8vZx1BUABE2GFOMq04w@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I remember listening to a Keith presentation about the testing against 
the multi-level RFile index which was introduced in 1.4.0.

You also want to think about caching at the operating system level. I'm 
not entirely positive what Keith did to try to mitigate this, but I 
imagine writing a bunch of garbage from /dev/urandom out to disk should 
work. That, or you could actually reboot the nodes.

On 8/3/2012 8:55 PM, William Slacum wrote:
> Steve, I'm a little confused. The Rfile block cache is tied to a
> TServer, so if you kill a node, its cache should go away. Are you
> querying for the same data after you kill the node that hosted the
> tablet which contained the data? Also, between runs, you could stop and
> restart everything, thereby eliminating the cache.
>
> On Fri, Aug 3, 2012 at 5:50 PM, Steven Troxell <steven.troxell@gmail.com
> <mailto:steven.troxell@gmail.com>> wrote:
>
>     Hi  all,
>
>     I am running a benchmarking project on accumulo looking at RDF
>     queries for clusters with different node sizes.   While I intend to
>     look at caching for each optimizing each individual run, I do NOT
>     want caching to interfere for example between runs involving the use
>     of 10 and 8 tablet servers.
>
>     Up to now I'd just been killing nodes via the bin/stop-here.sh
>     script but I realize that may have allowed caching from previous
>     runs with different node sizes to influence my results.   It seemed
>     weird to me for exmaple when I realized dropping nodes actually
>     increased performance (as measured by query return times) in some
>     cases (though I acknowledge the code I'm working with has some
>     serious issues with how ineffectively it is actually utilizing
>     accumulo, but that's an issue I intend to address later).
>
>     I suppose one way would be between a change of node sizes,  stop and
>     restart ALL nodes ( as opposed to what I'd been doing in just
>     killing 2 nodes for example in transitioning from a 10 to 8 node
>     test).  Will this be sure to clear the influence of caching across
>     runs, and is there any cleaner way to do this?
>
>     thanks,
>     Steve
>
>