accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Vines <vi...@apache.org>
Subject Re: ISAM file location vs. read performance
Date Sun, 12 Jan 2014 17:36:43 GMT
So I'm not certain on our performance with short circuit reads, aside from
them being better.

But because of the way hdfs writes get distributed, a tablet server has a
strong probability of being a local read, so that is there. This is because
a tserver with ultimately end up major compacting it's files, ensuring
locality. So simply constantly ingesting will lead to eventual locality if
it wasn't there before. It just so happens those reads go through a
datanode, but not necessarily through the network.

Sent from my phone, please pardon the typos and brevity.
On Jan 12, 2014 12:29 PM, "Arshak Navruzyan" <arshakn@gmail.com> wrote:

> One aspect of Accumulo architecture is still unclear to me.  Would you
> achieve better scan performance if you could guarantee that the tablet and
> its ISAM file lived on the same node?  Guessing ISAM files are not
> splittable so they pretty much stay on one HDFS data node (plus the replica
> copy). Or is the theory that SATA and a 10GBps network provide more or less
> the same throughput?
>
> I generally understand that as the table grows and Accumulo creates more
> splits (tablets) you get better distribution over the cluster but seems
> like data location would still be important.   HBase folks seem to think
> that you can approx. double your throughput if let the region server
> directly read the file (dfs.client.read.shortcircuit=true) as opposed to
> going through the data node. (
> http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf).  Perhaps this is
> due more to HDFS overhead?
>
> I do get that one really nice thing about Accumulo's architecture is that
> it costs almost nothing to reassign tablet to a different tserver and this
> is a huge problem for other systems.
>
>
>

Mime
View raw message