hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <ted...@yahoo.com.INVALID>
Subject Re: Hbase on HDFS versus Cassandra
Date Thu, 01 Dec 2016 09:09:58 GMT
w.r.t. "Unable to find cached index metadata" error, have you seen this ?


    On Wednesday, November 30, 2016 11:59 PM, Neelesh <neeleshs@gmail.com> wrote:

 Ted, we use HDP 2.3.4 (HBase 1.1.2, phoenix 4.4 - but with a lot of backports from later
The key of the data table is  <customerid- 11 bytes><userid- 36 bytes-  but it
really is a right padded long due to historic data><event type long><timestamp><random
The two global indexes are <customerid-11 bytes><event type long><timestamp>
<user id> and <customerid-11 bytes><campaign id long><timestamp> Around
100B rows in the main table. The main issues we see are
# Sudden spikes in queueSize - going all the way to 1G limit and staying there, without any
correlated client traffic# Boatloads of these errors 2016-11-30 11:28:54,907 INFO  [RW.default.writeRpcServer.handler=43,queue=9,port=16020]
util.IndexManagementUtil: Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ERROR
2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata.  key=120521194876100862
region=<region-key>. Index update failed
We have cross datacenter WAL replication enabled.We saw PHOENIX-1718, and changed all recommended
timeouts to 1 hour.  Our HBase version has HBase-11705. We also discovered that the queuesize
is global (across general/replication/priority queues) and if it reaches the 1GB limit, calls
to all queues will drop. That was interesting because even though the replication handlers
have  a different queue, the size is counted globally, affecting others. Please correct me
on this. I hope I'm wrong on this one :)
Our challenge has been to understand what's HBase doing under various scenarios. We monitor
call queue lengths, sizes and latencies as the primary alerting mechanism to tell us something
is going on with HBase.
On Wed, Nov 30, 2016 at 1:15 PM, Ted Yu <ted_yu@yahoo.com.invalid> wrote:

Neelesh:Can you share more details about the sluggish cluster performance (such as version
of hbase / phoenix, your schema, region server log snippet, stack traces, etc) ?
As hbase / phoenix evolve, I hope the performance keeps getting better for your use case.

    On Wednesday, November 30, 2016 10:07 AM, Neelesh <neeleshs@gmail.com> wrote:

 We use both, in different capacities. Cassandra is an x-DC archive store
with mostly batch writes and occasional key based reads. Hbase is for
real-time event ingestion. Our experience so far on hbase + phoenix is that
when it works, it is fast and scales like crazy. But if you ever hit a snag
around data patterns, you will have a VERY hard time figuring out what's
going on. A combination of global phoenix indexes and heavy writes leave an
entire cluster sluggish, if there is a hint of hotspotting.

On the other hand, we had a big struggle getting Cassandra when a node
recovery was in progress. What with twice the amount of disk requirements
during recovery etc. Other than that, it is quiet.
But the access patterns are not the same.

I think the old rule still stays. If you are already on hadoop , or
interested in using/analysing data in several different ways, go with hbase
. If you just need a big data store with a few predefined query patterns,
Cassandra is good

Of course, I'm biased towards HBase.

On Nov 30, 2016 7:02 AM, "Mich Talebzadeh" <mich.talebzadeh@gmail.com>

> Hi Guys,
> Used Hbase on HDFS reasonably well. Happy to to stick with it and more with
> Hive/Phoenix views and Phoenix indexes where I can.
> I have a bunch of users now vocal about the use case for Cassandra and
> whether it can do a better job than Hbase.
> Unfortunately I am no expert on Cassandra. However, some use case fit would
> be very valuable.
> Thanks
> Dr Mich Talebzadeh
> LinkedIn * https://www.linkedin.com/ profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCd OABUrV8Pw
> <https://www.linkedin.com/ profile/view?id= AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
> http://talebzadehmich. wordpress.com
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message