hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sathyafmt <sathya...@gmail.com>
Subject Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Date Wed, 25 Feb 2015 18:26:00 GMT
Hi John,

What's the fix (committed on Jan11) you mentioned ? Is it in opentsdb/hbase
? Do you have a JIRA #.. ?


On Wed, Feb 25, 2015 at 5:49 AM, brady2 [via Apache HBase] <
ml-node+s679495n4068627h88@n3.nabble.com> wrote:

> Hi Sathya and Nick,
> Here are the stack traces of the region server dumps when the huge .tmp
> files are created:
> https://drive.google.com/open?id=0B1tQg4D17jKQNDdFZkFQTlg4ZjQ&authuser=0
> As background we are not using compression. Compaction is occurs every
> hour. Everything else is default.
> OpenTSDB v2.0 is running on top of Cloudera 5.3.1 in AWS. We have a 7 node
> Cloudera cluster(each node with 32GB ram and 3TB disk space), with 5
> OpenTSDB instances dedicated for writing and 2 for reading. We are using
> AWS ELB’s in front of OpenTSDB to balance the read/writes.
> We are load testing OpenTSDB using SOCKETS, but running into several
> issues. Let me explain first how we do this load testing:
> 1.From another AWS system, we have written a testing framework to generate
> load.
> 2. The framework takes several parameters, we can specify the number of
> threads, the loop size (i.e. the number of sockets that each thread will
> open) and the batch size (i.e. the number of PUT’s, or inserts, that each
> socket connection will handle).
> 3. To simplify troubleshooting, we removed variables from the tests, we
> have just 1 OpenTSDB instance behind the AWS ELB so the load is being sent
> to 1 instance only.
> 4. We are initially creating the openTSDB tables without any pre-splitting
> of regions.
> 5. We are doing the loading with 1 metric only for ease of querying in the
> UI.
> 6. We are sending under 5000 inserts per second:
> 7. At the top of the hour, the row compaction kicks in and the region
> server is too busy so we lose data. it recovers the first time. But the 2nd
> hour, there is so much data presumably, that it doesn’t recover. To fix it,
> we have to restart cloudera, reboot the nodes, drop the tsdb tables and
> re-create them. Otherwise the .tmp file keeps growing until it fills the
> 3TB disks and the system is unresponsive.
> 8. We see problems with region splits happening under heavy load. We noted
> a code fix committed on Jan 11 for this but I presume that is not in RC2.1.
> Thanks
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068627.html
>  To unsubscribe from HBase with opentsdb creates huge .tmp file & runs out
> of hdfs space, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4067577&code=c2F0aHlhZm10QGdtYWlsLmNvbXw0MDY3NTc3fDUxNzU0MjkyMA==>
> .
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068650.html
Sent from the HBase User mailing list archive at Nabble.com.
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message