What about the case when even the HS2 Service restart is also not working.

So my bad is that after restarting HS2 the health is good(green) but still I cannot query the data.

Any quick thoughts/input?

Thanks!

On Sun, Oct 11, 2015 at 9:58 PM, Edward Capriolo <edward.capriolo@huffingtonpost.com> wrote:
Dynamic service discovery is an overkill solution. It should just work. 

On Sun, Oct 11, 2015 at 11:46 AM, Benjamin Kim <bbuild11@gmail.com> wrote:
We are also facing the same problem. We get a crash at least once a week. I am wondering when Cloudera will support Dynamic Service Discovery so we can load balance multiple HiveServer2 instances. We don't have to do our own load balancing setup if it's built-in.



On Sunday, October 11, 2015 at 6:11:00 AM UTC-7, Edward Capriolo wrote:
There is no doubte hs2 is better than hs1. But hs2 is still unreliable. I have run 3 versions of cloudera in the last 2 years. We are using hs2 for hue and for running about 8 queries an hour.

Hs2 typically lasts 2 or 3 days before it fails mid job with no real error message. Its best when it shuts down cause it restarts, but typically it end up alive but doing nothing.

On the other hand the cli never fails.

So we run multiple copies of hsw with load balancer in front and do application retries to deal qith hs2. This is a lot of engineering for somethinf rhat does 8 queries an hour.

If i could start over again i would just have our app create a .q file and launch the query from a cli subshell because that never fails.
On Saturday, October 10, 2015, Vineet Mishra <clearm...@gmail.com> wrote:
Hi Edward,

I am not sure whether you meant HiverServer2 or HS1 as I could see that HS2 is preferable over HS1 


and didn't heard about any such reliability issue of HS2. Moreover I am making multiple JDBC Connection to fire parallel query which is barely consuming cluster resources while the MR job is in state.

I would even like to have understanding if HS2 is unreliable then how could we go ahead to fire parallel queries for some process. Is it like only multiple Hive CLI instance which is possible to make queries, if yes? Then why the other connecting api (i.e Thrift, JDBC, etc) exists. If No? What could be the other possible alternative/solution.

Hi @(Rajat and Manoj),

I have recently increased the Heap Size for HS2, please find the same mentioned below

Client Java Heap Size = 1 GB 
Java Heap Size of WebHCat Server = 512MB

I feel as a developer one's perspective should be to look forward and fix the issues rather than simply relying on restarting the System/Service and get rid of it temporarily.

Any suggestions/input would be highly appreciated.

Thanks!

On Sat, Oct 10, 2015 at 10:22 PM, manoj donga <manojdonga@gmail.com> wrote:

We are facing same issue since long , could found reliable solution is restart all hive server 2 :)

On 10 Oct 2015 10:06 pm, "Rajat Dua" <rajat.dua09@gmail.com> wrote:
Increase heap size of hive aerver2 .might help

On Saturday 10 October 2015, Edward Capriolo <edward.capriolo@huffingtonpost.com> wrote:
HiveServer2 goes down all the time. Do yourself a favor : Go into cloudera manager and hit the checkbox to restart it if it goes down. Everyone is in denial about hiveserver 2 reliability, i run load balancers and run check jobs all day to make sure the thing is serviceable (barely). Its a bad situation that Hive talks about deprecating the CLI in favor of a CLI that works with HiveServer2 yet no one seems to run it for multiple days in a row to see how unrelable it is. Your never going to get anyone on this list to admit to any problems either... 

On Sat, Oct 10, 2015 at 11:42 AM, Vineet Mishra <clearmidoubt@gmail.com> wrote:

Any update on this?

URGENT CALL!

On Oct 10, 2015 9:25 AM, "Vineet Mishra" <clearmidoubt@gmail.com> wrote:
Hi Jimmy,

Any prominent reason for it run out of worker thread when maximum query fired at a time fired are only 8.

I am trying to open multiple session on different connection, so each thread corresponds to one connection.

Thanks!

On Sat, Oct 10, 2015 at 1:22 AM, Jimmy Xiang <jxiang@cloudera.com> wrote:
Could it be because HS2 runs out of worker threads?
Are you trying to open multiple sessions on the same connection?

Thanks,
Jimmy

On Fri, Oct 9, 2015 at 12:34 PM, Vineet Mishra <clearmidoubt@gmail.com> wrote:
Any idea about this?

Frequent Connectivity issue to HiveServer2

  2015-10-10 00:23:11,070 [main] ERROR (HiveConnection.java:439) - Error opening session
  org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160)
at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147)
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:429)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:192)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at com.sd.dwh.sc.tungsten.custom.HiveRunnable.<init>(HiveRunnable.java:42)
at com.sd.dwh.sc.tungsten.custom.HiveInvoker.main(HiveInvoker.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2015-10-10 00:23:11,071 [main] ERROR (HiveRunnable.java:48) - Could not establish connection to jdbc:hive2://hadoop-hs2:10000/mydb: null

URGENT CALL!

Thanks!

On Fri, Oct 9, 2015 at 2:42 PM, Vineet Mishra <clearmidoubt@gmail.com> wrote:
This looks out to be the issue 


Any workaround or resolution to the same.

Thanks!

On Fri, Oct 9, 2015 at 1:24 PM, Vineet Mishra <clearmidoubt@gmail.com> wrote:
Hi All,

I am trying to connect to HiveServer2 to query some data in parallel and landing up into some weird exception, stack trace mentioned below

java.sql.SQLException: Error while cleaning up the server resources
at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:569)
at com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:93)
at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:183)
at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:170)
at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:567)
... 5 more


Any quick revert will be highly appreciable!

Thanks!



--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.