mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Re: HAProxy for Hive Thrift Server on Mesos
Date Mon, 23 Mar 2015 15:26:44 GMT
So per my HA-Proxy logs, I am getting a condition where the client
side time expired while waiting for the client to send or receive
data, and this is occurring during the DATA phase. (the two letter
HAProxy code is cD . To test, I expanded the timeout (on all things)
to 100000.

This was interesting in that it showed me the problem a bit clearer.
Now, as long as I submit and results for a query in the 100 second
range, my connection stays open and happy.  If I let it sit for 100
seconds, and then try a query, then it breaks again.

It appears HA proxy gets irritable when things aren't being sent over
a connection for a long period of time.  For services like Minecraft,
this is not an issue, when a client is connected, there is always data
there, and no issues occur. For Thrift, there may be multiple long
pauses as an analyst thinks through the next query etc.  I wonder what
would be the best way to handle that on a per container level. (not
all containers need this sort of customization...)



On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xiaokunhou@gmail.com> wrote:
> I have a similar error when launch jenkins in mesos cluster. And I use
> HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time
> to time, connection between jenkins master and jenkins slave will lost,
> while will cause jobs fail.
>
> I will try to set timeout value for HAProxy
>
> 2015-03-23 4:18 GMT+08:00 John Omernik <john@omernik.com>:
>>
>> Yep all three are going through HAProxy and seem to be working well.
>> I should do some testing to determine if I run a thrift server that
>> connects to metastore in HAProxy to determine if that's the broken
>> link, but I believe it's between the thrift server. I will check out
>> the link you posted to get a better idea out of how it works. I am
>> running this all on a MapR cluster which is Great for Docker/Mesos as
>> I used the MapR FS, mounted locally on each node's NFS to put the data
>> for MySQL, the logs for the metastore/thift server, and my world files
>> for Minecraft.  So far it's all really slick.
>>
>> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
>> <ben.whitehead@mesosphere.io> wrote:
>> > It sounds like you may be running into the connection timeout that is
>> > configured for the HAProxy instance.
>> >
>> > You can see an instance of this in Marathon's
>> > HAProxy-marathon-bridge[1].
>> > You may need to raise the timeout values.
>> >
>> > Are you accessing MySql, Hive metastore and minecraft through HAProxy as
>> > well? I know that Minecraft running in docker on mesos works well,
>> > though
>> > I've always used a direct connection not a proxied one.
>> >
>> >
>> > [1]
>> >
>> > https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
>> >
>> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <john@omernik.com> wrote:
>> >>
>> >> I have a nice setup with a Hive thrift server running in a docker
>> >> container on Mesos. It works pretty well, but something, I believe in
>> >> how HAProxy works with the connection, is causing the thrift server
>> >> connection  to die after a time.  Basically, I can run a few queries,
>> >> but after 2 or 3, or specifically after a longer query, I get the
>> >> error below indicating End of File on the connection.  Then no more
>> >> connections work until I reestablish the connection to the thrift
>> >> server. I've tried looking in logs. the thrift server std err logs
>> >> show no issues. I am guess I need to dig into haproxy logs, but I am
>> >> not seeing any issues in syslog so far.  I'd love any pointers on how
>> >> to trouble shoot this.  By the way, I have MySQl, hive metastore, and
>> >> a minecraft server all running on Mesos/Docker with no issues, not
>> >> sure why the thrift server is so sensitive.
>> >>
>> >> :)
>> >>
>> >>
>> >> pyhs2 connection string:
>> >>
>> >> hs2 =
>> >>
>> >> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
>> >>
>> >>
>> >> Error
>> >>
>> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in
>> >> read(self, sz)
>> >>     118     if len(buff) == 0:
>> >>     119       raise
>> >> TTransportException(type=TTransportException.END_OF_FILE,
>> >> --> 120                                 message='TSocket read 0 bytes')
>> >>     121     return buff
>> >>     122
>> >>
>> >> TTransportException: TSocket read 0 bytes
>> >
>> >
>
>

Mime
View raw message