mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xiaokun <xiaokun...@gmail.com>
Subject Re: HAProxy for Hive Thrift Server on Mesos
Date Wed, 25 Mar 2015 05:00:51 GMT
After set the timeout value, the tcp request become stable.

2015-03-23 23:26 GMT+08:00 John Omernik <john@omernik.com>:

> So per my HA-Proxy logs, I am getting a condition where the client
> side time expired while waiting for the client to send or receive
> data, and this is occurring during the DATA phase. (the two letter
> HAProxy code is cD . To test, I expanded the timeout (on all things)
> to 100000.
>
> This was interesting in that it showed me the problem a bit clearer.
> Now, as long as I submit and results for a query in the 100 second
> range, my connection stays open and happy.  If I let it sit for 100
> seconds, and then try a query, then it breaks again.
>
> It appears HA proxy gets irritable when things aren't being sent over
> a connection for a long period of time.  For services like Minecraft,
> this is not an issue, when a client is connected, there is always data
> there, and no issues occur. For Thrift, there may be multiple long
> pauses as an analyst thinks through the next query etc.  I wonder what
> would be the best way to handle that on a per container level. (not
> all containers need this sort of customization...)
>
>
>
> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xiaokunhou@gmail.com> wrote:
> > I have a similar error when launch jenkins in mesos cluster. And I use
> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from
> time
> > to time, connection between jenkins master and jenkins slave will lost,
> > while will cause jobs fail.
> >
> > I will try to set timeout value for HAProxy
> >
> > 2015-03-23 4:18 GMT+08:00 John Omernik <john@omernik.com>:
> >>
> >> Yep all three are going through HAProxy and seem to be working well.
> >> I should do some testing to determine if I run a thrift server that
> >> connects to metastore in HAProxy to determine if that's the broken
> >> link, but I believe it's between the thrift server. I will check out
> >> the link you posted to get a better idea out of how it works. I am
> >> running this all on a MapR cluster which is Great for Docker/Mesos as
> >> I used the MapR FS, mounted locally on each node's NFS to put the data
> >> for MySQL, the logs for the metastore/thift server, and my world files
> >> for Minecraft.  So far it's all really slick.
> >>
> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
> >> <ben.whitehead@mesosphere.io> wrote:
> >> > It sounds like you may be running into the connection timeout that is
> >> > configured for the HAProxy instance.
> >> >
> >> > You can see an instance of this in Marathon's
> >> > HAProxy-marathon-bridge[1].
> >> > You may need to raise the timeout values.
> >> >
> >> > Are you accessing MySql, Hive metastore and minecraft through HAProxy
> as
> >> > well? I know that Minecraft running in docker on mesos works well,
> >> > though
> >> > I've always used a direct connection not a proxied one.
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
> >> >
> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <john@omernik.com>
> wrote:
> >> >>
> >> >> I have a nice setup with a Hive thrift server running in a docker
> >> >> container on Mesos. It works pretty well, but something, I believe
in
> >> >> how HAProxy works with the connection, is causing the thrift server
> >> >> connection  to die after a time.  Basically, I can run a few queries,
> >> >> but after 2 or 3, or specifically after a longer query, I get the
> >> >> error below indicating End of File on the connection.  Then no more
> >> >> connections work until I reestablish the connection to the thrift
> >> >> server. I've tried looking in logs. the thrift server std err logs
> >> >> show no issues. I am guess I need to dig into haproxy logs, but I am
> >> >> not seeing any issues in syslog so far.  I'd love any pointers on how
> >> >> to trouble shoot this.  By the way, I have MySQl, hive metastore, and
> >> >> a minecraft server all running on Mesos/Docker with no issues, not
> >> >> sure why the thrift server is so sensitive.
> >> >>
> >> >> :)
> >> >>
> >> >>
> >> >> pyhs2 connection string:
> >> >>
> >> >> hs2 =
> >> >>
> >> >>
> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
> >> >>
> >> >>
> >> >> Error
> >> >>
> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc
> in
> >> >> read(self, sz)
> >> >>     118     if len(buff) == 0:
> >> >>     119       raise
> >> >> TTransportException(type=TTransportException.END_OF_FILE,
> >> >> --> 120                                 message='TSocket read 0
> bytes')
> >> >>     121     return buff
> >> >>     122
> >> >>
> >> >> TTransportException: TSocket read 0 bytes
> >> >
> >> >
> >
> >
>

Mime
View raw message