hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Peña (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HIVE-15102) Hiveptest is killing nodes where IP is reused after previous node termination
Date Tue, 01 Nov 2016 16:07:59 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625821#comment-15625821
] 

Sergio Peña commented on HIVE-15102:
------------------------------------

It could be. This is one of 2 issues found. The other is due to a possible thread leak on
tomcat. However, the thread leak has been there for a while without causing other issues.
I need to dig more on this one, and try to reproduce it.

For more context, we recently moved the ptest server from Amazon to Google Compute. Amazon
was giving us different IPs all the time, but GCE is giving us only IPs on the range of 10.0.0.x;
so only 255 possible values, and reusing those is happening too often.

> Hiveptest is killing nodes where IP is reused after previous node termination
> -----------------------------------------------------------------------------
>
>                 Key: HIVE-15102
>                 URL: https://issues.apache.org/jira/browse/HIVE-15102
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 2.2.0
>            Reporter: Sergio Peña
>            Assignee: Sergio Peña
>         Attachments: HIVE-15102.1.patch
>
>
> NO PRECOMMIT TESTS
> The Hiveptest framework has a background thread that runs every hour, and attempts to
kill zombie nodes that are not being used by the test execution anymore. 
> These killed nodes are kept in a list of terminated nodes, and next time the background
thread is executed, it will attempt to kill all those nodes again because Hiveptest consider
them as zombie nodes.
> The problem is that cloud providers can give you the same IP numbers for new nodes, and
when the background thread runs, it will kill those nodes that may still be in used by Hiveptest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message