gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Timea Magyar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GEARPUMP-356) Launch of Gearpump on Hadoop/Yarn not possible if cluster protected by firewall
Date Wed, 11 Oct 2017 10:13:02 GMT

     [ https://issues.apache.org/jira/browse/GEARPUMP-356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Timea Magyar updated GEARPUMP-356:
----------------------------------
    Description: 
Launching Gearpump on a Hadoop/Yarn cluster protected by a firewall from an edge node outside
of the cluster communication will fail.
When launching Gearpump / submitting a Gearpump application the client side residing on the
above mentioned edge node has to establish communication with 4 components:

1. the YarnAppMaster in order to be able to retrieve/resolve an active configuration file
(via tcp)
2. the Gearpump master(s) (via tcp)
3. the JarStoreServer (via http) to transmit the application jar file to be submitted
4. the Gearpump AppMaster of the respective gearpump application to be submitted to send in
tasks/requests (via tcp)

In all four cases the ports used by the above mentioned components have to be open, otherwise
communication will fail. 
Currently the config parameter "akka.remote.netty.tcp.port" is set to 0, meaning that all
ports assigned for an actorsystem defined in Gearpump are random and can be anything between
 1024-65535
There is no configuration based way to set the a fixed port in the above four cases.



  was:
Launching Gearpump on a Hadoop/Yarn cluster protected by a firewall from an edge node outside
of the cluster communication will fail.
When launching Gearpump / submitting a Gearpump application the client side residing on the
above mentioned edge node has to establish communication with 4 components:

1. the YarnAppMaster in order to be able to retrieve/resolve an active configuration file
(via tcp)
2. the Gearpump master(s) (via tcp)
3. the JarStoreServer (via http) to transmit the application jar file to be submitted
4. the Gearpump AppMaster of the respective gearpump application to be submitted to send in
tasks/requests

In all four cases the ports used by the above mentioned components have to be open, otherwise
communication will fail. 
Currently the config parameter "akka.remote.netty.tcp.port" is set to 0, meaning that all
ports assigned for an actorsystem defined in Gearpump are random and can be anything between
 1024-65535
There is no configuration based way to set the a fixed port in the above four cases.




> Launch of Gearpump on Hadoop/Yarn not possible if cluster protected by firewall
> -------------------------------------------------------------------------------
>
>                 Key: GEARPUMP-356
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-356
>             Project: Apache Gearpump
>          Issue Type: Bug
>          Components: hadoop, security, yarn
>    Affects Versions: 0.8.4
>            Reporter: Timea Magyar
>             Fix For: 0.8.4
>
>
> Launching Gearpump on a Hadoop/Yarn cluster protected by a firewall from an edge node
outside of the cluster communication will fail.
> When launching Gearpump / submitting a Gearpump application the client side residing
on the above mentioned edge node has to establish communication with 4 components:
> 1. the YarnAppMaster in order to be able to retrieve/resolve an active configuration
file (via tcp)
> 2. the Gearpump master(s) (via tcp)
> 3. the JarStoreServer (via http) to transmit the application jar file to be submitted
> 4. the Gearpump AppMaster of the respective gearpump application to be submitted to send
in tasks/requests (via tcp)
> In all four cases the ports used by the above mentioned components have to be open, otherwise
communication will fail. 
> Currently the config parameter "akka.remote.netty.tcp.port" is set to 0, meaning that
all ports assigned for an actorsystem defined in Gearpump are random and can be anything between
 1024-65535
> There is no configuration based way to set the a fixed port in the above four cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message