beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3184) Not able to access GCS API when submitting Python jobs behind corporate firewall
Date Wed, 29 Nov 2017 22:54:01 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271706#comment-16271706
] 

ASF GitHub Bot commented on BEAM-3184:
--------------------------------------

charlesccychen commented on a change in pull request #4136: [BEAM-3184] Added ProxyInfoFromEnvironmentVar()
& GetNewHttp() methods for GCS
URL: https://github.com/apache/beam/pull/4136#discussion_r153940122
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/gcsio.py
 ##########
 @@ -87,6 +87,39 @@
 MAX_BATCH_OPERATION_SIZE = 100
 
 
+def proxy_info_from_environment_var(proxy_env_var):
+  """Reads proxy info from the environment and converts to httplib2.ProxyInfo.
+  Args:
+    proxy_env_var: environment variable string to read, http_proxy or
+       https_proxy (in lower case).
+       Example: http://myproxy.domain.com:8080
+  Returns:
+    httplib2.ProxyInfo constructed from the environment string.
+  """
+  proxy_url = os.environ.get(proxy_env_var)
+  if not proxy_url or not proxy_env_var.lower().startswith('http'):
+    return httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP, None, 0)
+    logging.warn("Ignoring proxy_env_var, incorrect format")
+  proxy_protocol = proxy_env_var.lower().split('_')[0]
 
 Review comment:
   <!--new_thread; commit:1033d84efb2ada3802967025d2a64d01f3649c59; resolved:0-->
   Where does the underscore come from?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Not able to access GCS API when submitting Python jobs behind corporate firewall
> --------------------------------------------------------------------------------
>
>                 Key: BEAM-3184
>                 URL: https://issues.apache.org/jira/browse/BEAM-3184
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>            Reporter: David Sabater
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> We should modify gcsio.py module in Python sdk to add methods to pick up proxy settings
from environment variables in httplib2 library. This will allow submitting jobs from behind
a corporate proxy.
> I do have the fix implemented in my forked repository.
> https://github.com/dsdinter/beam/commit/83a54b5b5695783967a175c4623af31997e52b35



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message