beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3184) Not able to access GCS API when submitting Python jobs behind corporate firewall
Date Wed, 29 Nov 2017 09:53:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270506#comment-16270506
] 

ASF GitHub Bot commented on BEAM-3184:
--------------------------------------

dsdinter commented on a change in pull request #4136: [BEAM-3184] Added ProxyInfoFromEnvironmentVar()
& GetNewHttp() methods for GCS
URL: https://github.com/apache/beam/pull/4136#discussion_r153737919
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/gcsio.py
 ##########
 @@ -87,6 +87,50 @@
 MAX_BATCH_OPERATION_SIZE = 100
 
 
+def ProxyInfoFromEnvironmentVar(proxy_env_var):
+  """Reads proxy info from the environment and converts to httplib2.ProxyInfo.
+  Args:
+    proxy_env_var: Environment variable string to read, such as http_proxy or
+       https_proxy.
+  Returns:
+    httplib2.ProxyInfo constructed from the environment string.
+  """
+  proxy_url = os.environ.get(proxy_env_var)
+  if not proxy_url or not proxy_env_var.lower().startswith('http'):
+    return httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP, None, 0)
+  proxy_protocol = proxy_env_var.lower().split('_')[0]
+  if not proxy_url.lower().startswith('http'):
+    # proxy_info_from_url requires a protocol, which is always http or https.
 
 Review comment:
   Logging a warning as it will attempt to append protocol from environment variable name.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Not able to access GCS API when submitting Python jobs behind corporate firewall
> --------------------------------------------------------------------------------
>
>                 Key: BEAM-3184
>                 URL: https://issues.apache.org/jira/browse/BEAM-3184
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>            Reporter: David Sabater
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> We should modify gcsio.py module in Python sdk to add methods to pick up proxy settings
from environment variables in httplib2 library. This will allow submitting jobs from behind
a corporate proxy.
> I do have the fix implemented in my forked repository.
> https://github.com/dsdinter/beam/commit/83a54b5b5695783967a175c4623af31997e52b35



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message