airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robin Bartholdson <robin.barthold...@formulate.se>
Subject Re: Missing region in S3Hook when using extra
Date Thu, 23 Mar 2017 17:05:07 GMT
I submitted our patch in the mean time, do test it and see if it works for you :-)

https://github.com/apache/incubator-airflow/pull/2181
https://issues.apache.org/jira/browse/AIRFLOW-1034

-Robin

> On 23 Mar 2017, at 17:43, Robin Bartholdson <robin.bartholdson@formulate.se> wrote:
> 
> There is some discussion in this JIRA, https://issues.apache.org/jira/browse/AIRFLOW-115
<https://issues.apache.org/jira/browse/AIRFLOW-115>, about moving to boto3 and restructuring
along the lines you discuss. 
> 
> I totally agree that it would indeed be very useful!
> 
> -Robin
> 
> 
>> On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <jalpedrinharamos@gmail.com <mailto:jalpedrinharamos@gmail.com>>
wrote:
>> 
>> Hi Sumit,
>> 
>> I was actually referring to what Robin (thank you) pointed out which is
>> reason enough for having region in S3Hook.
>> 
>> There are even other situations having the region would be beneficial,
>> that's why the AwsHook has that definition, although I would agree that
>> their goals are different they typically refer to the same credentials,
>> just different use cases.
>> 
>> This makes me feel that it becomes redundant to have both an S3 connection
>> and an AWS connection with the same credentials just be able to access
>> region. Wouldn't it be beneficial to implement S3Hook (and S3 connection)
>> as a special case of AwsHook (and Aws connection) and be able to use both
>> interchangeably ?
>> 
>> I'm also aware that one uses boto and the other uses boto3, and it seems
>> like the project is naturally slowly migrating towards the later. Maybe
>> there is even some discussion regarding this that I'm not aware.
>> 
>> If you think this is a desirable move, I may start to design this move,
>> making sure nothing gets broken in the process.
>> 
>> WDYT?
>> 
>> On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson <
>> robin.bartholdson@formulate.se <mailto:robin.bartholdson@formulate.se>>
wrote:
>> 
>>> I _think_ that’s a different issue (do correct me if I’m wrong)
>>> 
>>> For a number of regions (see the list on
>>> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html <http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html>),
>>> you have to specify an extra host-parameter in the boto-S3Connection as
>>> they only support signature V4 and the host-param is required to have this
>>> working (
>>> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError
<http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError>
>>> )
>>> 
>>> -Robin
>>> 
>>> On 23 Mar 2017, at 15:10, Sumit Maheshwari <sumeet.manit@gmail.com> wrote:
>>> 
>>> Are we talking about this
>>> https://github.com/apache/incubator-airflow/pull/2176 ?
>>> 
>>> 
>>> 
>>> 
>>> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
>>> robin.bartholdson@formulate.se> wrote:
>>> 
>>> Hi,
>>> 
>>> We actually have a patch for this that allows you to specify the host
>>> parameter in the S3 config in the connections.
>>> 
>>> Been thinking about submitting this for a while but have not had the time,
>>> maybe now’s the time to do it :-)
>>> 
>>> If you want to open a JIRA I can submit the patch we are using in
>>> production!
>>> 
>>> -Robin
>>> 
>>> 
>>> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
>>> jalpedrinharamos@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> I've been using S3Hook for a while, but I've run into some issues for not
>>> having a defined region. I can't really say why I can't access this bucket
>>> without defining the region, but I know that there is no way to define
>>> region in S3Hook via the connection extra parameter unless a config_file is
>>> used, which is not always an available method.
>>> 
>>> I haven't found any ticket regarding this, is there any alternative ?
>>> 
>>> Regards,
>>> Jorge
>>> 
>>> 
>>> 
>>> 
>>> 
> 


Mime
View raw message