airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hanneli Tavante <hannel...@gmail.com>
Subject Re: Question - HttpHook
Date Thu, 19 Jan 2017 00:48:08 GMT
Thank you very much for the reply!
I set these parameters using the web interface. I tried several combos for
an HTTP type:
- setting host, login, password only
- setting schema field with https + setting host, login, password only
- setting port 443 + setting host, login, password only
- setting port 443 + schema field https + setting host, login, password
only

for a certain my_dag.py, I run a PythonOperator calling a method that does
the following:
``` api_hook =
HttpHook(http_conn_id='databricks_dcs_convert_json_to_parquet_raw_workflow_job_id',
method='GET')

response = api_hook.run('/api/2.0/jobs/list')
```

I also tried a complete version of run method:
```
response = api_hook.run('/api/2.0/jobs/list', None, 'application/json',
None)

```
The problem is the same - I end up on the Login page.

(I know my credentials are fine since I can perform a plain curl)

It is odd since I succeed for other API integrations.

Thank you very much for the help!


On Wed, Jan 18, 2017 at 4:38 PM Matt Hickman <matt.hickman@gmail.com> wrote:

Hanneli,

We hit the data bricks api from Airflow. The connection info for our data
bricks instance is pretty simple:

conn type: http
host: https://<cloud_id>.cloud.databricks.com/
login: <your username>
password: <your password>

Knowing what error you get would be helpful! Also if you provide the code
your using for your hook.

Cheers,
Matt

On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannelita@gmail.com)
wrote:

Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message