airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MSR M <msrmaill...@gmail.com>
Subject Re: BashOperator strage behavior
Date Thu, 23 Jun 2016 19:22:42 GMT
Hi,

Please ignore this.

This is happening because remote EC2 has problem with some symbolic links
pointing python.

I have to give the full path of python in Bash command.

Thanks,
Madhu

On Thu, Jun 23, 2016 at 8:15 AM, Msr Msr <msrmaillist@gmail.com> wrote:

> Hi,
>
> I installed airflow<python3> on my laptop and using BashOperator to login
> to remote EC2 server and submit a python program. On my Ec2 machine I have
> both Python2 <2.7.6> and Python3 <3.5.1>.
>
> If I login manually to Ec2 instance and type just  "python" it opens
> Python3 shell.
>
> But when I execute through BashOperator command using airflow it is going
> to Python2.
>
> Please see below command and output. The first command "python --version"
> output is "{bash_operator.py:77} INFO - Python 2.7.6". for Python3
> --version it gives "Python 3.4.3 " But I have 3.5.1
>
> This is causing a bigger problem because I need to execute a Jupiter
> notebook that has to process utf-8 chars. my notebook is in python3.
> Python2 processing of same notebook gives utf-8 char related error and
> fixing that is not possible from my side.
>
> When I tried to execute the python program from BashOperator using
> something like "python3 abc.py" it still uses python2 some where internally
> so utf-8 char related error comes.
>
> Please suggest me or point me out what I am missing. I have been trying
> different options but didn't get any progress. Any help would be highly
> appreciated.
>
> Please see below log:
>
> sparkcmd = """
>            ssh -i /users/msr/Downloads/EC2/abc.pem
> ubuntu@ec2-xx-yyy-zz-aaa.us-west-1.compute.amazonaws.com 'python
> --version; python3 --version;'
> """
>
> t2 = BashOperator(
>     task_id='ClassifyNews',
>     bash_command=sparkcmd,
>     retries=3,
>     dag=dag)
>
> [2016-06-23 07:50:25,522] {models.py:1219} INFO - Executing
> <Task(BashOperator): ClassifyNews> on 2016-05-04 00:00:00
> [2016-06-23 07:50:25,535] {bash_operator.py:55} INFO - tmp dir root
> location:
> /var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T
> [2016-06-23 07:50:25,536] {bash_operator.py:64} INFO - Temporary script
> location
> :/var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s//var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s/ClassifyNewsgwx9mtr5
> [2016-06-23 07:50:25,536] {bash_operator.py:65} INFO - Running command:
>            ssh -i /users/msr/Downloads/EC2/abc.pem
> ubuntu@ec2-xx-yyy-zz-aaa.us-west-1.compute.amazonaws.com 'python --version;
>
>            python3 --version;'
>
> [2016-06-23 07:50:25,539] {bash_operator.py:73} INFO - Output:
> [2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 2.7.6
> [2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 3.4.3
> [2016-06-23 07:50:26,491] {bash_operator.py:80} INFO - Command exited with
> return code 0
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message