Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CANRT7T03GA4z2_pgQ+GS03V3NB63wjT4d-TKNZE4fJrQ8fm4Ug@mail.gmail.com>
References: <CANRT7T03GA4z2_pgQ+GS03V3NB63wjT4d-TKNZE4fJrQ8fm4Ug@mail.gmail.com>
From: Boris Tyukin <boris@boristyukin.com>
Date: Wed, 25 Jan 2017 16:25:59 -0500
Message-ID: <CANRT7T01NYBimfgsu7gEARu7SYhb1q85hb9dqw4zeL-jJ2g9aQ@mail.gmail.com>
Subject: Re: how to capture sqoop mapreduce counters
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a114aa33ad4fe420546f1e22c
archived-at: Wed, 25 Jan 2017 21:26:44 -0000

--001a114aa33ad4fe420546f1e22c
Content-Type: text/plain; charset=UTF-8

I figured that luckily for me, the number of rows loaded by sqoop is
reported to stdout as the very last line. So I just used BashOperator and
set xcom_push=True. Then I did something like that:

    # Log row_count ingested
    try:
        row_count = int(re.search('Retrieved (\d+) records',
                                  kwargs['ti'].xcom_pull(task_
ids='t_sqoop_from_cerner')).group(1))
        write_job_audit(get_job_audit_id_from_context(kwargs),
"rows_ingested_sqoop", row_count)
    except ValueError:
        write_job_audit(get_job_audit_id_from_context(kwargs),
"rows_ingested_sqoop", -1)

The alternative I was considering is to get mapreduce jobid and then use
mapred command to get the needed counter - here is an example:

mapred job -counter job_1484574566480_0002
org.apache.hadoop.mapreduce.TaskCounter
MAP_OUTPUT_RECORDS

But I could not figure out an easy way to get job_id from BashOperator /
sqoop output. I guess I could create my own operator that would capture all
stdout lines not only the last one.

On Tue, Jan 24, 2017 at 9:07 AM, Boris Tyukin <boris@boristyukin.com> wrote:

> Hello all,
>
> is there a way to capture sqoop counters either using bash or sqoop
> operator? Specifically I need to pull a total number of rows loaded.
>
> By looking at bash operator, I think there is an option to push the last
> line of output to xcom but sqoop and mapreduce output is a bit more
> complicated.
>
> Thanks!
>

--001a114aa33ad4fe420546f1e22c--