incubator-oozie-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Reddy <dre...@couponsinc.com>
Subject RE: Oozie coordinator xml data input dependency
Date Mon, 03 Oct 2011 04:33:47 GMT
Here is my coordinator xml file (I have set the done-flag to empty but it didn't help)

Also I don't have  _SUCCESS file in my input directories

<coordinator-app name="PROCESSLOGS" frequency="${freq}" start="${startdate}" end="${enddate}"
timezone="${timezone}" xmlns="uri:oozie:coordinator:0.1">
   <datasets>
      <dataset name="input1" frequency="${freq}" initial-instance="${startdate}" timezone="${timezone}">
         <uri-template>hdfs://${namenodeval}/data/user/dreddy/input/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
         <done-flag></done-flag>
      </dataset>
   </datasets>
   <action>
      <workflow>
         <app-path>${workflowpath}</app-path>
         <configuration>
            <property>
              <name>batchdate</name>
              <value>${coord:formatTime(coord:nominalTime(), 'yyyyMMddHHmm')}</value>
            </property>
            <property>
              <name>reducerval</name>
              <value>${redval}</value>
            </property>
            <property>
              <name>namenode</name>
              <value>${namenodeval}</value>
            </property>
         </configuration>
      </workflow>
   </action>
</coordinator-app>

Thanks,
DR

-----Original Message-----
From: Angelo K. Huang [mailto:angeloh@apache.org] 
Sent: Sunday, October 02, 2011 8:31 PM
To: oozie-users@incubator.apache.org
Subject: Re: Oozie coordinator xml data input dependency

Could you post your coordinator.xml so we can easily understand the context?
Do you have _SUCCESS at both directories? Oozie use that for checking dependencies.

http://yahoo.github.com/oozie/releases/3.1.0/CoordinatorFunctionalSpec.html#a5.1._Synchronous_Datasets

You can also set <done-flag></done-flag> so Oozie looks for the existence of the
directory itself.

Thanks,
Angelo

On Sun, Oct 2, 2011 at 8:06 PM, Deepak Reddy <dreddy@couponsinc.com> wrote:

> My job A writes the output as data/user/dreddy/input/yyyyMMddHHmm 
> (5min
> freq)
>
> Each instance of job B (5min freq) should look for corresponding job A 
> directory as above
>
> So in my <datasets> tag I have
> hdfs://${namenodeval}/data/user/dreddy/input/${YEAR}${MONTH}${DAY}${HO
> UR}${MINUTE}
>
> Where initial instance of job A is same as startdate of job B
>
> Also in the <input-events> section I have the start and end instance 
> as ${coord:current(0)} (since job B is dependent on only one dataset 
> of job A)
>
> I created 2 instances of job A as /data/user/dreddy/input/201110010545 
> and
> /data/user/dreddy/input/201110010550
>
> And when I ran the XML I see that the coord jobs are in Waiting state 
> all the time (doesn't launch any workflow jobs)
>
> Am I missing something in my coord XML ?
>
> Thanks,
> DR
>
> -----Original Message-----
> From: Angelo K. Huang [mailto:angelohuang@gmail.com]
> Sent: Sunday, October 02, 2011 5:44 PM
> To: oozie-users@incubator.apache.org
> Subject: Re: Oozie coordinator xml data input dependency
>
> If you want to specify the time stamp at url-template, there are other 
> variable that you can use: For example, ${YEAR}${MONTH}${DAY}${HOUR}.
> The coord:nominalTime
> is not available at url-templates. Could you try use YEAR, MONTH, etc 
> at your coordinator.xml's dataset instead? Let us know if it works for you.
>
>
> http://yahoo.github.com/oozie/releases/3.1.0/CoordinatorFunctionalSpec
> .html#a5.1._Synchronous_Datasets
>
> Thanks,
> Angelo
>
> On Sun, Oct 2, 2011 at 4:37 PM, Deepak Reddy <dreddy@couponsinc.com>
> wrote:
>
> > Hi,
> >
> > In my coordinator.xml I have a data input dependency
> >
> > When I try to use a coord:formatTime EL function in the 
> > <uri-template> and try to run the job it gives me
> >
> > Unable to evaluate
> > :hdfs://${namenodeval}/data/user/input/${coord:formatTime(coord:nomi
> > na
> > lTime(),
> > 'yyyyMMddHHmm')}: ], java.lang.Exception: Unable to evaluate 
> > :hdfs://${namenodeval}:54310/data/user/dreddy/input/${coord:formatTi
> > me
> > (coord:nominalTime(),
> > 'yyyyMMddHHmm')}
> >
> > Cant I use EL functions in the <datasets> part of the XML (the EL 
> > function works for me in the <configuration> part)
> >
> > Removing this EL function works for me but I would like to specify a 
> > batchdate in a particular format (this batchdate is part of the
> > filepath)
> >
> > Thanks,
> > DR
> >
>

Mime
View raw message