falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pragya mittal <mittal.pragy...@gmail.com>
Subject Re: Oozie uber mode and other modes of running the software
Date Mon, 26 Oct 2015 10:07:36 GMT
Typo :
Oozie version is 4.2 and RM HA testing is still in progress.

On Sun, Oct 25, 2015 at 9:25 AM, pragya mittal <mittal.pragya23@gmail.com>
wrote:

> Hi Venkat,
>
> From past three months, Inmobi has been using Oozie in uber mode.
> Currently we are using falcon-0.7 with oozie-3.2 (along with several
> patches specific to our use case). Following use cases have been tested and
> are active in our production environment.
>
> 1.  NN HA/RM HA
> 2.  Oozie ubermode for launcher job
> 3.  Multi cluster with prism
> 4.  Sanity of falcon features with oozie(in uber mode).
>
> After falcon-0.8 release we plan to go forward with Oozie HA feature.
>
> Please let me know if you have any more concerns.
>
> Regards,
> Pragya Mittal
>
> On Sun, Oct 25, 2015 at 8:15 AM, Srikanth Sundarrajan <sriksun@hotmail.com
> > wrote:
>
>> Am guessing venkat you meant to refer FALCON-1555. Good check list to go
>> through for each release.
>>
>> Specifically on
>> >> 6. Multi cluster managed with Oozie (with single Falcon instance)
>>
>> Since we haven't particularly done anything by isolating the classes and
>> their dependencies for each cluster with independent class loader, I think
>> this would work only as long as all the clusters are compatible. If things
>> were to change say with hadoop-3 for ex, this may no longer work.
>>
>> Regards
>> Srikanth Sundarrajan
>>
>> > From: ajay.yadav@inmobi.com
>> > Date: Sun, 25 Oct 2015 05:29:20 +0530
>> > Subject: Re: Oozie uber mode and other modes of running the software
>> > To: dev@falcon.apache.org
>> >
>> > Venkat,
>> >
>> > Sounds good, let's discuss it in the next sync up. For the benefit of
>> > everyone on the mailing list, I will summarise our use cases. Yes, we
>> > (Inmobi) use Oozie's uber mode in our production environment(IIRC we are
>> > using it since Apache Falcon 0.7). We do have several thousand jobs
>> running
>> > per day(several at minutely frequencies) and a heavily utilised
>> cluster, so
>> > this makes sense for us. We do test all Oozie releases, though sometimes
>> > our Oozie releases are not exactly aligned with Oozie's general release
>> -
>> > it might contain some extra patches, might be from trunk etc. and our
>> > testing might be specific to features being used at Inmobi. In fact last
>> > time Pragya Mittal did the testing for both Apache Falcon 0.7 and Oozie.
>> > She can add more on this.
>> >
>> > List of scenarios looks comprehensive. Just curious, what storage
>> backends
>> > are other users using for titan?
>> >
>> >
>> > Cheers
>> > Ajay Yadava
>> >
>> > On Sat, Oct 24, 2015 at 9:20 PM, Venkat Ranganathan <
>> > vranganathan@hortonworks.com> wrote:
>> >
>> > > FALCON-1551 was interesting for two reasons - that we are still
>> caught by
>> > > the FS caching issue (this should have been dealt by most of the
>> projects
>> > > one time or another), but the use of Oozie uber mode - I am wondering
>> if we
>> > > have done uber mode testing generally - Is this the mode that is used
>> in
>> > > your environments – It makes sense when thousands of Oozie jobs are
>> run –
>> > > saving one container.   I would like to understand more about your
>> > > experience with uber mode if you have – may be in the next Falcon
>> sync up.
>> > >
>> > >  I think we have create a list of modes that we expect the product to
>> run
>> > > before we introduce changes generally – Given that there are
>> different sets
>> > > of execution modes that we want to test.   Please add more on the
>> scenarios
>> > > that I am most likely missing.   Let us discuss this also if we have
>> time.
>> > >
>> > >   1.  Security (Kerberos and Wire encryption)
>> > >   2.  NN HA/RM HA
>> > >   3.  Oozie HA
>> > >   4.  Oozie ubermode for launcher job
>> > >   5.  Multi cluster with prism
>> > >   6.  Multi cluster managed with Oozie (with single Falcon instance)
>> > >   7.  Hadoop/Hive/Oozie versions and the implications (S3 and Azure
>> blob
>> > > storage went out of hadoop-common in 2.6.0+ and the implications).
>> > >
>> > >
>> > > Thanks
>> > >
>> > > Venkat
>> > >
>> > >
>> >
>> > --
>> > _____________________________________________________________
>> > The information contained in this communication is intended solely for
>> the
>> > use of the individual or entity to whom it is addressed and others
>> > authorized to receive it. It may contain confidential or legally
>> privileged
>> > information. If you are not the intended recipient you are hereby
>> notified
>> > that any disclosure, copying, distribution or taking any action in
>> reliance
>> > on the contents of this information is strictly prohibited and may be
>> > unlawful. If you have received this communication in error, please
>> notify
>> > us immediately by responding to this email and then delete it from your
>> > system. The firm is neither liable for the proper and complete
>> transmission
>> > of the information contained in this communication nor for any delay in
>> its
>> > receipt.
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message