falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pragya mittal <mittal.pragy...@gmail.com>
Subject Re: Oozie uber mode and other modes of running the software
Date Sun, 25 Oct 2015 09:25:00 GMT
Hi Venkat,

>From past three months, Inmobi has been using Oozie in uber mode. Currently
we are using falcon-0.7 with oozie-3.2 (along with several patches specific
to our use case). Following use cases have been tested and are active in
our production environment.

1.  NN HA/RM HA
2.  Oozie ubermode for launcher job
3.  Multi cluster with prism
4.  Sanity of falcon features with oozie(in uber mode).

After falcon-0.8 release we plan to go forward with Oozie HA feature.

Please let me know if you have any more concerns.

Regards,
Pragya Mittal

On Sun, Oct 25, 2015 at 8:15 AM, Srikanth Sundarrajan <sriksun@hotmail.com>
wrote:

> Am guessing venkat you meant to refer FALCON-1555. Good check list to go
> through for each release.
>
> Specifically on
> >> 6. Multi cluster managed with Oozie (with single Falcon instance)
>
> Since we haven't particularly done anything by isolating the classes and
> their dependencies for each cluster with independent class loader, I think
> this would work only as long as all the clusters are compatible. If things
> were to change say with hadoop-3 for ex, this may no longer work.
>
> Regards
> Srikanth Sundarrajan
>
> > From: ajay.yadav@inmobi.com
> > Date: Sun, 25 Oct 2015 05:29:20 +0530
> > Subject: Re: Oozie uber mode and other modes of running the software
> > To: dev@falcon.apache.org
> >
> > Venkat,
> >
> > Sounds good, let's discuss it in the next sync up. For the benefit of
> > everyone on the mailing list, I will summarise our use cases. Yes, we
> > (Inmobi) use Oozie's uber mode in our production environment(IIRC we are
> > using it since Apache Falcon 0.7). We do have several thousand jobs
> running
> > per day(several at minutely frequencies) and a heavily utilised cluster,
> so
> > this makes sense for us. We do test all Oozie releases, though sometimes
> > our Oozie releases are not exactly aligned with Oozie's general release -
> > it might contain some extra patches, might be from trunk etc. and our
> > testing might be specific to features being used at Inmobi. In fact last
> > time Pragya Mittal did the testing for both Apache Falcon 0.7 and Oozie.
> > She can add more on this.
> >
> > List of scenarios looks comprehensive. Just curious, what storage
> backends
> > are other users using for titan?
> >
> >
> > Cheers
> > Ajay Yadava
> >
> > On Sat, Oct 24, 2015 at 9:20 PM, Venkat Ranganathan <
> > vranganathan@hortonworks.com> wrote:
> >
> > > FALCON-1551 was interesting for two reasons - that we are still caught
> by
> > > the FS caching issue (this should have been dealt by most of the
> projects
> > > one time or another), but the use of Oozie uber mode - I am wondering
> if we
> > > have done uber mode testing generally - Is this the mode that is used
> in
> > > your environments – It makes sense when thousands of Oozie jobs are
> run –
> > > saving one container.   I would like to understand more about your
> > > experience with uber mode if you have – may be in the next Falcon sync
> up.
> > >
> > >  I think we have create a list of modes that we expect the product to
> run
> > > before we introduce changes generally – Given that there are different
> sets
> > > of execution modes that we want to test.   Please add more on the
> scenarios
> > > that I am most likely missing.   Let us discuss this also if we have
> time.
> > >
> > >   1.  Security (Kerberos and Wire encryption)
> > >   2.  NN HA/RM HA
> > >   3.  Oozie HA
> > >   4.  Oozie ubermode for launcher job
> > >   5.  Multi cluster with prism
> > >   6.  Multi cluster managed with Oozie (with single Falcon instance)
> > >   7.  Hadoop/Hive/Oozie versions and the implications (S3 and Azure
> blob
> > > storage went out of hadoop-common in 2.6.0+ and the implications).
> > >
> > >
> > > Thanks
> > >
> > > Venkat
> > >
> > >
> >
> > --
> > _____________________________________________________________
> > The information contained in this communication is intended solely for
> the
> > use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> transmission
> > of the information contained in this communication nor for any delay in
> its
> > receipt.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message