falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Ranganathan <vranganat...@hortonworks.com>
Subject Re: Oozie uber mode and other modes of running the software
Date Thu, 29 Oct 2015 06:03:53 GMT
Thanks Pragya.   Srikanth, good point about the cluster version requirements. 

It is definitely something to keep in mind.   Thanks for the heads up.

Venkat




On 10/26/15, 3:07 AM, "pragya mittal" <mittal.pragya23@gmail.com> wrote:

>Typo :
>Oozie version is 4.2 and RM HA testing is still in progress.
>
>On Sun, Oct 25, 2015 at 9:25 AM, pragya mittal <mittal.pragya23@gmail.com>
>wrote:
>
>> Hi Venkat,
>>
>> From past three months, Inmobi has been using Oozie in uber mode.
>> Currently we are using falcon-0.7 with oozie-3.2 (along with several
>> patches specific to our use case). Following use cases have been tested and
>> are active in our production environment.
>>
>> 1.  NN HA/RM HA
>> 2.  Oozie ubermode for launcher job
>> 3.  Multi cluster with prism
>> 4.  Sanity of falcon features with oozie(in uber mode).
>>
>> After falcon-0.8 release we plan to go forward with Oozie HA feature.
>>
>> Please let me know if you have any more concerns.
>>
>> Regards,
>> Pragya Mittal
>>
>> On Sun, Oct 25, 2015 at 8:15 AM, Srikanth Sundarrajan <sriksun@hotmail.com
>> > wrote:
>>
>>> Am guessing venkat you meant to refer FALCON-1555. Good check list to go
>>> through for each release.
>>>
>>> Specifically on
>>> >> 6. Multi cluster managed with Oozie (with single Falcon instance)
>>>
>>> Since we haven't particularly done anything by isolating the classes and
>>> their dependencies for each cluster with independent class loader, I think
>>> this would work only as long as all the clusters are compatible. If things
>>> were to change say with hadoop-3 for ex, this may no longer work.
>>>
>>> Regards
>>> Srikanth Sundarrajan
>>>
>>> > From: ajay.yadav@inmobi.com
>>> > Date: Sun, 25 Oct 2015 05:29:20 +0530
>>> > Subject: Re: Oozie uber mode and other modes of running the software
>>> > To: dev@falcon.apache.org
>>> >
>>> > Venkat,
>>> >
>>> > Sounds good, let's discuss it in the next sync up. For the benefit of
>>> > everyone on the mailing list, I will summarise our use cases. Yes, we
>>> > (Inmobi) use Oozie's uber mode in our production environment(IIRC we are
>>> > using it since Apache Falcon 0.7). We do have several thousand jobs
>>> running
>>> > per day(several at minutely frequencies) and a heavily utilised
>>> cluster, so
>>> > this makes sense for us. We do test all Oozie releases, though sometimes
>>> > our Oozie releases are not exactly aligned with Oozie's general release
>>> -
>>> > it might contain some extra patches, might be from trunk etc. and our
>>> > testing might be specific to features being used at Inmobi. In fact last
>>> > time Pragya Mittal did the testing for both Apache Falcon 0.7 and Oozie.
>>> > She can add more on this.
>>> >
>>> > List of scenarios looks comprehensive. Just curious, what storage
>>> backends
>>> > are other users using for titan?
>>> >
>>> >
>>> > Cheers
>>> > Ajay Yadava
>>> >
>>> > On Sat, Oct 24, 2015 at 9:20 PM, Venkat Ranganathan <
>>> > vranganathan@hortonworks.com> wrote:
>>> >
>>> > > FALCON-1551 was interesting for two reasons - that we are still
>>> caught by
>>> > > the FS caching issue (this should have been dealt by most of the
>>> projects
>>> > > one time or another), but the use of Oozie uber mode - I am wondering
>>> if we
>>> > > have done uber mode testing generally - Is this the mode that is used
>>> in
>>> > > your environments – It makes sense when thousands of Oozie jobs are
>>> run –
>>> > > saving one container.   I would like to understand more about your
>>> > > experience with uber mode if you have – may be in the next Falcon
>>> sync up.
>>> > >
>>> > >  I think we have create a list of modes that we expect the product
to
>>> run
>>> > > before we introduce changes generally – Given that there are
>>> different sets
>>> > > of execution modes that we want to test.   Please add more on the
>>> scenarios
>>> > > that I am most likely missing.   Let us discuss this also if we have
>>> time.
>>> > >
>>> > >   1.  Security (Kerberos and Wire encryption)
>>> > >   2.  NN HA/RM HA
>>> > >   3.  Oozie HA
>>> > >   4.  Oozie ubermode for launcher job
>>> > >   5.  Multi cluster with prism
>>> > >   6.  Multi cluster managed with Oozie (with single Falcon instance)
>>> > >   7.  Hadoop/Hive/Oozie versions and the implications (S3 and Azure
>>> blob
>>> > > storage went out of hadoop-common in 2.6.0+ and the implications).
>>> > >
>>> > >
>>> > > Thanks
>>> > >
>>> > > Venkat
>>> > >
>>> > >
>>> >
>>> > --
>>> > _____________________________________________________________
>>> > The information contained in this communication is intended solely for
>>> the
>>> > use of the individual or entity to whom it is addressed and others
>>> > authorized to receive it. It may contain confidential or legally
>>> privileged
>>> > information. If you are not the intended recipient you are hereby
>>> notified
>>> > that any disclosure, copying, distribution or taking any action in
>>> reliance
>>> > on the contents of this information is strictly prohibited and may be
>>> > unlawful. If you have received this communication in error, please
>>> notify
>>> > us immediately by responding to this email and then delete it from your
>>> > system. The firm is neither liable for the proper and complete
>>> transmission
>>> > of the information contained in this communication nor for any delay in
>>> its
>>> > receipt.
>>>
>>>
>>
>>
Mime
View raw message