ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tuong Truong <ira...@yahoo.com>
Subject Re: How to install and start apache distributed hadoop rather than hortonworks distribution
Date Thu, 13 Oct 2016 16:50:53 GMT
Hi Aman,
Unless you plan to do your own Hadoop distro (which will be quite an endeavor on building
your own stack + the RPMs),  I don't recommend trying this just to use your own Hadoop version.  
The RPMs are build specifically for a distrothe and stack version.   So if you want use
the HDP 2.3 version and update just Hadoop, for example, you will need to build your RPMs
for that exact version; RPM is installed into a very specific location for each version. 
The RPMs also has pre/post scripts that does extra setup required by the version of Ambari.  
There is a tight coupling between Ambari and Bigtop needed to produce a particular distro
release such as HDP 2.3.   Unless you  have the Bigtop RPM spec files for that release,
you will need to do some reverse engineer to get it right, and this is not trivial.   Then
there are potential issues with inter-operability between the service version in the stack
if you inject your own version.

You may be better off looking at just replacing Hadoop JARs if your aim is just to test out
a new Hadoop release.  
 Respectfully,
Tuong 

   

 On Wednesday, October 12, 2016 11:55 PM, aman poonia <aman.poonia.29@gmail.com> wrote:
 

 Hey Alejandro,
Thank you very much for pointing me to the right source code. I will see what can i figure
out of this. :-)
-- 
With Regards,
Aman Poonia

On Tue, Oct 11, 2016 at 12:28 AM, Alejandro Fernandez <afernandez@hortonworks.com> wrote:

I think that requirement is based on the fact that Ambari needs to be able to compare version
numbers.Typically, each service's metainfo.xml file defines how to performs a yum install
of its packages, which can replace variables like the stack_version, and insert *
E.g.,<osSpecifics>
  <osSpecific>
    <osFamily>any</osFamily>
    <packages>
      <package>
        <name>hbase</name>
      </package>
    </packages>
  </osSpecific>
</osSpecifics>Or
<osSpecific>
  <osFamily>redhat7,amazon2015, redhat6,suse11,suse12</osFamil y>
  <packages>
    <package>
      <name>atlas-metadata_${stack_ version}</name>
    </package>
    <package>
      <name>ambari-infra-solr-client </name>
      <condition>should_install_ infra_solr_client</condition>
    </package>
    <package>
      <name>kafka_${stack_version}</ name>
    </package>
  </packages>
</osSpecific>
However, you may have to change several other python functions if you want package names that
don't conform to that standard, or at least look at what these do
ambari-common/src/main/python/ resource_management/libraries/ functions/conf_select.pyambari-common/src/main/python/
resource_management/libraries/ functions/stack_select.pyambari-common/src/main/python/ resource_management/libraries/
functions/version.pyambari-server/src/main/ resources/custom_actions/ scripts/install_packages.py
Thanks,Alejandro
From: aman poonia <aman.poonia.29@gmail.com>
Date: Saturday, October 8, 2016 at 2:28 AM
To: Alejandro Fernandez <afernandez@hortonworks.com>
Subject: Re: How to install and start apache distributed hadoop rather than hortonworks distribution

Hi Alejandro,
I downloaded Bigtop and created the zookeeper and Hadoop rpm from apache provided tarballs.
And now i am trying to use these rpm instead of hortonworks to deploy a hadoop cluster. And
i am facing difficulty in this. As ambari searches for a specific names like"yum install hadoop_x_x_x_x-xxxx""yum
install hadoop_x_x_x_x-xxxx-hdfs"
and so on. 
How can i make it work with my own generated RPMs.

-- 
With Regards:-
Aman Poonia

On Fri, Oct 7, 2016 at 11:39 PM, Alejandro Fernandez <afernandez@hortonworks.com> wrote:

Hi Aman,
Making your own distribution is no easy task. You can literally spend months trying to do
this since it requires 
tooling (like the equivalent of conf-select and hdp-select to change symlinks)packaging of
Hadoop into RPMs (or equivalent for other Oses)finding compatible versions of each productproviding
default configs based on those versionsyour own stack advisorhandling configs during stack
upgrade (rolling/express)etc.
What exactly are you trying to accomplish?
Thanks,Alejandro
From: aman poonia <aman.poonia.29@gmail.com>
Date: Friday, October 7, 2016 at 4:53 AM
To: Alejandro Fernandez <afernandez@hortonworks.com>
Cc: "user@ambari.apache.org" <user@ambari.apache.org>
Subject: Re: How to install and start apache distributed hadoop rather than hortonworks distribution

So essentially if i want to use apache distribution i need to define my own stack? Can't i
just change some configuration so that it starts working with apache distribution. 
What i understood from documentation and code is to write a stack one needs to provide his
own replacement of "hdp-select" and "conf-select" and couldnot find documentation around
what is expected from these tools(like what all functions one need to implement) so it looks
like a dark area to me.
A did a quick grep to see if there is something around version number of stack and found this
in ambari-commons
ambari-common/src/main/python/ resource_management/libraries/ functions/stack_select.py:  
 match = re.match('[0-9]+.[0-9]+.[0-9]+ .[0-9]+-[0-9]+', stack_version)ambari-common/src/main/python/
resource_management/libraries/ functions/get_stack_version.py :  match = re.findall('[0-9]+.[0-9]+.[0-9
]+.[0-9]+-[0-9]+', home_dir_split[iSubdir])ambari-common/src/main/python/ resource_management/libraries/
functions/get_stack_version.py :  match = re.match('[0-9]+.[0-9]+.[0-9]+ .[0-9]+-[0-9]+',
stack_version)
Looks like there is some rule around the naming of rpm packages and stack naming which i am
completely missing!!


-- 
With Regards:-
Aman Poonia

On Wed, Oct 5, 2016 at 11:11 PM, Alejandro Fernandez <afernandez@hortonworks.com> wrote:

Hi Aman,
Ambari is meant to work with any distribution, as long as it has a stack definition, which
includes list of services, RPM names, etc. For example, https://github.com/ap ache/ambari/tree/trunk/ambari-
server/src/main/resources/stac ksAre you trying to build your own stack?
Thanks,Alejandro
From: aman poonia <aman.poonia.29@gmail.com>
Reply-To: "user@ambari.apache.org" <user@ambari.apache.org>
Date: Wednesday, October 5, 2016 at 3:10 AM
To: "user@ambari.apache.org" <user@ambari.apache.org>
Subject: How to install and start apache distributed hadoop rather than hortonworks distribution

I am new to Ambari and have been trying setting up a cluster. Amabri looks interesting to
use. 
However, i am having a tough time to understand how to install and start apache distributed
Hadoop rather than Hortonworks distributed Hadoop using Ambari. Is there a documentation i
can refer to.There are instances when i don't want to use Hortonworks distribution and want
to use apache distributed Hadoop.Also need some help in understanding naming convention of
rpm packages that Ambari expects. Have i missed something in the documentation? 

-- 
With Regards,
Aman Poonia








   
Mime
View raw message