hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6255) Create an rpm target in the build.xml
Date Fri, 25 Sep 2009 15:58:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759593#action_12759593

Steve Loughran commented on HADOOP-6255:

# This should be a separate project from the others, it's integration, and will soon get big.
# The project would be linux and OS/X (with rpmbuild installed) only. Even on linux, the right
tools need to be installed
# Basic RPMs are easy, passing rpmlint harder
# Testing, that's the fun part. 

We test our RPMs by 
# SCP to configured real/virtual machines. These are Centos 5.x VMs, usually hosted under
VMWare. Under VirtualBox, RHEL5 and Centos spins one CPU at 100% (Virtualbox bug #1233)
# force-uninstalling any old versions, install the new ones.
# SSH in, walk the shell scripts through their entry points
  <target name="rpm-remote-initd"
      description="check that initd parses">
    <rootssh command="${remote-smartfrogd} start"/>
    <rootssh command="${remote-smartfrogd} status"/>
    <rootssh command="${remote-smartfrogd} start"/>
    <rootssh command="${remote-smartfrogd} status"/>
    <rootssh command="${remote-smartfrogd} stop"/>
    <rootssh command="${remote-smartfrogd} stop"/>
    <rootssh command="${remote-smartfrogd} restart"/>
    <rootssh command="${remote-smartfrogd} status"/>
    <rootssh command="${remote-smartfrogd} restart"/>
    <rootssh command="${remote-smartfrogd} status"/>
    <rootssh command="${remote-smartfrogd} stop"/>
# run rpm -qf against various files, verify that they are owned. The RPM commands, executed
remotely over SSH, are no fun to use in tests as you have to look for certain strings in the
response; error codes are not used to signal failures. Ouch.
          <contains string="${rpm.queries.results}"
              substring="is not owned by any package"/>
          <contains string="${rpm.queries.results}"
              substring="No such file or directory"/>
      One of the directories/files in the RPM is not declared as being owned by any RPM.
      This file/directory will not be managed correctly, or have the correct permissions
      on a hardened linux.

For full functional testing, we also package up the test source trees as JAR files which are
published via Ivy, so that the release/ project can retrieve those test files and point them
(by way of java properties) at the remote machine. This is powerful as you can be sure that
 the RPM installations really do work as intended. If you only test the local machine, you
miss out on problems. 

These tests don't verify all possible upgrades. They can be trouble as RPM installs the new
files before uninstalling the old ones. Trouble.

The other issue is configuration. You can either mark all configuration files as {{%config(noreplace)}},
meaning people can edit them and upgrades won't stamp on them, or have a more structured process
for managing conf files. Cloudera provide a web site to create a new configuration RPM, Apache
could be provide a .tar.gz file which contains everything needed to create your own configuration

Therefore + 1 to RPMs and debs 
# In a separate package
# Named Apache-Hadoop. People out there are already releasing hadoop RPMs, we don't want confusion.
# With all config files in the RPM marked as {{%config)}} files, which end users can stamp
on, or a separate roll-your-own-config RPM tool
# Once the tests are designed to run against remote systems, they should be run against the
RPM installations.

I don't volunteer to write the spec files or the build files, all mine are up to look at,
and I will formally release them as Apache licensed if you want to use them as a starting
I could help with some of the functional testing now, provided it uses some of my real/virtual
cluster management stuff to pick target hosts.

> Create an rpm target in the build.xml
> -------------------------------------
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
> We should be able to create RPMs for Hadoop releases.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message