incubator-bigtop-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r..@apache.org
Subject svn commit: r1326652 - in /incubator/bigtop/branches/branch-0.3: ./ bigtop-deploy/puppet/modules/hadoop-oozie/manifests/ bigtop-deploy/puppet/modules/hadoop-oozie/templates/ bigtop-deploy/puppet/modules/hadoop/templates/ bigtop-packages/src/common/flum...
Date Mon, 16 Apr 2012 15:36:08 GMT
Author: rvs
Date: Mon Apr 16 15:36:06 2012
New Revision: 1326652

URL: http://svn.apache.org/viewvc?rev=1326652&view=rev
Log:
Merging trunk (0.3.0-based work) into branch-0.3

Added:
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/pom.xml
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopExport.groovy
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopImport.groovy
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-db.sql
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-tables.sql
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-insert-data.sql
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-all-tables.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-append.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-columns.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-non-string.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-string.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-query.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool-export.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date-export.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_fp.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_int.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_string.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-testtable.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-where-clause.out
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/sqoop-smokes/
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/sqoop-smokes/pom.xml
      - copied, changed from r1302958, incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml
Modified:
    incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp
    incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml
    incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/flume-node.init
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/install_flume.sh
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/hive/install_hive.sh
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/pig/install_pig.sh
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/whirr/do-component-build
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/control
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/install_init_scripts.sh
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/oozie/oozie.preinst
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/flume/SPECS/flume.spec
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec
    incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec
    incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/AptCmdLinePackageManager.groovy
    incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/PackageTestRepoMgr.groovy
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/TestPackagesBasics.groovy
    incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml
    incubator/bigtop/branches/branch-0.3/bigtop.mk
    incubator/bigtop/branches/branch-0.3/package.mk

Modified: incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp Mon Apr 16 15:36:06 2012
@@ -31,7 +31,7 @@ class hadoop-oozie {
       ensure => latest,
     }
 
-    file { "/etc/oozie/oozie-site.xml":
+    file { "/etc/oozie/conf/oozie-site.xml":
       content => template("hadoop-oozie/oozie-site.xml"),
       require => Package["oozie"],
     }

Modified: incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml Mon Apr 16 15:36:06 2012
@@ -230,7 +230,7 @@
 
     <property>
         <name>oozie.authentication.type</name>
-        <value>simple</value>
+        <value><%= kerberos_realm.empty? ? "simple" : "kerberos" %></value>
         <description>
             Defines authentication used for Oozie HTTP endpoint.
             Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#

Modified: incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml Mon Apr 16 15:36:06 2012
@@ -44,7 +44,7 @@
 <% end %>
 
 <% if has_variable?("hadoop_core_proxyusers") %>
-<% hadoop_core_proxyusers.each do |superuser, data| %>
+<% hadoop_core_proxyusers.sort.each do |superuser, data| %>
   <property>
     <name>hadoop.proxyuser.<%= superuser %>.hosts</name>
     <value><%= data['hosts'] %></value>

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/flume-node.init
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/flume-node.init?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/flume-node.init (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/flume-node.init Mon Apr 16 15:36:06 2012
@@ -62,14 +62,20 @@ FLUME_PID_FILE=${FLUME_RUN}/flume-node.p
 install -d -m 0755 -o flume -g flume ${FLUME_RUN} 
 USER=flume
 
-HOSTNAME=`hostname`
-FLUME_NODE_NAME=${FLUME_NODE_NAME:-$HOSTNAME}
+DEFAULT_FLUME_NODE_NAME="agent"
+FLUME_NODE_NAME=${FLUME_NODE_NAME:-${DEFAULT_FLUME_NODE_NAME}}
 
 start() {
   [ -x $exec ] || exit $ERROR_PROGRAM_NOT_INSTALLED
+
+  checkstatus
+  status=$?
+  if [ "$status" -eq "$STATUS_RUNNING" ]; then
+    exit 0
+  fi
+
   log_success_msg "Starting $desc (flume-node): "
-  export FLUME_PID_FILE
-  /bin/su -s /bin/bash -c "${EXEC_PATH} node --conf $FLUME_CONF_DIR --conf-file $FLUME_CONF_FILE --name $FLUME_NODE_NAME &>>${FLUME_LOG_DIR}/flume-node.log &" $USER
+  /bin/su -s /bin/bash -c "/bin/bash -c 'echo \$\$ > ${FLUME_PID_FILE} && exec ${EXEC_PATH} node --conf $FLUME_CONF_DIR --conf-file $FLUME_CONF_FILE --name $FLUME_NODE_NAME &>>${FLUME_LOG_DIR}/flume-node.log' &" $USER
   RETVAL=$?
   [ $RETVAL -eq 0 ] && touch $LOCKFILE
   return $RETVAL
@@ -78,24 +84,18 @@ start() {
 stop() {
   if [ ! -e $FLUME_PID_FILE ]; then
     log_failure_msg "Flume node is not running"
-    exit $STATUS_NOT_RUNNING
+    exit 0
   fi
 
   log_success_msg "Stopping $desc (flume-node): "
 
-  # FLUME-919 will put an end to such extreme violence
   FLUME_PID=`cat $FLUME_PID_FILE`
   if [ -n $FLUME_PID ]; then
-    FLUME_PID_GROUP=$(ps -o pgrp -p ${FLUME_PID} h)
-
-    if [ -n $FLUME_PID_GROUP ]; then
-      kill -TERM -${FLUME_PID_GROUP} &>/dev/null
-      sleep 5
-      kill -KILL -${FLUME_PID_GROUP} &>/dev/null
-
-      rm -f $LOCKFILE $FLUME_PID_FILE
-    fi
+    kill -TERM ${FLUME_PID} &>/dev/null
+    sleep 5
+    kill -KILL ${FLUME_PID} &>/dev/null
   fi
+  rm -f $LOCKFILE $FLUME_PID_FILE
   return 0
 }
 
@@ -129,7 +129,7 @@ checkstatus(){
 }
 
 condrestart(){
-  [ -e /var/lock/subsys/flume-node ] && restart || :
+  [ -e ${LOCKFILE} ] && restart || :
 }
 
 case "$1" in

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/install_flume.sh
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/install_flume.sh?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/install_flume.sh (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/flume/install_flume.sh Mon Apr 16 15:36:06 2012
@@ -105,8 +105,19 @@ install -d -m 0755 ${PREFIX}/${FLUME_DIR
 (cd ${PREFIX}/${FLUME_DIR} &&
   tar --strip-components=1 -xvzf ${BUILD_DIR}/flume-ng-dist/target/flume-ng-dist-*-dist.tar.gz)
 
-# Take out things we've installed elsewhere
-for x in flume-ng-* conf pom.xml CHANGELOG DEVNOTES DISCLAIMER LICENSE NOTICE README RELEASE-NOTES; do
+# Take out useless things or we've installed elsewhere
+for x in flume-ng-* \
+          conf \
+          pom.xml \
+          CHANGELOG \
+          DEVNOTES \
+          DISCLAIMER \
+          LICENSE \
+          NOTICE \
+          README \
+          RELEASE-NOTES \
+          bin/ia64 \
+          bin/amd64; do
   rm -rf ${PREFIX}/$FLUME_DIR/$x 
 done
 

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/hive/install_hive.sh
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/hive/install_hive.sh?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/hive/install_hive.sh (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/hive/install_hive.sh Mon Apr 16 15:36:06 2012
@@ -160,5 +160,8 @@ chmod 755 ${PYTHON_DIR}/hive_metastore/*
 # Dir for Metastore DB
 install -d -m 1777 $PREFIX/var/lib/hive/metastore/
 
-# We need to get rid of jars that happen to be shipped in other CDH packages
+# We need to get rid of jars that happen to be shipped in other packages
 rm -f ${LIB_DIR}/lib/hbase-*.jar ${LIB_DIR}/lib/zookeeper-*.jar
+
+# Remove some source which gets installed
+rm -rf ${LIB_DIR}/lib/php/ext

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/pig/install_pig.sh
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/pig/install_pig.sh?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/pig/install_pig.sh (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/pig/install_pig.sh Mon Apr 16 15:36:06 2012
@@ -106,11 +106,17 @@ install -d -m 0755 $LIB_DIR
 (cd $BUILD_DIR && tar -cf - .) | (cd $LIB_DIR && tar -xf -)
 
 # Remove directories that are going elsewhere
-for dir in conf src lib-src docs tutorial test build.xml
+for dir in conf src lib-src docs tutorial test build.xml contrib/zebra/src contrib/piggybank ivy contrib/zebra/src-gen 
+
 do
    rm -rf $LIB_DIR/$dir
 done
 
+# Remove the unwanted xml and build files
+rm -rf $LIB_DIR/ivy.xml
+rm -rf $LIB_DIR/build.properties
+rm -rf $LIB_DIR/contrib/zebra/src/*.xml
+
 # Copy in the configuration files
 install -d -m 0755 $PREFIX/$CONF_DIST_DIR
 cp *.properties $PREFIX/$CONF_DIST_DIR

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/whirr/do-component-build
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/whirr/do-component-build?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/whirr/do-component-build (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/common/whirr/do-component-build Mon Apr 16 15:36:06 2012
@@ -16,6 +16,8 @@
 
 set -ex
 
+# BIGTOP-472 whirr build needs more perm gen space
+export MAVEN_OPT="-XX:MaxPermSize=128m"
 mvn clean -Pjavadoc site source:jar package assembly:assembly -DskipTests "$@" 
 
 mkdir build

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/control
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/control?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/control (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/control Mon Apr 16 15:36:06 2012
@@ -48,6 +48,12 @@ Description: HRegionServer makes a set o
  It checks in with the HMaster. There are many HRegionServers in a single 
  HBase deployment.
 
+Package: hbase-rest
+Architecture: all
+Depends: hbase (= ${source:Version})
+Description: The Apache HBase REST gateway
+ The Apache HBase REST gateway
+
 Package: hbase-thrift
 Architecture: all
 Depends: hbase (= ${source:Version}) 

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/install_init_scripts.sh
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/install_init_scripts.sh?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/install_init_scripts.sh (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/hbase/install_init_scripts.sh Mon Apr 16 15:36:06 2012
@@ -16,7 +16,7 @@
 # limitations under the License.
 
 SRC_PKG=hbase
-for node in master regionserver thrift ; do
+for node in master regionserver rest thrift ; do
     service_pkgdir=debian/$SRC_PKG-$node
     debdir=$service_pkgdir/DEBIAN
     template="debian/service-init.d.tpl"

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/oozie/oozie.preinst
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/oozie/oozie.preinst?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/oozie/oozie.preinst (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/deb/oozie/oozie.preinst Mon Apr 16 15:36:06 2012
@@ -38,7 +38,7 @@ case "$1" in
                   --system \
                   --disabled-login \
                   --group \
-                  --home /var/run/oozie \
+                  --home /var/lib/oozie \
                   --gecos "Oozie User" \
                   --shell /bin/false \
                   oozie  >/dev/null

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/flume/SPECS/flume.spec
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/flume/SPECS/flume.spec?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/flume/SPECS/flume.spec (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/flume/SPECS/flume.spec Mon Apr 16 15:36:06 2012
@@ -58,7 +58,7 @@ Group: Development/Libraries
 Buildroot: %{_topdir}/INSTALL/%{name}-%{version}
 BuildArch: noarch
 License: APL2
-Source0: flume-%{flume_base_version}.tar.gz
+Source0: apache-%{name}-%{flume_base_version}.tar.gz
 Source1: do-component-build
 Source2: install_%{name}.sh
 Source3: %{name}-node.init
@@ -105,7 +105,7 @@ Requires: redhat-lsb
 Flume is a reliable, scalable, and manageable distributed data collection application for collecting data such as logs and delivering it to data stores such as Hadoop's HDFS.  It can efficiently collect, aggregate, and move large amounts of log data.  It has a simple, but flexible, architecture based on streaming data flows.  It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.  The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic applications.
 
 %prep
-%setup -n %{name}-%{flume_base_version}
+%setup -n apache-%{name}-%{flume_base_version}
 
 %build
 env FLUME_VERSION=%{version} sh %{SOURCE1}
@@ -161,14 +161,20 @@ fi
 
 
 %files 
-%defattr(-,flume,flume)
-%config(noreplace) %{etc_flume}.empty
+%defattr(644,root,root,755)
 %doc %{doc_flume}
 
-%attr(0755,root,root) %{bin_flume}
-%attr(0755,root,root) %{lib_flume}
+
+%dir %{etc_flume}.empty
+%dir %{lib_flume}
+%dir %{lib_flume}/bin
+%dir %{lib_flume}/lib
+
+%config(noreplace) %{etc_flume}.empty/*
+%attr(0755,root,root) %{bin_flume}/flume-ng
+%attr(0755,root,root) %{lib_flume}/bin/flume-ng
+%{lib_flume}/lib/*.jar
+%{lib_flume}/conf
 
 %files node
 %attr(0755,root,root)/%{initd_dir}/%{name}-node
-%dir %{lib_flume}/bin
-%dir %{lib_flume}

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec Mon Apr 16 15:36:06 2012
@@ -23,7 +23,7 @@
 %define webapps_hbase %{hbase_home}/hbase-webapps
 %define man_dir %{_mandir}
 %define hbase_username hbase
-%define hbase_services master regionserver thrift
+%define hbase_services master regionserver thrift rest
 %define hadoop_home /usr/lib/hadoop
 %define zookeeper_home /usr/lib/zookeeper
 %define hbase_jar_deps %{hadoop_home}/hadoop-core.jar %{zookeeper_home}/zookeeper.jar
@@ -154,7 +154,6 @@ HRegionServer makes a set of HRegions av
 %package thrift
 Summary: The Hadoop HBase Thrift Interface
 Group: System/Daemons
-Provides: %{name}-thrift
 Requires: %{name} = %{version}-%{release}
 Requires(pre): %{name} = %{version}-%{release}
 
@@ -189,6 +188,32 @@ Obsoletes: %{name}-docs
 %description doc
 Documentation for Hbase
 
+%package rest
+Summary: The Apache HBase REST gateway
+Group: System/Daemons
+Requires: %{name} = %{version}-%{release}
+Requires(pre): %{name} = %{version}-%{release}
+
+%if  %{?suse_version:1}0
+# Required for init scripts
+Requires: insserv
+%endif
+
+%if  0%{?mgaversion}
+# Required for init scripts
+Requires: initscripts
+%endif
+
+# CentOS 5 does not have any dist macro
+# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora
+%if %{!?suse_version:1}0 && %{!?mgaversion:1}0
+# Required for init scripts
+Requires: redhat-lsb
+%endif
+
+
+%description rest
+The Apache HBase REST gateway
 
 %prep
 %setup -n %{name}-%{hbase_base_version}
@@ -294,3 +319,4 @@ fi
 %service_macro master
 %service_macro thrift
 %service_macro regionserver
+%service_macro rest

Modified: incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec Mon Apr 16 15:36:06 2012
@@ -143,7 +143,7 @@ Requires: bigtop-utils
 
 %pre
 getent group oozie >/dev/null || /usr/sbin/groupadd -r oozie >/dev/null
-getent passwd oozie >/dev/null || /usr/sbin/useradd --comment "Oozie User" --shell /bin/false -M -r -g oozie --home /var/run/oozie oozie >/dev/null
+getent passwd oozie >/dev/null || /usr/sbin/useradd --comment "Oozie User" --shell /bin/false -M -r -g oozie --home %{data_oozie} oozie >/dev/null
 
 %post 
 /sbin/chkconfig --add oozie 
@@ -184,7 +184,7 @@ fi
 %defattr(-, oozie, oozie)
 %dir %{_localstatedir}/log/oozie
 %dir %{_localstatedir}/run/oozie
-%{data_oozie}
+%attr(0755,oozie,oozie) %{data_oozie}
 
 %files client
 %defattr(-,root,root)

Modified: incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/AptCmdLinePackageManager.groovy
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/AptCmdLinePackageManager.groovy?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/AptCmdLinePackageManager.groovy (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/AptCmdLinePackageManager.groovy Mon Apr 16 15:36:06 2012
@@ -29,6 +29,18 @@ class AptCmdLinePackageManager extends P
      shRoot.exec("debconf-set-selections <<__EOT__\n${defaults}\n__EOT__");
   }
 
+  public int addBinRepo(String record, String url, String key) {
+    if (key) {
+      def text = key.toURL().text;
+      shRoot.exec("apt-key add - <<__EOT__\n${text}\n__EOT__");
+      if (shRoot.getRet()) {
+        return shRoot.getRet();
+      }
+    } else {
+      return addBinRepo(record, url);
+    }
+  }
+
   public int addBinRepo(String record, String url, String key, String cookie) {
     if (!url)
       url = ROOT_URL;

Modified: incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy Mon Apr 16 15:36:06 2012
@@ -42,6 +42,19 @@ public abstract class PackageManager {
    */
   abstract public int addBinRepo(String record, String url, String key, String cookie)
   /**
+  * Register a binary package repository so that packages can be accessed from it.
+  * NOTE: repository management is assumed to follow a KVP API with unique implementation
+  * specific keys (records) referencing tuples of information describing a repository
+  *
+  * @param record a package manager specific KEY portion of the repository registration (null is default)
+  * @param url a URL containing the packages constituting the repository (null is default)
+  * @param key an optional (can be null) cryptographic key for authenticating the content of the repository
+  * @return int return code of the operation: o in case of success, non-zero otherwise
+  */
+  public int addBinRepo(String record, String url, String key) {
+    addBinRepo(record, url);
+  }
+  /**
    * Register a binary package repository so that packages can be accessed from it.
    * NOTE: repository management is assumed to follow a KVP API with unique implementation
    * specific keys (records) referencing tuples of information describing a repository

Modified: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/PackageTestRepoMgr.groovy
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/PackageTestRepoMgr.groovy?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/PackageTestRepoMgr.groovy (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/PackageTestRepoMgr.groovy Mon Apr 16 15:36:06 2012
@@ -72,7 +72,7 @@ class PackageTestRepoMgr {
     pm.cleanup();
     try {
       String repoText = cdhRepoFileURL.toURL().text;
-      if (pm.addBinRepo(repoName, repoText)) {
+      if (pm.addBinRepo(repoName, repoText, cdhKeyURL)) {
         return false;
       }
 

Modified: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/TestPackagesBasics.groovy
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/TestPackagesBasics.groovy?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/TestPackagesBasics.groovy (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/TestPackagesBasics.groovy Mon Apr 16 15:36:06 2012
@@ -146,7 +146,7 @@ class TestPackagesBasics extends Package
   synchronized void testRemoteMetadata() { 
     if (!isUpgrade()) {
       if (pkg.isInstalled()) {
-        checkThat("package $name is alredy installed and could not be removed",
+        checkThat("package $name is already installed and could not be removed",
                   pkg.remove(), equalTo(0));
       }
 

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/pom.xml
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/pom.xml?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/pom.xml (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/pom.xml Mon Apr 16 15:36:06 2012
@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>org.apache.bigtop.itest</groupId>
+    <artifactId>bigtop-smokes</artifactId>
+    <version>0.3.0-incubating-SNAPSHOT</version>
+    <relativePath>../pom.xml</relativePath>
+  </parent>
+
+  <groupId>org.apache.bigtop.itest</groupId>
+    <artifactId>sqoopsmokes</artifactId>
+    <version>0.3.0-incubating-SNAPSHOT</version>
+    <name>sqoopsmokes</name>
+</project>

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopExport.groovy
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopExport.groovy?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopExport.groovy (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopExport.groovy Mon Apr 16 15:36:06 2012
@@ -0,0 +1,164 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p/>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p/>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.itest.sqoop;
+
+import static org.junit.Assert.assertEquals
+import static org.junit.Assert.assertNotNull
+import static org.junit.Assert.assertTrue
+import org.junit.AfterClass
+import org.junit.BeforeClass
+import org.junit.Test
+
+import org.apache.bigtop.itest.JarContent
+import org.apache.bigtop.itest.shell.Shell
+
+class TestSqoopExport {
+  private static String mysql_user =
+    System.getenv("MYSQL_USER");
+  private static String mysql_password =
+    System.getenv("MYSQL_PASSWORD");
+  private static final String MYSQL_USER =
+    (mysql_user == null) ? "root" : mysql_user;
+  private static final String MYSQL_PASSWORD =
+    (mysql_password == null) ? "" : mysql_password;
+  private static final String MYSQL_HOST = System.getProperty("mysql.host", "localhost");
+  private static final String HADOOP_HOME =
+    System.getenv('HADOOP_HOME');
+  private static String streaming_home = System.getenv('STREAMING_HOME');
+  private static final String STREAMING_HOME =
+    (streaming_home == null) ? HADOOP_HOME + "/contrib/streaming" :
+        streaming_home;
+  private static final String SQOOP_HOME =
+    System.getenv("SQOOP_HOME");
+  static {
+    assertNotNull("HADOOP_HOME is not set", HADOOP_HOME);
+    assertNotNull("SQOOP_HOME is not set", SQOOP_HOME);
+    assertNotNull("mysql connector jar is required to be present in $SQOOP_HOME/lib",
+      JarContent.getJarName("$SQOOP_HOME/lib", "mysql-connector-java.*.jar"));
+  }
+  private static String sqoop_jar =
+    JarContent.getJarName(SQOOP_HOME, "sqoop-1.*.jar");
+  private static String streaming_jar =
+    JarContent.getJarName(STREAMING_HOME, "hadoop.*streaming.*.jar");
+  static {
+    assertNotNull("Can't find sqoop.jar", sqoop_jar);
+    assertNotNull("Can't find hadoop-streaming.jar", streaming_jar);
+  }
+  private static final String SQOOP_JAR = SQOOP_HOME + "/" + sqoop_jar;
+  private static final String STREAMING_JAR = STREAMING_HOME + "/" + streaming_jar;
+  private static final String MYSQL_COMMAND =
+    "mysql --user=$MYSQL_USER" +
+    (("".equals(MYSQL_PASSWORD)) ? "" : " --password=$MYSQL_PASSWORD");
+  private static final String MYSQL_DBNAME = System.getProperty("mysql.dbname", "mysqltestdb");
+  private static final String SQOOP_CONNECTION =
+    "--connect jdbc:mysql://$MYSQL_HOST/$MYSQL_DBNAME --username=$MYSQL_USER" +
+    (("".equals(MYSQL_PASSWORD)) ? "" : " --password=$MYSQL_PASSWORD");
+  static {
+    System.out.println("SQOOP_CONNECTION string is " + SQOOP_CONNECTION );
+  }
+  private static final String DATA_DIR = System.getProperty("data.dir", "mysql-files");
+  private static final String INPUT = System.getProperty("input.dir", "input-dir");
+  private static Shell sh = new Shell("/bin/bash -s");
+
+  @BeforeClass
+  static void setUp() {
+    sh.exec("hadoop fs -test -e $INPUT");
+    if (sh.getRet() == 0) {
+      sh.exec("hadoop fs -rmr -skipTrash $INPUT");
+      assertTrue("Deletion of previous $INPUT from HDFS failed",
+          sh.getRet() == 0);
+    }
+    sh.exec("hadoop fs -mkdir $INPUT");
+    assertTrue("Could not create $INPUT directory", sh.getRet() == 0);
+    // unpack resource
+    JarContent.unpackJarContainer(TestSqoopExport.class, '.' , null)
+    // upload data to HDFS 
+    sh.exec("hadoop fs -put $DATA_DIR/sqoop-testtable.out input-dir/testtable/part-m-00000");
+    sh.exec("hadoop fs -put $DATA_DIR/sqoop-t_bool.out $INPUT/t_bool/part-m-00000");
+    sh.exec("hadoop fs -put $DATA_DIR/sqoop-t_date-export.out $INPUT/t_date/part-m-00000");
+    sh.exec("hadoop fs -put $DATA_DIR/sqoop-t_string.out $INPUT/t_string/part-m-00000");
+    sh.exec("hadoop fs -put $DATA_DIR/sqoop-t_fp.out $INPUT/t_fp/part-m-00000");
+    sh.exec("hadoop fs -put $DATA_DIR/sqoop-t_int.out $INPUT/t_int/part-m-00000"); 
+    
+    //create db
+    sh.exec("cat $DATA_DIR/mysql-create-db.sql | $MYSQL_COMMAND");
+    //create tables
+    sh.exec("cat $DATA_DIR/mysql-create-tables.sql | $MYSQL_COMMAND");
+  }
+
+  @AfterClass
+  static void tearDown() {
+    if ('YES'.equals(System.getProperty('delete.testdata','no').toUpperCase())) {
+      sh.exec("hadoop fs -test -e $INPUT");
+      if (sh.getRet() == 0) {
+       // sh.exec("hadoop fs -rmr -skipTrash $INPUT");
+        assertTrue("Deletion of $INPUT from HDFS failed",
+            sh.getRet() == 0);
+      }
+    }
+  }
+
+  @Test
+  public void testBooleanExport() {
+    sh.exec("sqoop export $SQOOP_CONNECTION --table t_bool --export-dir $INPUT/t_bool");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("echo 'use mysqltestdb;select * from t_bool' | $MYSQL_COMMAND --skip-column-names | sed 's/\t/,/g' > t_bool.out");
+    assertEquals("sqoop export did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_bool-export.out t_bool.out").getRet());
+  }
+
+  
+  @Test
+  public void testIntegerExport() {
+    sh.exec("sqoop export $SQOOP_CONNECTION --table t_int --export-dir $INPUT/t_int");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("echo 'use mysqltestdb;select * from t_int' | $MYSQL_COMMAND --skip-column-names | sed 's/\t/,/g' > t_int.out");
+    assertEquals("sqoop export did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_int.out t_int.out").getRet());
+  }
+
+  @Test
+  public void testFixedPointFloatingPointExport() {
+    sh.exec("sqoop export $SQOOP_CONNECTION --table t_fp --export-dir $INPUT/t_fp");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("echo 'use mysqltestdb;select * from t_fp' | $MYSQL_COMMAND --skip-column-names | sed 's/\t/,/g' > t_fp.out");
+    assertEquals("sqoop export did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_fp.out t_fp.out").getRet());
+  }
+
+  @Test
+  public void testDateTimeExport() {
+    sh.exec("sqoop export $SQOOP_CONNECTION --table t_date --export-dir $INPUT/t_date");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("echo 'use mysqltestdb;select * from t_date' | $MYSQL_COMMAND --skip-column-names | sed 's/\t/,/g' > t_date.out");
+    assertEquals("sqoop export did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_date.out t_date.out").getRet());
+  }
+
+  @Test
+  public void testStringExport() {
+    sh.exec("sqoop export $SQOOP_CONNECTION --table t_string --export-dir $INPUT/t_string");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("echo 'use mysqltestdb;select * from t_string' | $MYSQL_COMMAND --skip-column-names | sed 's/\t/,/g' > t_string.out");
+    assertEquals("sqoop export did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_string.out t_string.out").getRet());
+  }
+
+}
+

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopImport.groovy
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopImport.groovy?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopImport.groovy (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/groovy/org/apache/itest/sqoop/TestSqoopImport.groovy Mon Apr 16 15:36:06 2012
@@ -0,0 +1,264 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p/>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p/>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.itest.sqoop;
+
+import static org.junit.Assert.assertEquals
+import static org.junit.Assert.assertNotNull
+import static org.junit.Assert.assertTrue
+import org.junit.AfterClass
+import org.junit.BeforeClass
+import org.junit.Test
+
+import org.apache.bigtop.itest.JarContent
+import org.apache.bigtop.itest.shell.Shell
+
+class TestSqoopImport {
+  private static String mysql_user =
+    System.getenv("MYSQL_USER");
+  private static String mysql_password =
+    System.getenv("MYSQL_PASSWORD");
+  private static final String MYSQL_USER =
+    (mysql_user == null) ? "root" : mysql_user;
+  private static final String MYSQL_PASSWORD =
+    (mysql_password == null) ? "" : mysql_password;
+  private static final String MYSQL_HOST = System.getProperty("mysql.host", "localhost");
+  private static final String HADOOP_HOME =
+    System.getenv('HADOOP_HOME');
+  private static String streaming_home = System.getenv('STREAMING_HOME');
+  private static final String STREAMING_HOME =
+    (streaming_home == null) ? HADOOP_HOME + "/contrib/streaming" :
+        streaming_home;
+  private static final String SQOOP_HOME =
+    System.getenv("SQOOP_HOME");
+  static {
+    assertNotNull("HADOOP_HOME is not set", HADOOP_HOME);
+    assertNotNull("SQOOP_HOME is not set", SQOOP_HOME);
+    assertNotNull("mysql connector jar is required to be present in $SQOOP_HOME/lib",
+      JarContent.getJarName("$SQOOP_HOME/lib", "mysql-connector-java.*.jar"));
+  }
+  private static String sqoop_jar =
+    JarContent.getJarName(SQOOP_HOME, "sqoop-1.*.jar");
+  private static String streaming_jar =
+    JarContent.getJarName(STREAMING_HOME, "hadoop.*streaming.*.jar");
+  static {
+    assertNotNull("Can't find sqoop.jar", sqoop_jar);
+    assertNotNull("Can't find hadoop-streaming.jar", streaming_jar);
+  }
+  private static final String SQOOP_JAR = SQOOP_HOME + "/" + sqoop_jar;
+  private static final String STREAMING_JAR = STREAMING_HOME + "/" + streaming_jar;
+  private static final String MYSQL_COMMAND =
+    "mysql --user=$MYSQL_USER" +
+    (("".equals(MYSQL_PASSWORD)) ? "" : " --password=$MYSQL_PASSWORD");
+  private static final String MYSQL_DBNAME = System.getProperty("mysql.dbname", "mysqltestdb");
+  private static final String SQOOP_CONNECTION =
+    "--connect jdbc:mysql://$MYSQL_HOST/$MYSQL_DBNAME --username=$MYSQL_USER" +
+    (("".equals(MYSQL_PASSWORD)) ? "" : " --password=$MYSQL_PASSWORD");
+  static {
+    System.out.println("SQOOP_CONNECTION string is " + SQOOP_CONNECTION );
+  }
+  private static final String DATA_DIR = System.getProperty("data.dir", "mysql-files");
+  private static final String OUTPUT = System.getProperty("output.dir", "output-dir");
+  private static Shell sh = new Shell("/bin/bash -s");
+
+  @BeforeClass
+  static void setUp() {
+    sh.exec("hadoop fs -test -e $OUTPUT");
+    if (sh.getRet() == 0) {
+      sh.exec("hadoop fs -rmr -skipTrash $OUTPUT");
+      assertTrue("Deletion of previous $OUTPUT from HDFS failed",
+          sh.getRet() == 0);
+    }
+    sh.exec("hadoop fs -mkdir $OUTPUT");
+    assertTrue("Could not create $OUTPUT directory", sh.getRet() == 0);
+    // unpack resource
+    JarContent.unpackJarContainer(TestSqoopImport.class, '.' , null)
+    // create the database
+    sh.exec("cat $DATA_DIR/mysql-create-db.sql | $MYSQL_COMMAND");
+    //create tables
+    sh.exec("cat $DATA_DIR/mysql-create-tables.sql | $MYSQL_COMMAND");
+    //populate data
+    sh.exec("cat $DATA_DIR/mysql-insert-data.sql | $MYSQL_COMMAND");
+  }
+
+  @AfterClass
+  static void tearDown() {
+    if ('YES'.equals(System.getProperty('delete.testdata','no').toUpperCase())) {
+      sh.exec("hadoop fs -test -e $OUTPUT");
+      if (sh.getRet() == 0) {
+        sh.exec("hadoop fs -rmr -skipTrash $OUTPUT");
+        assertTrue("Deletion of $OUTPUT from HDFS failed",
+            sh.getRet() == 0);
+      }
+    }
+  }
+
+  @Test
+  public void testBooleanImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table t_bool --target-dir $OUTPUT/t_bool");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/t_bool/part-* > t_bool.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_bool.out t_bool.out").getRet());
+  }
+
+  
+  @Test
+  public void testIntegerImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table t_int --target-dir $OUTPUT/t_int");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/t_int/part-* > t_int.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_int.out t_int.out").getRet());
+  }
+
+  @Test
+  public void testFixedPointFloatingPointImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table t_fp --target-dir $OUTPUT/t_fp");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/t_fp/part-* > t_fp.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_fp.out t_fp.out").getRet());
+  }
+
+  @Test
+  public void testDateTimeImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table t_date --target-dir $OUTPUT/t_date");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/t_date/part-* > t_date.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_date.out t_date.out").getRet());
+  }
+
+  @Test
+  public void testStringImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table t_string --target-dir $OUTPUT/t_string");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/t_string/part-* > t_string.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-t_string.out t_string.out").getRet());
+  }
+  
+  @Test
+  public void testAppendImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --target-dir $OUTPUT/append");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    //import again with append
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --append --target-dir $OUTPUT/append");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/append/part-* > append.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-append.out append.out").getRet());
+  }
+  
+  @Test
+  public void testColumnsImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --columns id,fname --target-dir $OUTPUT/columns");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/columns/part-* > columns.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-columns.out columns.out").getRet());
+  }
+
+  @Test
+  public void testDirectImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --direct --target-dir $OUTPUT/direct");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/direct/part-* > direct.out");    
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-testtable.out direct.out").getRet());
+  }
+  
+  @Test
+  public void testNumMappersImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --num-mappers 1 --target-dir $OUTPUT/num-mappers");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/num-mappers/part-*0 > num-mappers.out");    
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-testtable.out num-mappers.out").getRet());
+  }
+
+  @Test
+  public void testQueryImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --query 'select t1.id as id, t2.fname as fname from testtable as t1 join testtable2 as t2 on (t1.id = t2.id        ) where t1.id < 3 AND \$CONDITIONS' --split-by t1.id --target-dir $OUTPUT/query");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/query/part-* > query.out");    
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-query.out query.out").getRet());
+  }
+
+  @Test
+  public void testSplityByImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --split-by fname --target-dir $OUTPUT/split-by");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/split-by/part-* > split-by.out");    
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-testtable.out split-by.out").getRet());
+  }
+
+  @Test
+  public void testWarehouseDirImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --warehouse-dir $OUTPUT/warehouse-dir");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/warehouse-dir/testtable/part-* > warehouse-dir.out");    
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-testtable.out warehouse-dir.out").getRet());
+  }
+
+  @Test
+  public void testWhereClauseImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testtable --where \"id < 5\" --target-dir $OUTPUT/where-clause");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);    
+    sh.exec("hadoop fs -cat $OUTPUT/where-clause/part-* > where-clause.out");    
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-where-clause.out where-clause.out").getRet());
+  }
+
+  @Test
+  public void testNullStringImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testnullvalues --null-string mynullstring --target-dir $OUTPUT/null-string");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/null-string/part-* > null-string.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-null-string.out null-string.out").getRet());
+  }
+
+  @Test
+  public void testNullNonStringImport() {
+    sh.exec("sqoop import $SQOOP_CONNECTION --table testnullvalues --null-non-string 10 --target-dir $OUTPUT/non-null-string");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/non-null-string/part-* > non-null-string.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-null-non-string.out non-null-string.out").getRet());
+  }
+  
+  //database name is hardcoded here 
+  @Test
+    public void testImportAllTables() {
+    String SQOOP_CONNECTION_IMPORT_ALL =
+    "--connect jdbc:mysql://$MYSQL_HOST/mysqltestdb2 --username=$MYSQL_USER" +
+    (("".equals(MYSQL_PASSWORD)) ? "" : " --password=$MYSQL_PASSWORD");
+
+    sh.exec("sqoop import-all-tables $SQOOP_CONNECTION_IMPORT_ALL --warehouse-dir $OUTPUT/alltables");
+    assertTrue("Sqoop job failed!", sh.getRet() == 0);
+    sh.exec("hadoop fs -cat $OUTPUT/alltables/testtable*/part-* > all-tables.out");
+    assertEquals("sqoop import did not write expected data",
+        0, sh.exec("diff -u $DATA_DIR/sqoop-all-tables.out all-tables.out").getRet());
+  }
+}
+

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-db.sql
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-db.sql?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-db.sql (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-db.sql Mon Apr 16 15:36:06 2012
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p/>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p/>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+drop database if exists mysqltestdb;
+
+create database mysqltestdb;
+
+use mysqltestdb;
+
+grant all privileges on mysqltestdb.* to 'root'@'%';
+
+flush privileges;
+
+/* create database2 for import-all */
+
+drop database if exists mysqltestdb2;
+
+create database mysqltestdb2;
+
+use mysqltestdb2;
+
+grant all privileges on mysqltestdb2.* to 'root'@'%';
+
+flush privileges;

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-tables.sql
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-tables.sql?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-tables.sql (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-create-tables.sql Mon Apr 16 15:36:06 2012
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p/>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p/>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/* functional test db tables */
+use mysqltestdb;
+
+drop table if exists t_bool;
+create table t_bool ( pri int not null auto_increment, a boolean, primary key (pri));
+
+drop table if exists t_int;
+create table t_int ( pri int not null auto_increment,a tinyint, b smallint, c mediumint , d int, e integer , f bigint,primary key (pri));
+
+drop table if exists t_fp;
+create table t_fp (pri int not null auto_increment, a decimal (6,4), b float (6,4), c double precision (6,4),d float (4),primary key (pri));
+
+drop table if exists t_date;
+create table t_date (pri int not null auto_increment,a date , c timestamp, d time , e year(4),primary key (pri));
+
+drop table if exists t_string;
+create table t_string (pri int not null auto_increment, a char (5), b varchar (5), c text , d tinytext, e mediumtext, f mediumtext, g longtext, h enum ('A','B'),primary key (pri));
+
+drop table if exists testtable;
+create table testtable ( id int , fname varchar (20), lname varchar(20) , primary key (id)); 
+
+drop table if exists testtable2;
+create table testtable2 ( id int , fname varchar (20), lname varchar(20) , primary key (id));
+
+drop table if exists testnullvalues;
+create table testnullvalues ( id int , a int , b varchar(20) , primary key (id));
+
+/* import-all test db-tables */
+
+use mysqltestdb2;
+
+drop table if exists testtable;
+create table testtable ( id int , fname varchar (20), lname varchar(20) , primary key (id)); 
+
+drop table if exists testtable2;
+create table testtable2 ( id int , fname varchar (20), lname varchar(20) , primary key (id));

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-insert-data.sql
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-insert-data.sql?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-insert-data.sql (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/mysql-insert-data.sql Mon Apr 16 15:36:06 2012
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p/>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p/>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mysqltestdb;
+
+insert into t_int (a,b,c,d,e,f) values (127,32767,8388607,2147483647,-2147483648,9223372036854775807);
+
+insert into t_bool (a) values (1);
+
+insert into t_fp(a,b,c,d) values (-99.9999,-99.9999,99.9999,99.99);
+
+insert into t_date (a,c,d,e) values  ('9999-12-31',19731230153000,'23:59:59','2155');
+
+insert into t_string (a,b,c,d,e,f,g,h) values ('aaa','aaa','0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR','0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR','0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR','0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR','0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9
 Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR','A');
+
+insert into testtable values (1,'aaa','aaa');
+insert into testtable values (2,'bbb','bbb');
+insert into testtable values (3,'ccc','ccc');
+insert into testtable values (4,'ddd','ddd');
+insert into testtable values (5,'eee','eee');
+insert into testtable values (6,'fff','fff');
+insert into testtable values (7,'ggg','ggg');
+insert into testtable values (8,'hhh','hhh');
+insert into testtable values (9,'iii','iii');
+insert into testtable values (10,'jjj','jjj');
+
+
+insert into testtable2 values (1,'111','111');
+insert into testtable2 values (2,'222','222');
+insert into testtable2 values (3,'333','333');
+
+
+insert into testnullvalues (id) values (1);
+insert into testnullvalues (id,a) values (2,2);
+insert into testnullvalues (id,b) values (3,'aaa');
+
+/* data for import-all test */
+
+use mysqltestdb2;
+
+insert into testtable values (1,'aaa','aaa');
+insert into testtable2 values (1,'aaa','aaa');

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-all-tables.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-all-tables.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-all-tables.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-all-tables.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,2 @@
+1,aaa,aaa
+1,aaa,aaa

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-append.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-append.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-append.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-append.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,20 @@
+1,aaa,aaa
+2,bbb,bbb
+3,ccc,ccc
+4,ddd,ddd
+5,eee,eee
+6,fff,fff
+7,ggg,ggg
+8,hhh,hhh
+9,iii,iii
+10,jjj,jjj
+1,aaa,aaa
+2,bbb,bbb
+3,ccc,ccc
+4,ddd,ddd
+5,eee,eee
+6,fff,fff
+7,ggg,ggg
+8,hhh,hhh
+9,iii,iii
+10,jjj,jjj

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-columns.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-columns.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-columns.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-columns.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,10 @@
+1,aaa
+2,bbb
+3,ccc
+4,ddd
+5,eee
+6,fff
+7,ggg
+8,hhh
+9,iii
+10,jjj

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-non-string.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-non-string.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-non-string.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-non-string.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,3 @@
+1,10,null
+2,2,null
+3,10,aaa

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-string.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-string.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-string.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-null-string.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,3 @@
+1,null,mynullstring
+2,2,mynullstring
+3,null,aaa

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-query.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-query.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-query.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-query.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,2 @@
+1,111
+2,222

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool-export.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool-export.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool-export.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool-export.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1 @@
+1,1

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_bool.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1 @@
+1,true

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date-export.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date-export.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date-export.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date-export.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1 @@
+1,"9999-12-31","9999-12-31 23:59:59","1973-12-30 15:30:00","23:59:59","2155"

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_date.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1 @@
+1,9999-12-31,1973-12-30 15:30:00.0,23:59:59,2155

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_fp.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_fp.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_fp.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_fp.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1 @@
+1,-99.9999,-99.9999,99.9999,99.99

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_int.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_int.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_int.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_int.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1 @@
+1,127,32767,8388607,2147483647,-2147483648,9223372036854775807

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_string.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_string.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_string.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-t_string.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1 @@
+1,aaa,aaa,0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR,0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR,0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR,0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7lhI0HJXmvyDkJEqOc0VxdT0ySR,0GIQM3CONg5jAUloRoP7f76TJrYkYnto5i76IUDbKuG1dMSP5znIM6Ct3qc1WdRCD9THFjWXBIxOoAqhOgdFQEwICX7fYN4n9P3AOoo3ZrnUBu3rtuwf9Y2HXJvUBgAqs44Ypdg6iv511JQSufMeKvbLIpIbk9saJW82EeTgoWkAS7l
 hI0HJXmvyDkJEqOc0VxdT0ySR,A

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-testtable.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-testtable.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-testtable.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-testtable.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,10 @@
+1,aaa,aaa
+2,bbb,bbb
+3,ccc,ccc
+4,ddd,ddd
+5,eee,eee
+6,fff,fff
+7,ggg,ggg
+8,hhh,hhh
+9,iii,iii
+10,jjj,jjj

Added: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-where-clause.out
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-where-clause.out?rev=1326652&view=auto
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-where-clause.out (added)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-artifacts/sqoop-smokes/src/main/resources/mysql-files/sqoop-where-clause.out Mon Apr 16 15:36:06 2012
@@ -0,0 +1,4 @@
+1,aaa,aaa
+2,bbb,bbb
+3,ccc,ccc
+4,ddd,ddd

Modified: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml Mon Apr 16 15:36:06 2012
@@ -49,6 +49,11 @@
       <scope>test</scope>
     </dependency>
     <dependency>
+      <groupId>org.apache.cxf</groupId>
+      <artifactId>cxf-rt-frontend-jaxrs</artifactId>
+      <version>2.5.0</version>
+    </dependency>
+    <dependency>
       <groupId>org.apache.hadoop</groupId>
       <artifactId>hadoop-core</artifactId>
     </dependency>

Copied: incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/sqoop-smokes/pom.xml (from r1302958, incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/sqoop-smokes/pom.xml?p2=incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/sqoop-smokes/pom.xml&p1=incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml&r1=1302958&r2=1326652&rev=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/hadoop/pom.xml (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop-tests/test-execution/smokes/sqoop-smokes/pom.xml Mon Apr 16 15:36:06 2012
@@ -17,46 +17,36 @@
 -->
 <project xmlns="http://maven.apache.org/POM/4.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
-  <modelVersion>4.0.0</modelVersion>
+                  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
   <parent>
     <groupId>org.apache.bigtop.itest</groupId>
     <artifactId>smoke-tests</artifactId>
-    <version>0.3.0-incubating</version>
+    <version>0.3.0-incubating-SNAPSHOT</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 
+  <modelVersion>4.0.0</modelVersion>
   <groupId>org.apache.bigtop.itest</groupId>
-  <artifactId>hadoop-smoke-execution</artifactId>
-  <version>0.3.0-incubating</version>
-  <name>Hadoop smoke test execution</name>
-
+  <artifactId>sqoop-smoke-execution</artifactId>
+  <version>0.3.0-incubating-SNAPSHOT</version>
+  <name>Sqoop smoke test execution</name>
+  
   <properties>
+    <!--Additional environment variables are required-->
+    <SQOOP_HOME>${env.SQOOP_HOME}</SQOOP_HOME>
+    <!-- Integration tests are special -->
     <org.apache.maven-dependency-plugin.groupId>org.apache.bigtop.itest</org.apache.maven-dependency-plugin.groupId>
-    <org.apache.maven-dependency-plugin.artifactId>hadoop-smoke</org.apache.maven-dependency-plugin.artifactId>
-    <org.apache.maven-dependency-plugin.version>${hadoop-smoke.version}</org.apache.maven-dependency-plugin.version>
-    <org.apache.maven-dependency-plugin.output>${project.build.directory}</org.apache.maven-dependency-plugin.output>
-    <org.apache.maven-dependency-plugin.pattern>**/*</org.apache.maven-dependency-plugin.pattern>
+    <org.apache.maven-dependency-plugin.artifactId>sqoopsmokes</org.apache.maven-dependency-plugin.artifactId>
+    <org.apache.maven-dependency-plugin.version>0.3.0-incubating-SNAPSHOT</org.apache.maven-dependency-plugin.version>
     <org.apache.maven-dependency-plugin.type>jar</org.apache.maven-dependency-plugin.type>
   </properties>
 
   <dependencies>
     <dependency>
-      <groupId>commons-logging</groupId>
-      <artifactId>commons-logging</artifactId>
-      <version>1.1</version>
-      <scope>test</scope>
-    </dependency>
-    <dependency>
-      <groupId>org.apache.hadoop</groupId>
-      <artifactId>hadoop-core</artifactId>
-    </dependency>
-    <!-- explicit dependency is needed to pull down transient deps. such as hadoop-examples -->
-    <dependency>
       <groupId>${org.apache.maven-dependency-plugin.groupId}</groupId>
       <artifactId>${org.apache.maven-dependency-plugin.artifactId}</artifactId>
-      <version>${hadoop-smoke.version}</version>
+      <version>${org.apache.maven-dependency-plugin.version}</version>
     </dependency>
   </dependencies>
 
@@ -69,16 +59,31 @@
 
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-enforcer-plugin</artifactId>
+        <version>1.0</version>
+        <executions>
+          <execution>
+            <id>enforce-property</id>
+            <goals>
+              <goal>enforce</goal>
+            </goals>
+            <configuration>
+              <rules>
+                <requireProperty>
+                  <property>SQOOP_HOME</property>
+                  <message>SQOOP_HOME env. variable has to be set</message>
+                </requireProperty>
+              </rules>
+              <fail>true</fail>
+            </configuration>
+          </execution>
+        </executions>
+      </plugin>
+
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-failsafe-plugin</artifactId>
         <version>2.7.1</version>
-        <configuration>
-          <forkMode>always</forkMode>
-          <systemPropertyVariables>
-            <test.cache.data>
-              ${project.build.directory}/clitest_data
-            </test.cache.data>
-          </systemPropertyVariables>
-        </configuration>
       </plugin>
     </plugins>
   </build>

Modified: incubator/bigtop/branches/branch-0.3/bigtop.mk
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/bigtop.mk?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/bigtop.mk (original)
+++ incubator/bigtop/branches/branch-0.3/bigtop.mk Mon Apr 16 15:36:06 2012
@@ -23,7 +23,9 @@ HADOOP_PKG_VERSION=$(HADOOP_BASE_VERSION
 HADOOP_RELEASE_VERSION=1
 HADOOP_TARBALL_DST=$(HADOOP_NAME)-$(HADOOP_BASE_VERSION).tar.gz
 HADOOP_TARBALL_SRC=$(HADOOP_TARBALL_DST)
-HADOOP_SITE=$(APACHE_MIRROR)/hadoop/common/$(HADOOP_NAME)-$(HADOOP_BASE_VERSION)/
+HADOOP_DOWNLOAD_PATH=/hadoop/common/$(HADOOP_NAME)-$(HADOOP_BASE_VERSION)
+HADOOP_SITE=$(APACHE_MIRROR)$(HADOOP_DOWNLOAD_PATH)
+HADOOP_ARCHIVE=$(APACHE_ARCHIVE)$(HADOOP_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,hadoop,HADOOP))
 
 # ZooKeeper
@@ -35,7 +37,9 @@ ZOOKEEPER_PKG_VERSION=3.4.3
 ZOOKEEPER_RELEASE_VERSION=1
 ZOOKEEPER_TARBALL_DST=zookeeper-$(ZOOKEEPER_BASE_VERSION).tar.gz
 ZOOKEEPER_TARBALL_SRC=$(ZOOKEEPER_TARBALL_DST)
-ZOOKEEPER_SITE=$(APACHE_MIRROR)/zookeeper/zookeeper-$(ZOOKEEPER_BASE_VERSION)/
+ZOOKEEPER_DOWNLOAD_PATH=/zookeeper/zookeeper-$(ZOOKEEPER_BASE_VERSION)
+ZOOKEEPER_SITE=$(APACHE_MIRROR)$(ZOOKEEPER_DOWNLOAD_PATH)
+ZOOKEEPER_ARCHIVE=$(APACHE_ARCHIVE)$(ZOOKEEPER_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,zookeeper,ZOOKEEPER))
 
 # HBase
@@ -47,7 +51,9 @@ HBASE_PKG_VERSION=$(HBASE_BASE_VERSION)
 HBASE_RELEASE_VERSION=1
 HBASE_TARBALL_DST=hbase-$(HBASE_BASE_VERSION).tar.gz
 HBASE_TARBALL_SRC=$(HBASE_TARBALL_DST)
-HBASE_SITE=$(APACHE_MIRROR)/hbase/hbase-$(HBASE_BASE_VERSION)/
+HBASE_DOWNLOAD_PATH=/hbase/hbase-$(HBASE_BASE_VERSION)
+HBASE_SITE=$(APACHE_MIRROR)$(HBASE_DOWNLOAD_PATH)
+HBASE_ARCHIVE=$(APACHE_ARCHIVE)$(HBASE_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,hbase,HBASE))
 
 # Pig
@@ -59,7 +65,9 @@ PIG_RELNOTES_NAME=Apache Pig
 PIG_PKG_NAME=pig
 PIG_TARBALL_DST=pig-$(PIG_BASE_VERSION).tar.gz
 PIG_TARBALL_SRC=$(PIG_TARBALL_DST)
-PIG_SITE=$(APACHE_MIRROR)/pig/pig-$(PIG_BASE_VERSION)/
+PIG_DOWNLOAD_PATH=/pig/pig-$(PIG_BASE_VERSION)
+PIG_SITE=$(APACHE_MIRROR)$(PIG_DOWNLOAD_PATH)
+PIG_ARCHIVE=$(APACHE_ARCHIVE)$(PIG_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,pig,PIG))
 
 # Hive
@@ -71,7 +79,9 @@ HIVE_PKG_VERSION=$(HIVE_BASE_VERSION)
 HIVE_RELEASE_VERSION=1
 HIVE_TARBALL_DST=hive-$(HIVE_BASE_VERSION).tar.gz
 HIVE_TARBALL_SRC=$(HIVE_TARBALL_DST)
-HIVE_SITE=$(APACHE_MIRROR)/hive/hive-$(HIVE_BASE_VERSION)/
+HIVE_DOWNLOAD_PATH=/hive/hive-$(HIVE_BASE_VERSION)
+HIVE_SITE=$(APACHE_MIRROR)$(HIVE_DOWNLOAD_PATH)
+HIVE_ARCHIVE=$(APACHE_ARCHIVE)$(HIVE_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,hive,HIVE))
 
 # Sqoop
@@ -83,10 +93,11 @@ SQOOP_PKG_VERSION=1.4.1
 SQOOP_RELEASE_VERSION=1
 SQOOP_TARBALL_DST=$(SQOOP_NAME)-$(SQOOP_BASE_VERSION).tar.gz
 SQOOP_TARBALL_SRC=$(SQOOP_NAME)-$(SQOOP_BASE_VERSION)-src.tar.gz
-SQOOP_SITE=$(APACHE_MIRROR)/incubator/$(SQOOP_NAME)/$(SQOOP_NAME)-$(SQOOP_BASE_VERSION)/
+SQOOP_DOWNLOAD_PATH=/sqoop/$(SQOOP_BASE_VERSION)
+SQOOP_SITE=$(APACHE_MIRROR)$(SQOOP_DOWNLOAD_PATH)
+SQOOP_ARCHIVE=$(APACHE_ARCHIVE)$(SQOOP_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,sqoop,SQOOP))
 
-
 # Oozie
 OOZIE_NAME=oozie
 OOZIE_RELNOTES_NAME=Apache Oozie
@@ -96,19 +107,24 @@ OOZIE_PKG_VERSION=3.1.3
 OOZIE_RELEASE_VERSION=1
 OOZIE_TARBALL_DST=oozie-$(OOZIE_BASE_VERSION).tar.gz
 OOZIE_TARBALL_SRC=oozie-$(OOZIE_BASE_VERSION)-src.tar.gz
-OOZIE_SITE=$(APACHE_MIRROR)/incubator/$(OOZIE_NAME)/$(OOZIE_NAME)-$(OOZIE_BASE_VERSION)/
+OOZIE_DOWNLOAD_PATH=/incubator/$(OOZIE_NAME)/$(OOZIE_NAME)-$(OOZIE_BASE_VERSION)
+OOZIE_SITE=$(APACHE_MIRROR)$(OOZIE_DOWNLOAD_PATH)
+OOZIE_ARCHIVE=$(APACHE_ARCHIVE)$(OOZIE_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,oozie,OOZIE))
 
 # Whirr
 WHIRR_NAME=whirr
 WHIRR_RELNOTES_NAME=Apache Whirr
 WHIRR_PKG_NAME=whirr
-WHIRR_BASE_VERSION=0.7.0
-WHIRR_PKG_VERSION=0.7.0
+WHIRR_BASE_VERSION=0.7.1
+WHIRR_PKG_VERSION=0.7.1
 WHIRR_RELEASE_VERSION=1
 WHIRR_TARBALL_DST=whirr-$(WHIRR_BASE_VERSION).tar.gz
 WHIRR_TARBALL_SRC=$(WHIRR_TARBALL_DST)
-WHIRR_SITE=$(APACHE_ARCHIVE)/whirr/whirr-$(WHIRR_BASE_VERSION)/
+WHIRR_DOWNLOAD_PATH=/whirr/whirr-$(WHIRR_BASE_VERSION)
+WHIRR_SITE=$(APACHE_MIRROR)$(WHIRR_DOWNLOAD_PATH)
+WHIRR_ARCHIVE=$(APACHE_ARCHIVE)$(WHIRR_DOWNLOAD_PATH)
+
 $(eval $(call PACKAGE,whirr,WHIRR))
 
 # Mahout
@@ -120,19 +136,23 @@ MAHOUT_PKG_VERSION=0.6
 MAHOUT_RELEASE_VERSION=1
 MAHOUT_TARBALL_DST=mahout-distribution-$(MAHOUT_BASE_VERSION)-src.tar.gz
 MAHOUT_TARBALL_SRC=$(MAHOUT_TARBALL_DST)
-MAHOUT_SITE=$(APACHE_MIRROR)/mahout/$(MAHOUT_BASE_VERSION)/
+MAHOUT_DOWNLOAD_PATH=/mahout/$(MAHOUT_BASE_VERSION)
+MAHOUT_SITE=$(APACHE_MIRROR)$(MAHOUT_DOWNLOAD_PATH)
+MAHOUT_ARCHIVE=$(APACHE_ARCHIVE)$(MAHOUT_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,mahout,MAHOUT))
 
 # Flume
 FLUME_NAME=flume
 FLUME_RELNOTES_NAME=Flume
 FLUME_PKG_NAME=flume
-FLUME_BASE_VERSION=1.0.0-incubating
-FLUME_PKG_VERSION=1.0.0
+FLUME_BASE_VERSION=1.1.0-incubating
+FLUME_PKG_VERSION=1.1.0
 FLUME_RELEASE_VERSION=1
-FLUME_TARBALL_DST=$(FLUME_NAME)-$(FLUME_BASE_VERSION).tar.gz
-FLUME_TARBALL_SRC=$(FLUME_NAME)-$(FLUME_BASE_VERSION).tar.gz
-FLUME_SITE=$(APACHE_MIRROR)/incubator/$(FLUME_NAME)/$(FLUME_NAME)-$(FLUME_BASE_VERSION)/
+FLUME_TARBALL_DST=apache-$(FLUME_NAME)-$(FLUME_BASE_VERSION).tar.gz
+FLUME_TARBALL_SRC=apache-$(FLUME_NAME)-$(FLUME_BASE_VERSION).tar.gz
+FLUME_DOWNLOAD_PATH=/incubator/flume/$(FLUME_PKG_NAME)-$(FLUME_BASE_VERSION)
+FLUME_SITE=$(APACHE_MIRROR)$(FLUME_DOWNLOAD_PATH)
+FLUME_ARCHIVE=$(APACHE_ARCHIVE)$(FLUME_DOWNLOAD_PATH)
 $(eval $(call PACKAGE,flume,FLUME))
 
 # Bigtop-utils

Modified: incubator/bigtop/branches/branch-0.3/package.mk
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/branch-0.3/package.mk?rev=1326652&r1=1326651&r2=1326652&view=diff
==============================================================================
--- incubator/bigtop/branches/branch-0.3/package.mk (original)
+++ incubator/bigtop/branches/branch-0.3/package.mk Mon Apr 16 15:36:06 2012
@@ -141,7 +141,11 @@ $(2)_SOURCE_DIR       = $$($(2)_BUILD_DI
 $(2)_DOWNLOAD_URL = $($(2)_SITE)/$($(2)_TARBALL_SRC)
 $(2)_DOWNLOAD_DST = $(DL_DIR)/$($(2)_TARBALL_DST)
 
-# Define the file stamps
+# test that the download url will return http 200.  If it does not, use the ARCHIVE url instead of the MIRROR SITE url
+ifneq ($$(shell curl -o /dev/null --silent --head --write-out '%{http_code}' $$($(2)_DOWNLOAD_URL)),200)
+	$(2)_DOWNLOAD_URL = $($(2)_ARCHIVE)/$($(2)_TARBALL_SRC)
+endif 
+
 $(2)_TARGET_DL       = $$($(2)_BUILD_DIR)/.download
 $(2)_TARGET_SRPM     = $$($(2)_BUILD_DIR)/.srpm
 $(2)_TARGET_RPM      = $$($(2)_BUILD_DIR)/.rpm



Mime
View raw message