Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8B385D681 for ; Mon, 30 Jul 2012 13:53:31 +0000 (UTC) Received: (qmail 79022 invoked by uid 500); 30 Jul 2012 13:53:31 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 78984 invoked by uid 500); 30 Jul 2012 13:53:31 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 78974 invoked by uid 99); 30 Jul 2012 13:53:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Jul 2012 13:53:31 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mardan8310@gmail.com designates 209.85.213.179 as permitted sender) Received: from [209.85.213.179] (HELO mail-yx0-f179.google.com) (209.85.213.179) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Jul 2012 13:53:24 +0000 Received: by yenr13 with SMTP id r13so4436774yen.38 for ; Mon, 30 Jul 2012 06:53:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=JVQ7hPQ8nddlvOsr59knMxMs1K8drWdC7rXqwYWFwJ4=; b=MOQpGE6LaL8ZQDgf7BNpvHpqs9U+Y+9r2NIhjyQfiKkU+SnTNsMWdu6JwzCrVZz7uC XArCaR2os27jbjPWGGPiXaBecmRxrhPjIP+PmVRVuk0IlGhbf+3FUCHn8405LZQ4v0GF gmHg2t3MhP1sOpusCMv+bOJ8diElKPh+Lhkqow/bOMgzQ69rvnc2kxgw8mHsCuYc/gA0 dNoP7iVNvyOcP7CuV43DjucmcwdTdiRmnnUxo1qSMIbCaaOBnUFdQyF0KK4/EIlZxUD2 eR6CgiXtNPZG87Kmc7hWpvKBYCVjjz64mLNrI7uuzUkf2d8RI+8TjbPzocnJVE9w9Y/g n0FQ== MIME-Version: 1.0 Received: by 10.42.19.2 with SMTP id z2mr6737416ica.33.1343656383233; Mon, 30 Jul 2012 06:53:03 -0700 (PDT) Received: by 10.42.130.193 with HTTP; Mon, 30 Jul 2012 06:53:03 -0700 (PDT) In-Reply-To: References: Date: Mon, 30 Jul 2012 14:53:03 +0100 Message-ID: Subject: Re: Installation of full flume on window OS From: mardan Khan To: user@flume.apache.org Content-Type: multipart/alternative; boundary=20cf3040ee5c6c3eeb04c60c618e --20cf3040ee5c6c3eeb04c60c618e Content-Type: text/plain; charset=ISO-8859-1 Thanks Inder, when I am running the $echo $CLASSPATH, I am just getting . (dot) and nothing more. Please see below the flume-ng files. Can you just pointout where I should put echo$CLASSPATH. Thanks CLEAN_FLAG=1 ################################ # functions ################################ info() { if [ ${CLEAN_FLAG} -ne 0 ]; then local msg=$1 echo "Info: $msg" >&2 fi } warn() { if [ ${CLEAN_FLAG} -ne 0 ]; then local msg=$1 echo "Warning: $msg" >&2 fi } error() { local msg=$1 local exit_code=$2 echo "Error: $msg" >&2 if [ -n "$exit_code" ] ; then exit $exit_code fi } # If avail, add Hadoop paths to the FLUME_CLASSPATH and to the # FLUME_JAVA_LIBRARY_PATH env vars. # Requires Flume jars to already be on FLUME_CLASSPATH. add_hadoop_paths() { local HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" \ which hadoop 2>/dev/null) if [ -f "${HADOOP_IN_PATH}" ]; then info "Including Hadoop libraries found via ($HADOOP_IN_PATH) for HDFS access" # determine hadoop java.library.path and use that for flume local HADOOP_CLASSPATH="" local HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$FLUME_CLASSPATH" \ ${HADOOP_IN_PATH} org.apache.flume.tools.GetJavaProperty \ java.library.path 2>/dev/null) # look for the line that has the desired property value # (considering extraneous output from some GC options that write to stdout) # IFS = InternalFieldSeparator (set to recognize only newline char as delimiter) IFS=$'\n' for line in $HADOOP_JAVA_LIBRARY_PATH; do if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then HADOOP_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]} break fi done unset IFS if [ -n "${HADOOP_JAVA_LIBRARY_PATH}" ]; then FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HADOOP_JAVA_LIBRARY_PATH" fi # determine hadoop classpath HADOOP_CLASSPATH=$($HADOOP_IN_PATH classpath 2>/dev/null) # hack up and filter hadoop classpath local ELEMENTS=$(sed -e 's/:/ /g' <<<${HADOOP_CLASSPATH}) local ELEMENT for ELEMENT in $ELEMENTS; do local PIECE for PIECE in $(echo $ELEMENT); do if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then info "Excluding $PIECE from classpath" continue else FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE" fi done done fi } add_HBASE_paths() { local HBASE_IN_PATH=$(PATH="${HBASE_HOME}/bin:$PATH" \ which hbase 2>/dev/null) if [ -f "${HBASE_IN_PATH}" ]; then info "Including HBASE libraries found via ($HBASE_IN_PATH) for HBASE access" # determine HBASE java.library.path and use that for flume local HBASE_CLASSPATH="" local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \ ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \ java.library.path 2>/dev/null) # look for the line that has the desired property value # (considering extraneous output from some GC options that write to stdout) # IFS = InternalFieldSeparator (set to recognize only newline char as delimiter) IFS=$'\n' for line in $HBASE_JAVA_LIBRARY_PATH; do if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then HBASE_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]} break fi done unset IFS if [ -n "${HBASE_JAVA_LIBRARY_PATH}" ]; then FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HBASE_JAVA_LIBRARY_PATH" fi # determine HBASE classpath HBASE_CLASSPATH=$($HBASE_IN_PATH classpath 2>/dev/null) # hack up and filter HBASE classpath local ELEMENTS=$(sed -e 's/:/ /g' <<<${HBASE_CLASSPATH}) local ELEMENT for ELEMENT in $ELEMENTS; do local PIECE for PIECE in $(echo $ELEMENT); do if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then info "Excluding $PIECE from classpath" continue else FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE" fi done done FLUME_CLASSPATH="$FLUME_CLASSPATH:$HBASE_HOME/conf" fi } display_help() { cat < [options]... commands: help display this help text agent run a Flume agent avro-client run an avro Flume client version show Flume version info global options: --conf,-c use configs in directory --classpath,-C append to the classpath --dryrun,-d do not actually start Flume, just print the command -Dproperty=value sets a JDK system property value agent options: --conf-file,-f specify a config file (required) --name,-n the name of this agent (required) --help,-h display help text avro-client options: --host,-H hostname to which events will be sent (required) --port,-p port of the avro source (required) --filename,-F text file to stream to avro source [default: std input] --headerFile,-R headerFile containing headers as key/value pairs on each new line --help,-h display help text Note that if directory is specified, then it is always included first in the classpath. EOF } run_flume() { local FLUME_APPLICATION_CLASS if [ "$#" -gt 0 ]; then FLUME_APPLICATION_CLASS=$1 shift else error "Must specify flume application class" 1 fi if [ ${CLEAN_FLAG} -ne 0 ]; then set -x fi $EXEC $JAVA_HOME/bin/java $JAVA_OPTS -cp "$FLUME_CLASSPATH" \ -Djava.library.path=$FLUME_JAVA_LIBRARY_PATH "$FLUME_APPLICATION_CLASS" $* } ################################ # main ################################ # set default params FLUME_CLASSPATH="" FLUME_JAVA_LIBRARY_PATH="" JAVA_OPTS="-Xmx20m" opt_conf="" opt_classpath="" opt_java_props="" opt_dryrun="" mode=$1 shift case "$mode" in help) display_help exit 0 ;; agent) opt_agent=1 ;; node) opt_agent=1 warn "The \"node\" command is deprecated. Please use \"agent\" instead." ;; avro-client) opt_avro_client=1 ;; version) opt_version=1 CLEAN_FLAG=0 ;; *) error "Unknown or unspecified command '$mode'" echo display_help exit 1 ;; esac while [ -n "$*" ] ; do arg=$1 shift case "$arg" in --conf|-c) [ -n "$1" ] || error "Option --conf requires an argument" 1 opt_conf=$1 shift ;; --classpath|-C) [ -n "$1" ] || error "Option --classpath requires an argument" 1 opt_classpath=$1 shift ;; --dryrun|-d) opt_dryrun="1" ;; -D*) opt_java_props="$opt_java_props $arg" ;; *) args="$args $arg" ;; esac done # make opt_conf absolute if [[ -n "$opt_conf" && -d "$opt_conf" ]]; then opt_conf=$(cd $opt_conf; pwd) fi # allow users to override the default env vars via conf/flume-env.sh if [ -z "$opt_conf" ]; then warn "No configuration directory set! Use --conf to override." elif [ -f "$opt_conf/flume-env.sh" ]; then info "Sourcing environment configuration script $opt_conf/flume-env.sh" source "$opt_conf/flume-env.sh" fi # append command-line java options to stock or env script JAVA_OPTS if [ -n "${opt_java_props}" ]; then JAVA_OPTS="${JAVA_OPTS} ${opt_java_props}" fi # prepend command-line classpath to env script classpath if [ -n "${opt_classpath}" ]; then if [ -n "${FLUME_CLASSPATH}" ]; then FLUME_CLASSPATH="${opt_classpath}:${FLUME_CLASSPATH}" else FLUME_CLASSPATH="${opt_classpath}" fi fi if [ -z "${FLUME_HOME}" ]; then FLUME_HOME=$(cd $(dirname $0)/..; pwd) fi # prepend $FLUME_HOME/lib jars to the specified classpath (if any) if [ -n "${FLUME_CLASSPATH}" ] ; then FLUME_CLASSPATH="${FLUME_HOME}/lib/*:$FLUME_CLASSPATH" else FLUME_CLASSPATH="${FLUME_HOME}/lib/*" fi # find java if [ -z "${JAVA_HOME}" ] ; then warn "JAVA_HOME is not set!" # Try to use Bigtop to autodetect JAVA_HOME if it's available if [ -e /usr/libexec/bigtop-detect-javahome ] ; then . /usr/libexec/bigtop-detect-javahome elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ] ; then . /usr/lib/bigtop-utils/bigtop-detect-javahome fi # Using java from path if bigtop is not installed or couldn't find it if [ -z "${JAVA_HOME}" ] ; then JAVA_DEFAULT=$(type -p java) [ -n "$JAVA_DEFAULT" ] || error "Unable to find java executable. Is it in your PATH?" 1 JAVA_HOME=$(cd $(dirname $JAVA_DEFAULT)/..; pwd) fi fi # look for hadoop libs add_hadoop_paths add_HBASE_paths # prepend conf dir to classpath if [ -n "$opt_conf" ]; then FLUME_CLASSPATH="$opt_conf:$FLUME_CLASSPATH" fi # allow dryrun EXEC="exec" if [ -n "${opt_dryrun}" ]; then warn "Dryrun mode enabled (will not actually initiate startup)" EXEC="echo" fi # finally, invoke the appropriate command if [ -n "$opt_agent" ] ; then run_flume $FLUME_AGENT_CLASS $args elif [ -n "$opt_avro_client" ] ; then run_flume $FLUME_AVRO_CLIENT_CLASS $args elif [ -n "${opt_version}" ] ; then run_flume $FLUME_VERSION_CLASS $args else error "This message should never appear" 1 fi exit 0 On Mon, Jul 30, 2012 at 2:40 PM, Inder Pall wrote: > Hey if you edit flume-ng script and just before java is invoked put an > > echo $CLASSPATH that should help in figuring what all dir's are there in > CLASSPATH. > I am sorry but i am a big fan of windows hence don't run that O.S. to try > it out. > > Thanks, > - inder > > On Mon, Jul 30, 2012 at 7:00 PM, mardan Khan wrote: > >> HI Inder, >> >> Sorry for the previous email. Actually I past the snapshot directly into >> editor which convert into some noisy text. Please see the snapshot as >> attachment . >> >> Please let me know for any solution >> >> Thanks >> >> >> >> >> On Mon, Jul 30, 2012 at 1:20 PM, Inder Pall wrote: >> >>> can you echo your CLASSPATH in the flume startup script. >>> >>> - inder >>> >>> >>> On Mon, Jul 30, 2012 at 4:25 PM, mardan Khan wrote: >>> >>>> Hi Brock >>>> >>>> Thanks for reply. I have solved the problem of cygwin. Now i am getting >>>> another error message: Could not find or load main class of >>>> org.apache.flume.node.Application. >>>> >>>> I am running the following command: >>>> >>>> bin/flume-ng agent -n agent -c conf -f >>>> conf/flume-conf.properties.template >>>> >>>> Thanks >>>> >>>> >>>> >>>> On Sun, Jul 29, 2012 at 6:03 PM, Brock Noland wrote: >>>> >>>>> Hi, >>>>> >>>>> Unfortunately I am not familiar with cygwin and don't have a windows >>>>> computer to test on. Hopefully someone else on the list has an insight. >>>>> >>>>> Brock >>>>> >>>>> >>>>> On Sun, Jul 29, 2012 at 11:27 AM, mardan Khan wrote: >>>>> >>>>>> Thanks Brock. >>>>>> >>>>>> >>>>>> Now when i am run the command: flume$ bin/flume-ng agent -n agent -f >>>>>> conf/flume-conf.properties. >>>>>> template. >>>>>> >>>>>> >>>>>> Cygwin Warning: MS-DOS style path detected >>>>>> /usr/local/bin/c:\program...................................................................... >>>>>> bin/flume-ng line:210 exec c:\program not found. >>>>>> >>>>>> How to solve this problem. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Jul 29, 2012 at 4:41 PM, Brock Noland wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Inline >>>>>>> >>>>>>> On Sat, Jul 28, 2012 at 11:54 PM, mardan Khan wrote: >>>>>>> >>>>>>>> Hi Brock. >>>>>>>> >>>>>>>> Thanks for reply. I want to use the full flume as i have to upload >>>>>>>> the data into hdfs. I have installed the cygwin and hadoop. >>>>>>>> >>>>>>> >>>>>>> OK this is a great start. >>>>>>> >>>>>>> >>>>>>>> I have download the flume binary from the mentioned link. Now you >>>>>>>> please let me explain the follow: >>>>>>>> >>>>>>>> 1). I need just to decopress the downloaded folder? >>>>>>>> >>>>>>> >>>>>>> Yes you download a tar.gz which is like a zip. In cygwin you can use >>>>>>> the command tar -zxvf to untar it. >>>>>>> >>>>>>> >>>>>>>> 2). where i should decompress the flume binary. Should i decompress >>>>>>>> c:\cygwin\home\user-name >>>>>>>> >>>>>>> >>>>>>> Whereever you want, however a common place for "tarball" software is >>>>>>> /usr/local or /opt >>>>>>> >>>>>>> >>>>>>>> 3). How I will run the flume-ng. >>>>>>>> >>>>>>> >>>>>>> You configure flume and then run the flume-ng command in the bin >>>>>>> directory >>>>>>> >>>>>>> >>>>>>>> 4) The flume.conf file will configure same as configuring using >>>>>>>> Linux? >>>>>>>> >>>>>>>> >>>>>>> Yes configure flume will be the same as on Linux in cygwin. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Many thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Jul 29, 2012 at 4:20 AM, Brock Noland wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> If you just want run a flume agent on Windows you just download >>>>>>>>> the binary here: http://flume.apache.org/download.html >>>>>>>>> >>>>>>>>> Note that the flume startup script is written in bash so unless >>>>>>>>> you use cygwin you'll need to start the JVM and setup the classpath >>>>>>>>> yourself. >>>>>>>>> >>>>>>>>> Brock >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Jul 28, 2012 at 8:01 PM, mardan Khan >>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I am struggling from couple of weeks to install and configure the >>>>>>>>>> flume1.x on windows but could not success. Could you please someone show me >>>>>>>>>> step-by-step how to install flume on window. I have read a couple of posts >>>>>>>>>> which mentioned the git and Maven but i dont understand how can use it to >>>>>>>>>> install the flume. >>>>>>>>>> >>>>>>>>>> Please help me as i am total stuck and running out of time. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Apache MRUnit - Unit testing MapReduce - >>>>>>>>> http://incubator.apache.org/mrunit/ >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Apache MRUnit - Unit testing MapReduce - >>>>>>> http://incubator.apache.org/mrunit/ >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Apache MRUnit - Unit testing MapReduce - >>>>> http://incubator.apache.org/mrunit/ >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> - Inder >>> Tech Platforms @Inmobi >>> Linkedin - http://goo.gl/eR4Ub >>> >> >> > > > -- > Thanks, > - Inder > Tech Platforms @Inmobi > Linkedin - http://goo.gl/eR4Ub > --20cf3040ee5c6c3eeb04c60c618e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Inder,

when I am running the $echo $CLASSPATH, I am just gett= ing . (dot) and nothing more.

Please see below the flume-ng files. = Can you just pointout where I should put echo$CLASSPATH.

Thanks


CLEAN_FLAG=3D1
################################
# functions################################

info() {
=A0 if [ ${CLEAN_FLAG= } -ne 0 ]; then
=A0=A0=A0 local msg=3D$1
=A0=A0=A0 echo "Info: $= msg" >&2
=A0 fi
}

warn() {
=A0 if [ ${CLEAN_FLAG} -ne 0 ]; then
=A0= =A0=A0 local msg=3D$1
=A0=A0=A0 echo "Warning: $msg" >&= 2
=A0 fi
}

error() {
=A0 local msg=3D$1
=A0 local exit_c= ode=3D$2

=A0 echo "Error: $msg" >&2

=A0 if [ -n "$exit_code" ] ; then
=A0=A0=A0 exit $exit_cod= e
=A0 fi
}

# If avail, add Hadoop paths to the FLUME_CLASSPATH= and to the
# FLUME_JAVA_LIBRARY_PATH env vars.
# Requires Flume jars= to already be on FLUME_CLASSPATH.
add_hadoop_paths() {
=A0 local HADOOP_IN_PATH=3D$(PATH=3D"${HADOOP_= HOME:-${HADOOP_PREFIX}}/bin:$PATH" \
=A0=A0=A0=A0=A0 which hadoop 2= >/dev/null)

=A0 if [ -f "${HADOOP_IN_PATH}" ]; then
= =A0=A0=A0 info "Including Hadoop libraries found via ($HADOOP_IN_PATH)= for HDFS access"

=A0=A0=A0 # determine hadoop java.library.path and use that for flume=A0=A0=A0 local HADOOP_CLASSPATH=3D""
=A0=A0=A0 local HADOOP= _JAVA_LIBRARY_PATH=3D$(HADOOP_CLASSPATH=3D"$FLUME_CLASSPATH" \=A0=A0=A0=A0=A0=A0=A0 ${HADOOP_IN_PATH} org.apache.flume.tools.GetJavaProp= erty \
=A0=A0=A0=A0=A0=A0=A0 java.library.path 2>/dev/null)

=A0=A0=A0 # = look for the line that has the desired property value
=A0=A0=A0 # (consi= dering extraneous output from some GC options that write to stdout)
=A0= =A0=A0 # IFS =3D InternalFieldSeparator (set to recognize only newline char= as delimiter)
=A0=A0=A0 IFS=3D$'\n'
=A0=A0=A0 for line in $HADOOP_JAVA_LIBRARY= _PATH; do
=A0=A0=A0=A0=A0 if [[ $line =3D~ ^java\.library\.path=3D(.*)$ = ]]; then
=A0=A0=A0=A0=A0=A0=A0 HADOOP_JAVA_LIBRARY_PATH=3D${BASH_REMATCH= [1]}
=A0=A0=A0=A0=A0=A0=A0 break
=A0=A0=A0=A0=A0 fi
=A0=A0=A0 done=
=A0=A0=A0 unset IFS

=A0=A0=A0 if [ -n "${HADOOP_JAVA_LIBRARY_PA= TH}" ]; then
=A0=A0=A0=A0=A0 FLUME_JAVA_LIBRARY_PATH=3D"$FLUME= _JAVA_LIBRARY_PATH:$HADOOP_JAVA_LIBRARY_PATH"
=A0=A0=A0 fi

= =A0=A0=A0 # determine hadoop classpath
=A0=A0=A0 HADOOP_CLASSPATH=3D$($HADOOP_IN_PATH classpath 2>/dev/null)
=A0=A0=A0 # hack up and filter hadoop classpath
=A0=A0=A0 local ELE= MENTS=3D$(sed -e 's/:/ /g' <<<${HADOOP_CLASSPATH})
=A0= =A0=A0 local ELEMENT
=A0=A0=A0 for ELEMENT in $ELEMENTS; do
=A0=A0=A0=A0=A0 local PIECE
=A0=A0=A0=A0=A0 for PIECE in $(echo $ELEMENT= ); do
=A0=A0=A0=A0=A0=A0=A0 if [[ $PIECE =3D~ slf4j-(api|log4j12).*\.jar= ]]; then
=A0=A0=A0=A0=A0=A0=A0=A0=A0 info "Excluding $PIECE from c= lasspath"
=A0=A0=A0=A0=A0=A0=A0=A0=A0 continue
=A0=A0=A0=A0=A0= =A0=A0 else
=A0=A0=A0=A0=A0=A0=A0=A0=A0 FLUME_CLASSPATH=3D"$FLUME_CLASSPATH:$PIECE= "
=A0=A0=A0=A0=A0=A0=A0 fi
=A0=A0=A0=A0=A0 done
=A0=A0=A0 don= e

=A0 fi
}
add_HBASE_paths() {
=A0 local HBASE_IN_PATH=3D$(= PATH=3D"${HBASE_HOME}/bin:$PATH" \
=A0=A0=A0=A0=A0 which hbase= 2>/dev/null)

=A0 if [ -f "${HBASE_IN_PATH}" ]; then
=A0=A0=A0 info &quo= t;Including HBASE libraries found via ($HBASE_IN_PATH) for HBASE access&quo= t;

=A0=A0=A0 # determine HBASE java.library.path and use that for fl= ume
=A0=A0=A0 local HBASE_CLASSPATH=3D""
=A0=A0=A0 local HBASE_JAVA_LIBRARY_PATH=3D$(HBASE_CLASSPATH=3D"$FLUME_= CLASSPATH" \
=A0=A0=A0=A0=A0=A0=A0 ${HBASE_IN_PATH} org.apache.flum= e.tools.GetJavaProperty \
=A0=A0=A0=A0=A0=A0=A0 java.library.path 2>/= dev/null)

=A0=A0=A0 # look for the line that has the desired propert= y value
=A0=A0=A0 # (considering extraneous output from some GC options that write = to stdout)
=A0=A0=A0 # IFS =3D InternalFieldSeparator (set to recognize = only newline char as delimiter)
=A0=A0=A0 IFS=3D$'\n'
=A0=A0= =A0 for line in $HBASE_JAVA_LIBRARY_PATH; do
=A0=A0=A0=A0=A0 if [[ $line =3D~ ^java\.library\.path=3D(.*)$ ]]; then
= =A0=A0=A0=A0=A0=A0=A0 HBASE_JAVA_LIBRARY_PATH=3D${BASH_REMATCH[1]}
=A0= =A0=A0=A0=A0=A0=A0 break
=A0=A0=A0=A0=A0 fi
=A0=A0=A0 done
=A0=A0= =A0 unset IFS

=A0=A0=A0 if [ -n "${HBASE_JAVA_LIBRARY_PATH}&quo= t; ]; then
=A0=A0=A0=A0=A0 FLUME_JAVA_LIBRARY_PATH=3D"$FLUME_JAVA_LIBRARY_PATH:$H= BASE_JAVA_LIBRARY_PATH"
=A0=A0=A0 fi

=A0=A0=A0 # determine H= BASE classpath
=A0=A0=A0 HBASE_CLASSPATH=3D$($HBASE_IN_PATH classpath 2&= gt;/dev/null)

=A0=A0=A0 # hack up and filter HBASE classpath
=A0=A0=A0 local ELEMENTS=3D$(sed -e 's/:/ /g' <<<${HBASE_C= LASSPATH})
=A0=A0=A0 local ELEMENT
=A0=A0=A0 for ELEMENT in $ELEMENTS= ; do
=A0=A0=A0=A0=A0 local PIECE
=A0=A0=A0=A0=A0 for PIECE in $(echo = $ELEMENT); do
=A0=A0=A0=A0=A0=A0=A0 if [[ $PIECE =3D~ slf4j-(api|log4j12= ).*\.jar ]]; then
=A0=A0=A0=A0=A0=A0=A0=A0=A0 info "Excluding $PIECE from classpath"= ;
=A0=A0=A0=A0=A0=A0=A0=A0=A0 continue
=A0=A0=A0=A0=A0=A0=A0 else
= =A0=A0=A0=A0=A0=A0=A0=A0=A0 FLUME_CLASSPATH=3D"$FLUME_CLASSPATH:$PIECE= "
=A0=A0=A0=A0=A0=A0=A0 fi
=A0=A0=A0=A0=A0 done
=A0=A0=A0 don= e
=A0=A0=A0 FLUME_CLASSPATH=3D"$FLUME_CLASSPATH:$HBASE_HOME/conf&qu= ot;

=A0 fi
}

display_help() {
=A0 cat <<EOF
Usage: $0= <command> [options]...

commands:
=A0 help=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 display this help text
=A0 agent=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 run a Flume agent
=A0 avro= -client=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 run an avro Flume client
=A0 version=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 show Flume version in= fo

global options:
=A0 --conf,-c <conf>=A0=A0=A0=A0=A0 use = configs in <conf> directory
=A0 --classpath,-C <cp>=A0=A0 ap= pend to the classpath
=A0 --dryrun,-d=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 do n= ot actually start Flume, just print the command
=A0 -Dproperty=3Dvalue=A0=A0=A0=A0=A0 sets a JDK system property value
<= br>agent options:
=A0 --conf-file,-f <file> specify a config file = (required)
=A0 --name,-n <name>=A0=A0=A0=A0=A0 the name of this ag= ent (required)
=A0 --help,-h=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 display= help text

avro-client options:
=A0 --host,-H <host>=A0=A0=A0=A0=A0 hostn= ame to which events will be sent (required)
=A0 --port,-p <port>= =A0=A0=A0=A0=A0 port of the avro source (required)
=A0 --filename,-F <= ;file>=A0 text file to stream to avro source [default: std input]
=A0 --headerFile,-R <file> headerFile containing headers as key/value= pairs on each new line
=A0 --help,-h=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 display help text

Note that if <conf> directory is specifi= ed, then it is always included first
in the classpath.

EOF
}

run_flume() {
=A0 local FLUME_A= PPLICATION_CLASS

=A0 if [ "$#" -gt 0 ]; then
=A0=A0=A0 = FLUME_APPLICATION_CLASS=3D$1
=A0=A0=A0 shift
=A0 else
=A0=A0=A0 er= ror "Must specify flume application class" 1
=A0 fi

=A0 if [ ${CLEAN_FLAG} -ne 0 ]; then
=A0=A0=A0 set -x
= =A0 fi
=A0 $EXEC $JAVA_HOME/bin/java $JAVA_OPTS -cp "$FLUME_CLASSPA= TH" \
=A0=A0=A0=A0=A0 -Djava.library.path=3D$FLUME_JAVA_LIBRARY_PAT= H "$FLUME_APPLICATION_CLASS" $*
}

################################
# main
####################= ############

# set default params
FLUME_CLASSPATH=3D""<= br>FLUME_JAVA_LIBRARY_PATH=3D""
JAVA_OPTS=3D"-Xmx20m"= ;

opt_conf=3D""
opt_classpath=3D""
opt_java_pro= ps=3D""
opt_dryrun=3D""

mode=3D$1
shift
case "$mode" in
=A0 help)
=A0=A0=A0 display_help
= =A0=A0=A0 exit 0
=A0=A0=A0 ;;
=A0 agent)
=A0=A0=A0 opt_agent=3D1
=A0=A0=A0 ;;
=A0 node)
=A0= =A0=A0 opt_agent=3D1
=A0=A0=A0 warn "The \"node\" command= is deprecated. Please use \"agent\" instead."
=A0=A0=A0 = ;;
=A0 avro-client)
=A0=A0=A0 opt_avro_client=3D1
=A0=A0=A0 ;;
=A0 version)
=A0=A0 opt_version=3D1
=A0=A0 CLEAN_FLAG= =3D0
=A0=A0 ;;
=A0 *)
=A0=A0=A0 error "Unknown or unspecified= command '$mode'"
=A0=A0=A0 echo
=A0=A0=A0 display_help<= br>=A0=A0=A0 exit 1
=A0=A0=A0 ;;
esac

while [ -n "$*"= ; ] ; do
=A0 arg=3D$1
=A0 shift

=A0 case "$arg" in
=A0=A0=A0 = --conf|-c)
=A0=A0=A0=A0=A0 [ -n "$1" ] || error "Option -= -conf requires an argument" 1
=A0=A0=A0=A0=A0 opt_conf=3D$1
=A0= =A0=A0=A0=A0 shift
=A0=A0=A0=A0=A0 ;;
=A0=A0=A0 --classpath|-C)
=A0=A0=A0=A0=A0 [ -n "$1" ] || error "Option --classpath req= uires an argument" 1
=A0=A0=A0=A0=A0 opt_classpath=3D$1
=A0=A0= =A0=A0=A0 shift
=A0=A0=A0=A0=A0 ;;
=A0=A0=A0 --dryrun|-d)
=A0=A0= =A0=A0=A0 opt_dryrun=3D"1"
=A0=A0=A0=A0=A0 ;;
=A0=A0=A0 -D*= )
=A0=A0=A0=A0=A0 opt_java_props=3D"$opt_java_props $arg"
=A0=A0= =A0=A0=A0 ;;
=A0=A0=A0 *)
=A0=A0=A0=A0=A0 args=3D"$args $arg&quo= t;
=A0=A0=A0=A0=A0 ;;
=A0 esac
done

# make opt_conf absolut= e
if [[ -n "$opt_conf" && -d "$opt_conf" ]];= then
=A0 opt_conf=3D$(cd $opt_conf; pwd)
fi

# allow users to override = the default env vars via conf/flume-env.sh
if [ -z "$opt_conf"= ]; then
=A0 warn "No configuration directory set! Use --conf <d= ir> to override."
elif [ -f "$opt_conf/flume-env.sh" ]; then
=A0 info "Sour= cing environment configuration script $opt_conf/flume-env.sh"
=A0 s= ource "$opt_conf/flume-env.sh"
fi

# append command-line= java options to stock or env script JAVA_OPTS
if [ -n "${opt_java_props}" ]; then
=A0 JAVA_OPTS=3D"${JA= VA_OPTS} ${opt_java_props}"
fi

# prepend command-line classp= ath to env script classpath
if [ -n "${opt_classpath}" ]; then=
=A0 if [ -n "${FLUME_CLASSPATH}" ]; then
=A0=A0=A0 FLUME_CLASS= PATH=3D"${opt_classpath}:${FLUME_CLASSPATH}"
=A0 else
=A0= =A0=A0 FLUME_CLASSPATH=3D"${opt_classpath}"
=A0 fi
fi
if [ -z "${FLUME_HOME}" ]; then
=A0 FLUME_HOME=3D$(cd $(dirname $0)/..; pwd)
fi

# prepend $FLUME_= HOME/lib jars to the specified classpath (if any)
if [ -n "${FLUME_= CLASSPATH}" ] ; then
=A0 FLUME_CLASSPATH=3D"${FLUME_HOME}/lib/= *:$FLUME_CLASSPATH"
else
=A0 FLUME_CLASSPATH=3D"${FLUME_HOME}/lib/*"
fi

= # find java
if [ -z "${JAVA_HOME}" ] ; then
=A0 warn "= JAVA_HOME is not set!"
=A0 # Try to use Bigtop to autodetect JAVA_H= OME if it's available
=A0 if [ -e /usr/libexec/bigtop-detect-javahome ] ; then
=A0=A0=A0 . /us= r/libexec/bigtop-detect-javahome
=A0 elif [ -e /usr/lib/bigtop-utils/big= top-detect-javahome ] ; then
=A0=A0=A0 . /usr/lib/bigtop-utils/bigtop-de= tect-javahome
=A0 fi

=A0 # Using java from path if bigtop is not installed or coul= dn't find it
=A0 if [ -z "${JAVA_HOME}" ] ; then
=A0=A0= =A0 JAVA_DEFAULT=3D$(type -p java)
=A0=A0=A0 [ -n "$JAVA_DEFAULT&qu= ot; ] || error "Unable to find java executable. Is it in your PATH?&qu= ot; 1
=A0=A0=A0 JAVA_HOME=3D$(cd $(dirname $JAVA_DEFAULT)/..; pwd)
=A0 fi
f= i

# look for hadoop libs
add_hadoop_paths
add_HBASE_paths
<= br># prepend conf dir to classpath
if [ -n "$opt_conf" ]; then=
=A0 FLUME_CLASSPATH=3D"$opt_conf:$FLUME_CLASSPATH"
fi

# allow dryrun
EXEC=3D"exec"
if [ -n "${opt_= dryrun}" ]; then
=A0 warn "Dryrun mode enabled (will not actua= lly initiate startup)"
=A0 EXEC=3D"echo"
fi

# f= inally, invoke the appropriate command
if [ -n "$opt_agent" ] ; then
=A0 run_flume $FLUME_AGENT_CLASS= $args
elif [ -n "$opt_avro_client" ] ; then
=A0 run_flume = $FLUME_AVRO_CLIENT_CLASS $args
elif [ -n "${opt_version}" ] ; = then
=A0 run_flume $FLUME_VERSION_CLASS $args
else
=A0 error "This me= ssage should never appear" 1
fi

exit 0






On Mon, Jul 30, 2012 at 2:40 PM, Inder= Pall <inder.pall@gmail.com> wrote:
Hey if you edit flume-ng script and just bef= ore java is invoked put an

echo $CLASSPATH that should h= elp in figuring what all dir's are there in CLASSPATH.
I am sorry but i am a big fan of windows hence don't run that O.S.= to try it out.

Thanks,
- inder

On Mon, Jul 30, 2012 at= 7:00 PM, mardan Khan <mardan8310@gmail.com> wrote:
HI Inder,

Sorry for the previous emai= l. Actually I past the snapshot directly into editor which convert into som= e noisy text. Please see the snapshot as attachment .

Please let me know for any solution

Thanks


=A0

On Mon, Jul = 30, 2012 at 1:20 PM, Inder Pall <inder.pall@gmail.com> wr= ote:
can you echo your CLASSPATH in the flume startup script.

- inder


On Mon, Jul 30, 2012 a= t 4:25 PM, mardan Khan <mardan8310@gmail.com> wrote:
Hi Brock

Thanks for reply. I have sol= ved the problem of cygwin. Now i am getting another error message: Could no= t find or load main class of org.apache.flume.node.Application.

I am running the following command:

bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties.templ= ate

Thanks



On Sun, J= ul 29, 2012 at 6:03 PM, Brock Noland <brock@cloudera.com> w= rote:
Hi,

Unfortunately=A0I am = not=A0familiar=A0with cygwin and don't have a windows computer to test = on. Hopefully someone else on the list has an insight.

Brock


On Sun, Jul 29, 2012 at 11:27 AM, mardan Khan <mardan8310@gmail.com= > wrote:
Thanks Brock.


Now when i am run the command:=A0 flume$ bin/flume= -ng agent -n agent -f conf/flume-conf.properties.
template.


Cygwin Warning: MS-DOS style path det= ected /usr/local/bin/c:\program............................................= ..........................
bin/flume-ng line:210 exec c:\program not found.

How to solve = this problem.








On Sun, Jul 29, 2012 at 4:41 PM, Brock Noland <= brock@cloudera.com<= /a>> wrote:
Hi,

Inline

On Sat, Jul 28, 2012 at 11:54 PM, mardan Khan= <mardan8310@gmail.com> wrote:
Hi Brock.
=A0
Thank= s for reply. I want to use the full flume as i have to upload the data into= hdfs. I have installed the cygwin and hadoop.
=A0
OK this is a great start.
<= div>
=A0
I have down= load the flume binary from the mentioned link. Now you please let me explai= n the follow:
=A0
1). I need just to decopress the downloaded folder?

Yes you download a tar.gz which is = like a zip. In cygwin you can use the command tar -zxvf to untar it.
=A0
2). where i should decompress = the flume binary. Should i decompress c:\cygwin\home\user-name

Whereever you want, however a common place for &q= uot;tarball" software is /usr/local or /opt
=A0
3). How I will run the flume-ng.

You configure flume and then run the flume-ng command in the bin dir= ectory
=A0
4) The flume.conf file will configure same as configuring using Linux?
=A0
Yes configure flume will be the same= as on Linux in cygwin.=A0
=A0
=A0
Many thanks
=A0
<= div>=A0

On Sun, Jul 29, 2012 at 4:2= 0 AM, Brock Noland <brock@cloudera.com> wrote:
Hi,

If you just want run a flume agent = on Windows you just download the binary here:=A0http://flume.apache.org/download.h= tml

Note that the flume startup script is written in bash s= o unless you use cygwin you'll need to start the JVM and setup the clas= spath yourself.

Brock


On Sa= t, Jul 28, 2012 at 8:01 PM, mardan Khan <mardan8310@gmail.com> wrote:
Hi,
=A0
I am struggling from coup= le of weeks to install and configure the flume1.x on windows but could not = success. Could you=A0please someone show me step-by-step how to install flu= me on window. I have read a couple of posts which mentioned the git and Mav= en but i dont understand how can use it to install the flume.
=A0
Please help me as i am total stuck and running out of ti= me.
=A0
Thanks



<= font color=3D"#888888">--
Apache MRUnit - Unit testing MapReduce - http://incuba= tor.apache.org/mrunit/




--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrun= it/




--
= Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/




<= /div>--
Thanks,
- Inder
=A0 Tech Platforms @Inmobi
=A0 Linkedin - http://goo.gl/= eR4Ub



--
= Thanks,
- Inder
=A0 Tech Platforms @Inmobi
=A0 Linkedin - http://goo.gl/= eR4Ub

--20cf3040ee5c6c3eeb04c60c618e--