flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@gmail.com>
Subject Re: Installation of full flume on window OS
Date Mon, 30 Jul 2012 14:58:11 GMT
Ah, okay.

For non-cygwin:

You have done git-checkout and build flume successfully (mvn -DSkipTests)?

After you've build you've copied the binaries, located in flume's source 
directory 
(flume-ng-dist\target\flume-ng-dist-1.3.0-SNAPSHOT-dist.tar.gz) onto 
another directory and unpacked them? And that directory is apache-flume?

cheers,
  Alex



> mardan Khan <mailto:mardan8310@gmail.com>
> July 30, 2012 4:44 PM
> HI,
>
> Actually i am doing conversation with Inder regarding the Cygwin. As I 
> am trying to run on cygwin and with you on MS-DOS command prompt.
>
> I am just telling you important location
>
>
> 1). My flume installation directory: D:\apache-flume
> 2). Lib path:                                 D:\apache-flume\lib
> 3). Configuration File:                    
> D:\apache-flume\conf\flume-conf.properties.template (I have copied the 
> configuration file from your previous post).
> 4). My Java path:                          D:\program 
> files\java\jdk1.7.0_03
>
> My Command:
>
> D:\apache-flume>"c:\Program Files\Java\jdk1.7.0_03\bin\java.exe" 
> -Xmx20m -Dlog4j
>
> .configuration=file:///%CD%\conf\log4j.properties -cp 
> "d:\apache-flume\lib\*" org.apache.flume.node.Application
>
> -f d:\apache-flume\conf\flume-conf.properties.template -n syslog-agent
>
>
>
> Error: Could not find or load main class 
> d:\apache-flume\lib\asynchbase-1.2.0.jar
>
>
>
> Now please let me know where I am doing mistake.
>
>
>
>
> alo alt <mailto:wget.null@gmail.com>
> July 30, 2012 4:20 PM
> I think we have two different conversations about the same topic. I 
> reply here.
>
> Thats not an issue of a env script in windows, because we haven't one 
> so far. You may need to cp (Classpath) the flume - libs as Inder says.
> As I wrote in my howto:
>
> As example, starting from inside of your flume-installation (the 
> directory where you unzip'ed the build), lets assume D:\apache-flume:
> d:\apache flume > "PATH\to\java.exe" -Xmx20m 
> -Dlog4j.configuration=file:///%CD%\conf\log4j.properties -cp 
> "d:\apache-flume\flume-1.3.0-SNAPSHOT\lib\*" 
> org.apache.flume.node.Application -f 
> d:\flume\flume-1.3.0-SNAPSHOT\conf\YOUR_CONFIG.conf -n agent
>
> Please be sure, that the lib-directory from flume is in place as well 
> in the CLASSPATH (cp).
>
> - Alex
> Inder Pall <mailto:inder.pall@gmail.com>
> July 30, 2012 4:02 PM
> That's the problem. I am not well versed with this script to fix on 
> WIN but i guess as a workaround you might want to add 
> $FLUME_INSTALL_DIR/lib/. this lib has all the required jars and should 
> get you moving forward.
>
> Mind, filing a JIRA  for the same.
>
> - inder
>
>
>
>
> -- 
> Thanks,
> - Inder
>   Tech Platforms @Inmobi
>   Linkedin - http://goo.gl/eR4Ub
> mardan Khan <mailto:mardan8310@gmail.com>
> July 30, 2012 3:53 PM
> Thanks Inder,
>
> when I am running the $echo $CLASSPATH, I am just getting . (dot) and 
> nothing more.
>
> Please see below the flume-ng files. Can you just pointout where I 
> should put echo$CLASSPATH.
>
> Thanks
>
>
> CLEAN_FLAG=1
> ################################
> # functions
> ################################
>
> info() {
>   if [ ${CLEAN_FLAG} -ne 0 ]; then
>     local msg=$1
>     echo "Info: $msg" >&2
>   fi
> }
>
> warn() {
>   if [ ${CLEAN_FLAG} -ne 0 ]; then
>     local msg=$1
>     echo "Warning: $msg" >&2
>   fi
> }
>
> error() {
>   local msg=$1
>   local exit_code=$2
>
>   echo "Error: $msg" >&2
>
>   if [ -n "$exit_code" ] ; then
>     exit $exit_code
>   fi
> }
>
> # If avail, add Hadoop paths to the FLUME_CLASSPATH and to the
> # FLUME_JAVA_LIBRARY_PATH env vars.
> # Requires Flume jars to already be on FLUME_CLASSPATH.
> add_hadoop_paths() {
>   local 
> HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" \
>       which hadoop 2>/dev/null)
>
>   if [ -f "${HADOOP_IN_PATH}" ]; then
>     info "Including Hadoop libraries found via ($HADOOP_IN_PATH) for 
> HDFS access"
>
>     # determine hadoop java.library.path and use that for flume
>     local HADOOP_CLASSPATH=""
>     local HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$FLUME_CLASSPATH" \
>         ${HADOOP_IN_PATH} org.apache.flume.tools.GetJavaProperty \
>         java.library.path 2>/dev/null)
>
>     # look for the line that has the desired property value
>     # (considering extraneous output from some GC options that write 
> to stdout)
>     # IFS = InternalFieldSeparator (set to recognize only newline char 
> as delimiter)
>     IFS=$'\n'
>     for line in $HADOOP_JAVA_LIBRARY_PATH; do
>       if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then
>         HADOOP_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]}
>         break
>       fi
>     done
>     unset IFS
>
>     if [ -n "${HADOOP_JAVA_LIBRARY_PATH}" ]; then
>       
> FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HADOOP_JAVA_LIBRARY_PATH"
>     fi
>
>     # determine hadoop classpath
>     HADOOP_CLASSPATH=$($HADOOP_IN_PATH classpath 2>/dev/null)
>
>     # hack up and filter hadoop classpath
>     local ELEMENTS=$(sed -e 's/:/ /g' <<<${HADOOP_CLASSPATH})
>     local ELEMENT
>     for ELEMENT in $ELEMENTS; do
>       local PIECE
>       for PIECE in $(echo $ELEMENT); do
>         if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then
>           info "Excluding $PIECE from classpath"
>           continue
>         else
>           FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE"
>         fi
>       done
>     done
>
>   fi
> }
> add_HBASE_paths() {
>   local HBASE_IN_PATH=$(PATH="${HBASE_HOME}/bin:$PATH" \
>       which hbase 2>/dev/null)
>
>   if [ -f "${HBASE_IN_PATH}" ]; then
>     info "Including HBASE libraries found via ($HBASE_IN_PATH) for 
> HBASE access"
>
>     # determine HBASE java.library.path and use that for flume
>     local HBASE_CLASSPATH=""
>     local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
>         ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
>         java.library.path 2>/dev/null)
>
>     # look for the line that has the desired property value
>     # (considering extraneous output from some GC options that write 
> to stdout)
>     # IFS = InternalFieldSeparator (set to recognize only newline char 
> as delimiter)
>     IFS=$'\n'
>     for line in $HBASE_JAVA_LIBRARY_PATH; do
>       if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then
>         HBASE_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]}
>         break
>       fi
>     done
>     unset IFS
>
>     if [ -n "${HBASE_JAVA_LIBRARY_PATH}" ]; then
>       
> FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HBASE_JAVA_LIBRARY_PATH"
>     fi
>
>     # determine HBASE classpath
>     HBASE_CLASSPATH=$($HBASE_IN_PATH classpath 2>/dev/null)
>
>     # hack up and filter HBASE classpath
>     local ELEMENTS=$(sed -e 's/:/ /g' <<<${HBASE_CLASSPATH})
>     local ELEMENT
>     for ELEMENT in $ELEMENTS; do
>       local PIECE
>       for PIECE in $(echo $ELEMENT); do
>         if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then
>           info "Excluding $PIECE from classpath"
>           continue
>         else
>           FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE"
>         fi
>       done
>     done
>     FLUME_CLASSPATH="$FLUME_CLASSPATH:$HBASE_HOME/conf"
>
>   fi
> }
>
> display_help() {
>   cat <<EOF
> Usage: $0 <command> [options]...
>
> commands:
>   help                  display this help text
>   agent                 run a Flume agent
>   avro-client           run an avro Flume client
>   version               show Flume version info
>
> global options:
>   --conf,-c <conf>      use configs in <conf> directory
>   --classpath,-C <cp>   append to the classpath
>   --dryrun,-d           do not actually start Flume, just print the 
> command
>   -Dproperty=value      sets a JDK system property value
>
> agent options:
>   --conf-file,-f <file> specify a config file (required)
>   --name,-n <name>      the name of this agent (required)
>   --help,-h             display help text
>
> avro-client options:
>   --host,-H <host>      hostname to which events will be sent (required)
>   --port,-p <port>      port of the avro source (required)
>   --filename,-F <file>  text file to stream to avro source [default: 
> std input]
>   --headerFile,-R <file> headerFile containing headers as key/value 
> pairs on each new line
>   --help,-h             display help text
>
> Note that if <conf> directory is specified, then it is always included 
> first
> in the classpath.
>
> EOF
> }
>
> run_flume() {
>   local FLUME_APPLICATION_CLASS
>
>   if [ "$#" -gt 0 ]; then
>     FLUME_APPLICATION_CLASS=$1
>     shift
>   else
>     error "Must specify flume application class" 1
>   fi
>
>   if [ ${CLEAN_FLAG} -ne 0 ]; then
>     set -x
>   fi
>   $EXEC $JAVA_HOME/bin/java $JAVA_OPTS -cp "$FLUME_CLASSPATH" \
>       -Djava.library.path=$FLUME_JAVA_LIBRARY_PATH 
> "$FLUME_APPLICATION_CLASS" $*
> }
>
> ################################
> # main
> ################################
>
> # set default params
> FLUME_CLASSPATH=""
> FLUME_JAVA_LIBRARY_PATH=""
> JAVA_OPTS="-Xmx20m"
>
> opt_conf=""
> opt_classpath=""
> opt_java_props=""
> opt_dryrun=""
>
> mode=$1
> shift
>
> case "$mode" in
>   help)
>     display_help
>     exit 0
>     ;;
>   agent)
>     opt_agent=1
>     ;;
>   node)
>     opt_agent=1
>     warn "The \"node\" command is deprecated. Please use \"agent\" 
> instead."
>     ;;
>   avro-client)
>     opt_avro_client=1
>     ;;
>   version)
>    opt_version=1
>    CLEAN_FLAG=0
>    ;;
>   *)
>     error "Unknown or unspecified command '$mode'"
>     echo
>     display_help
>     exit 1
>     ;;
> esac
>
> while [ -n "$*" ] ; do
>   arg=$1
>   shift
>
>   case "$arg" in
>     --conf|-c)
>       [ -n "$1" ] || error "Option --conf requires an argument" 1
>       opt_conf=$1
>       shift
>       ;;
>     --classpath|-C)
>       [ -n "$1" ] || error "Option --classpath requires an argument" 1
>       opt_classpath=$1
>       shift
>       ;;
>     --dryrun|-d)
>       opt_dryrun="1"
>       ;;
>     -D*)
>       opt_java_props="$opt_java_props $arg"
>       ;;
>     *)
>       args="$args $arg"
>       ;;
>   esac
> done
>
> # make opt_conf absolute
> if [[ -n "$opt_conf" && -d "$opt_conf" ]]; then
>   opt_conf=$(cd $opt_conf; pwd)
> fi
>
> # allow users to override the default env vars via conf/flume-env.sh
> if [ -z "$opt_conf" ]; then
>   warn "No configuration directory set! Use --conf <dir> to override."
> elif [ -f "$opt_conf/flume-env.sh" ]; then
>   info "Sourcing environment configuration script $opt_conf/flume-env.sh"
>   source "$opt_conf/flume-env.sh"
> fi
>
> # append command-line java options to stock or env script JAVA_OPTS
> if [ -n "${opt_java_props}" ]; then
>   JAVA_OPTS="${JAVA_OPTS} ${opt_java_props}"
> fi
>
> # prepend command-line classpath to env script classpath
> if [ -n "${opt_classpath}" ]; then
>   if [ -n "${FLUME_CLASSPATH}" ]; then
>     FLUME_CLASSPATH="${opt_classpath}:${FLUME_CLASSPATH}"
>   else
>     FLUME_CLASSPATH="${opt_classpath}"
>   fi
> fi
>
> if [ -z "${FLUME_HOME}" ]; then
>   FLUME_HOME=$(cd $(dirname $0)/..; pwd)
> fi
>
> # prepend $FLUME_HOME/lib jars to the specified classpath (if any)
> if [ -n "${FLUME_CLASSPATH}" ] ; then
>   FLUME_CLASSPATH="${FLUME_HOME}/lib/*:$FLUME_CLASSPATH"
> else
>   FLUME_CLASSPATH="${FLUME_HOME}/lib/*"
> fi
>
> # find java
> if [ -z "${JAVA_HOME}" ] ; then
>   warn "JAVA_HOME is not set!"
>   # Try to use Bigtop to autodetect JAVA_HOME if it's available
>   if [ -e /usr/libexec/bigtop-detect-javahome ] ; then
>     . /usr/libexec/bigtop-detect-javahome
>   elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ] ; then
>     . /usr/lib/bigtop-utils/bigtop-detect-javahome
>   fi
>
>   # Using java from path if bigtop is not installed or couldn't find it
>   if [ -z "${JAVA_HOME}" ] ; then
>     JAVA_DEFAULT=$(type -p java)
>     [ -n "$JAVA_DEFAULT" ] || error "Unable to find java executable. 
> Is it in your PATH?" 1
>     JAVA_HOME=$(cd $(dirname $JAVA_DEFAULT)/..; pwd)
>   fi
> fi
>
> # look for hadoop libs
> add_hadoop_paths
> add_HBASE_paths
>
> # prepend conf dir to classpath
> if [ -n "$opt_conf" ]; then
>   FLUME_CLASSPATH="$opt_conf:$FLUME_CLASSPATH"
> fi
>
> # allow dryrun
> EXEC="exec"
> if [ -n "${opt_dryrun}" ]; then
>   warn "Dryrun mode enabled (will not actually initiate startup)"
>   EXEC="echo"
> fi
>
> # finally, invoke the appropriate command
> if [ -n "$opt_agent" ] ; then
>   run_flume $FLUME_AGENT_CLASS $args
> elif [ -n "$opt_avro_client" ] ; then
>   run_flume $FLUME_AVRO_CLIENT_CLASS $args
> elif [ -n "${opt_version}" ] ; then
>   run_flume $FLUME_VERSION_CLASS $args
> else
>   error "This message should never appear" 1
> fi
>
> exit 0
>
>
>
>
>
>
>
>
> Inder Pall <mailto:inder.pall@gmail.com>
> July 30, 2012 3:40 PM
> Hey if you edit flume-ng script and just before java is invoked put an
>
> echo $CLASSPATH that should help in figuring what all dir's are there 
> in CLASSPATH.
> I am sorry but i am a big fan of windows hence don't run that O.S. to 
> try it out.
>
> Thanks,
> - inder
>
>
>
>
> -- 
> Thanks,
> - Inder
>   Tech Platforms @Inmobi
>   Linkedin - http://goo.gl/eR4Ub

Mime
View raw message