hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jürgen Kaatz <ka...@etracker.com>
Subject RE: Hadoop/HDFS Thrift/PHP connection doesnt work
Date Mon, 20 Jul 2009 09:16:36 GMT
Hi dhruba,

thank you for your answer. I had the same idea like you before, that the thrift server configuration
cannot read the hadoop cluster configuration. The Hadoop-Cluster is installed in '/usr/local/hadoop'
and the respective configuration in '/usr/local/hadoop/conf'. You can watch the classpath
of the thrift server below (and the respective script to start the thrift server). For me
it seems all is ok...

The Java-Code of the thrift server has a method 'public HadoopThriftHandler()', where a configuration
is generated. But for me it seems, this must be an empty or default configuration, isn't it?

Thanks in advance
Juergen


Method 'HadoopThriftHandler' from 'hadoop/src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java'

    /**
     * HadoopThriftServer
     *
     * Constructor for the HadoopThriftServer glue with Thrift Class.
     *
     * @param name - the name of this handler
     */
    public HadoopThriftHandler(String name) {
      conf = new Configuration();
      now = now();
      try {
        inactivityThread = new Daemon(new InactivityMonitor());
        fs = FileSystem.get(conf);
      } catch (IOException e) {
        LOG.warn("Unable to open hadoop file system...");
        Runtime.getRuntime().exit(-1);
      }
    } 

Classpath of the thrift server:
/usr/local/hadoop/hadoop-0.19.1-ant.jar
/usr/local/hadoop/hadoop-0.19.1-core.jar
/usr/local/hadoop/hadoop-0.19.1-examples.jar
/usr/local/hadoop/hadoop-0.19.1-test.jar
/usr/local/hadoop/hadoop-0.19.1-tools.jar
/usr/local/hadoop/lib/commons-cli-2.0-SNAPSHOT.jar
/usr/local/hadoop/lib/commons-codec-1.3.jar
/usr/local/hadoop/lib/commons-httpclient-3.0.1.jar
/usr/local/hadoop/lib/commons-logging-1.0.4.jar
/usr/local/hadoop/lib/commons-logging-api-1.0.4.jar
/usr/local/hadoop/lib/commons-net-1.4.1.jar
/usr/local/hadoop/lib/hsqldb-1.8.0.10.jar
/usr/local/hadoop/lib/jets3t-0.6.1.jar
/usr/local/hadoop/lib/jetty-5.1.4.jar
/usr/local/hadoop/lib/junit-3.8.1.jar
/usr/local/hadoop/lib/kfs-0.2.0.jar
/usr/local/hadoop/lib/log4j-1.2.15.jar
/usr/local/hadoop/lib/oro-2.0.8.jar
/usr/local/hadoop/lib/servlet-api.jar
/usr/local/hadoop/lib/slf4j-api-1.4.3.jar
/usr/local/hadoop/lib/slf4j-log4j12-1.4.3.jar
/usr/local/hadoop/lib/xmlenc-0.52.jar
/usr/local/hadoop/contrib/hive/lib/antlr-runtime-3.0.1.jar
/usr/local/hadoop/contrib/hive/lib/asm-3.1.jar
/usr/local/hadoop/contrib/hive/lib/commons-collections-3.2.1.jar
/usr/local/hadoop/contrib/hive/lib/commons-jexl-1.1.jar
/usr/local/hadoop/contrib/hive/lib/commons-lang-2.4.jar
/usr/local/hadoop/contrib/hive/lib/derby.jar
/usr/local/hadoop/contrib/hive/lib/hive_cli.jar
/usr/local/hadoop/contrib/hive/lib/hive_common.jar
/usr/local/hadoop/contrib/hive/lib/hive_exec.jar
/usr/local/hadoop/contrib/hive/lib/hive_metastore.jar
/usr/local/hadoop/contrib/hive/lib/jdo2-api-2.1.jar
/usr/local/hadoop/contrib/hive/lib/jline-0.9.94.jar
/usr/local/hadoop/contrib/hive/lib/jpox-core-1.2.2.jar
/usr/local/hadoop/contrib/hive/lib/jpox-enhancer-1.2.2.jar
/usr/local/hadoop/contrib/hive/lib/jpox-rdbms-1.2.2.jar
/usr/local/hadoop/contrib/hive/lib/libfb303.jar
/usr/local/hadoop/contrib/hive/lib/libthrift.jar
/usr/local/hadoop/contrib/hive/lib/stringtemplate-3.1b1.jar
/usr/local/hadoop/contrib/hive/lib/TestSerDe.jar
/usr/local/hadoop/contrib/hive/lib/velocity-1.5.jar
/usr/local/hadoop/contrib/thriftfs/hadoop-0.19.1-thriftfs.jar
/usr/local/hadoop/src/contrib/thriftfs/lib/hadoopthriftapi.jar
/usr/local/hadoop/src/contrib/thriftfs/lib/libthrift.jar

Script to start the thrift server:
#!/bin/sh

CLASSPATH=
#TOP=../../../..
TOP=/usr/local/hadoop

# the hadoop libraries
for f in $TOP/*.jar ; do
  CLASSPATH=$CLASSPATH:$f
done

# the apache libraries
for f in $TOP/lib/*.jar ; do
  CLASSPATH=$CLASSPATH:$f
done

# the thrift libraries
for f in $TOP/contrib/hive/lib/*.jar ; do
  CLASSPATH=$CLASSPATH:$f
done

# the thrift server
for f in $TOP/contrib/thriftfs/*.jar ; do
  CLASSPATH=$CLASSPATH:$f
done
# the thrift hadoop api
for f in $TOP/src/contrib/thriftfs/lib/*.jar ; do
  CLASSPATH=$CLASSPATH:$f
done

echo $CLASSPATH
exit
#-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n
java -Dcom.sun.management.jmxremote -cp $CLASSPATH org.apache.hadoop.thriftfs.HadoopThriftServer
$*



> -----Original Message-----
> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> Sent: Saturday, July 18, 2009 2:28 AM
> To: general@hadoop.apache.org
> Subject: Re: Hadoop/HDFS Thrift/PHP connection doesnt work
> 
> I am suspecting that the thrift server is started using the configuration in
> your hadoop workspace rather than using the configuration of the HDFS
> cluster that you must have setup earlier. Please check the CLASSPATH of the
> thrit server....it should include the configuration of your HDFS cluster.
> 
> thanks,
> dhruba
> 
> 
> 2009/7/17 Jürgen Kaatz <kaatz@etracker.com>
> 
> > Hi,
> >
> > I'm trying to use thrift with PHP to handle file system operations on HDFS
> > without any success.
> >
> > I run a thrift server instance started with the script
> > 'start_thrift_server.sh' is supposed located in
> > 'hadoop/src/contrib/thriftfs/scripts/'. Then I use a little script
> > 'hdfs.php' to connect to this thrift server and try to create a new file
> > 'foo' on HDFS. But the file is created on my local filesystem not on HDFS.
> >
> > Can anybody help me?
> > Thanks Juergen
> >
> > -- hdfs.php
> > ------------------------------------------------------------------------
> > <?php
> > error_reporting(E_ALL);
> > ini_set('display_errors', 'on');
> >
> > $GLOBALS['THRIFT_ROOT'] = '/home/hadoop/cc/thrift';
> > define('ETCC_THRIFT_ROOT', $GLOBALS['THRIFT_ROOT']);
> >
> > require_once(ETCC_THRIFT_ROOT.'/Thrift.php' );
> > require_once(ETCC_THRIFT_ROOT.'/transport/TSocket.php' );
> > require_once(ETCC_THRIFT_ROOT.'/transport/TBufferedTransport.php' );
> > require_once(ETCC_THRIFT_ROOT.'/protocol/TBinaryProtocol.php' );
> >
> > $socket = new TSocket('hadoop-master', 50021);
> > $socket->setSendTimeout(10000); $socket->setRecvTimeout(20000); $transport
=
> > new TBufferedTransport($socket); $protocol = new
> > TBinaryProtocol($transport);
> >
> >
> >
> require_once(ETCC_THRIFT_ROOT.'/packages/hadoopfs/ThriftHadoopFileSystem.php')
> ;
> > $client = new ThriftHadoopFileSystemClient($protocol);
> >
> > $transport->open();
> >
> > try
> > {
> >    $pathname = new Pathname(array('pathname' => 'foo'));
> >    // $result = $client->listStatus($pathname);
> >    $result = $client->create($pathname);
> >    $client->close($result);
> >    print_r($result);
> > }
> > catch(Exception $e)
> > {
> >    print_r($e);
> > }
> >
> > $transport->close();
> >
> >

Mime
View raw message