hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Kunz <ck...@yahoo-inc.com>
Subject Re: Help: libhdfs SIGSEGV
Date Thu, 03 Apr 2008 01:49:00 GMT
Hi Yingyuan,

Did you try to connect to hdfs in main function before spawning off the
threads and using the same handle in all the threads?

-Christian


On 4/2/08 6:21 PM, "Yingyuan Cheng" <yingyuan@staff.sina.com.cn> wrote:

> Hello Arun and all.
> 
> I generated a data block in my main function, then spawned serveral
> threads to write this block serveral times to different files in hdfs,
> my thread function as following:
> 
> -------------------------------------------------------------------
> // File: hdfsbench.cpp
> 
> void* worker_thread_w(void *arg)
> {
> int idx = (int)arg;
> std::string path = g_config.path_prefix;
> string_append(path, "%d", idx);
> 
> hdfsFS fs = hdfsConnect("default", 0);
> if(!fs) {
> PERRORL(ERROR, "Thread %d failed to connect to hdfs!\n", idx);
> g_workers->at(idx).status = TS_FAIL;
> return (void*)false;
> }
> 
> hdfsFile writeFile = hdfsOpenFile(fs, path.c_str(), O_WRONLY, 0, 0, 0);
> if(!writeFile) {
> PERRORL(ERROR, "Thread %d failed to open %s for writing!\n", idx,
> path.c_str());
> g_workers->at(idx).status = TS_FAIL;
> //hdfsDisconnect(fs);
> return (void*)false;
> }
> 
> int i;
> int bwrite, boffset;
> struct timeval tv_start, tv_end;
> 
> gettimeofday(&tv_start, NULL);
> 
> for (i = 0; i < g_config.num_blocks; i++) {
> boffset = 0;
> while (boffset < g_config.block_size) {
> bwrite = hdfsWrite(fs, writeFile, g_buffer->ptr + boffset,
> g_config.block_size - boffset);
> if (bwrite <0) {
> PERRORL(ERROR, "Thread %d failed when writing %s at block %d offset %d\n",
> idx, path.c_str(), i, boffset);
> g_workers->at(idx).status = TS_FAIL;
> i = g_config.num_blocks;
> break;
> }
> boffset += bwrite;
> g_workers->at(idx).nbytes += bwrite;
> }
> }
> 
> gettimeofday(&tv_end, NULL);
> hdfsCloseFile(fs, writeFile);
> //hdfsDisconnect(fs);
> g_workers->at(idx).elapsed = get_timeval_differ(tv_start, tv_end);
> 
> if (g_workers->at(idx).status == TS_UNSET ||
> g_workers->at(idx).status == TS_RUNNING) {
> g_workers->at(idx).status = TS_SUCCESS;
> }
> 
> return (void*)true;
> }
> 
> -------------------------------------------------------------------
> 
> And the following is my running script:
> 
> -------------------------------------------------------------------
> 
> #!/bin/bash
> #
> # File: hdfsbench.sh
> 
> HADOOP_HOME="/opt/hadoop"
> JAVA_HOME="/opt/java"
> 
> export 
> CLASSPATH="$HADOOP_HOME/hadoop-0.16.1-core.jar:$HADOOP_HOME/lib/commons-loggin
> g-1.0.4.jar:$HADOOP_HOME/lib/log4j-1.2.13.jar:$HADOOP_HOME/conf"
> export LD_LIBRARY_PATH="$HADOOP_HOME/libhdfs:$JAVA_HOME/lib/i386/server"
> 
> ./hdfsbench $@
> 
> -------------------------------------------------------------------
> 
> 
> Yingyuan
> 
> Arun C Murthy 写道:
>> 
>> On Apr 2, 2008, at 1:36 AM, Yingyuan Cheng wrote:
>> 
>>> Hello.
>>> 
>>> Is libhdfs thread-safe? I can run single thread reading/writing HDFS
>>> through libhdfs well, but when incrementing number of threads to 2 or
>>> above, I received sigsegv error:
>>> 
>> 
>> Could you explain a bit more? What are you doing in different threads?
>> Are you writing to the same file?
>> 
>> Arun
>> 
> 


Mime
View raw message