hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yingyuan Cheng <yingy...@staff.sina.com.cn>
Subject Re: Help: libhdfs SIGSEGV
Date Thu, 03 Apr 2008 01:21:19 GMT
Hello Arun and all.

I generated a data block in my main function, then spawned serveral 
threads to write this block serveral times to different files in hdfs, 
my thread function as following:

-------------------------------------------------------------------
// File: hdfsbench.cpp

void* worker_thread_w(void *arg)
{
int idx = (int)arg;
std::string path = g_config.path_prefix;
string_append(path, "%d", idx);

hdfsFS fs = hdfsConnect("default", 0);
if(!fs) {
PERRORL(ERROR, "Thread %d failed to connect to hdfs!\n", idx);
g_workers->at(idx).status = TS_FAIL;
return (void*)false;
}

hdfsFile writeFile = hdfsOpenFile(fs, path.c_str(), O_WRONLY, 0, 0, 0);
if(!writeFile) {
PERRORL(ERROR, "Thread %d failed to open %s for writing!\n", idx, 
path.c_str());
g_workers->at(idx).status = TS_FAIL;
//hdfsDisconnect(fs);
return (void*)false;
}

int i;
int bwrite, boffset;
struct timeval tv_start, tv_end;

gettimeofday(&tv_start, NULL);

for (i = 0; i < g_config.num_blocks; i++) {
boffset = 0;
while (boffset < g_config.block_size) {
bwrite = hdfsWrite(fs, writeFile, g_buffer->ptr + boffset,
g_config.block_size - boffset);
if (bwrite <0) {
PERRORL(ERROR, "Thread %d failed when writing %s at block %d offset %d\n",
idx, path.c_str(), i, boffset);
g_workers->at(idx).status = TS_FAIL;
i = g_config.num_blocks;
break;
}
boffset += bwrite;
g_workers->at(idx).nbytes += bwrite;
}
}

gettimeofday(&tv_end, NULL);
hdfsCloseFile(fs, writeFile);
//hdfsDisconnect(fs);
g_workers->at(idx).elapsed = get_timeval_differ(tv_start, tv_end);

if (g_workers->at(idx).status == TS_UNSET ||
g_workers->at(idx).status == TS_RUNNING) {
g_workers->at(idx).status = TS_SUCCESS;
}

return (void*)true;
}

-------------------------------------------------------------------

And the following is my running script:

-------------------------------------------------------------------

#!/bin/bash
#
# File: hdfsbench.sh

HADOOP_HOME="/opt/hadoop"
JAVA_HOME="/opt/java"

export 
CLASSPATH="$HADOOP_HOME/hadoop-0.16.1-core.jar:$HADOOP_HOME/lib/commons-logging-1.0.4.jar:$HADOOP_HOME/lib/log4j-1.2.13.jar:$HADOOP_HOME/conf"
export LD_LIBRARY_PATH="$HADOOP_HOME/libhdfs:$JAVA_HOME/lib/i386/server"

./hdfsbench $@

-------------------------------------------------------------------


Yingyuan

Arun C Murthy 写道:
>
> On Apr 2, 2008, at 1:36 AM, Yingyuan Cheng wrote:
>
>> Hello.
>>
>> Is libhdfs thread-safe? I can run single thread reading/writing HDFS
>> through libhdfs well, but when incrementing number of threads to 2 or
>> above, I received sigsegv error:
>>
>
> Could you explain a bit more? What are you doing in different threads? 
> Are you writing to the same file?
>
> Arun
>


Mime
View raw message