hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Tatarinov <i...@decide.com>
Subject figuring out the right setting for dfs.datanode.max.xcievers
Date Wed, 30 Mar 2011 17:38:38 GMT
I haven't found a good description on this setting and the costs in setting
it too high. Hope somebody can explain.

I have about a year's worth of data partitioned by date. Using 10 nodes and
setting xcievers to 5000, I can only save into 100 or so partitions. As a
result, I have to do 4 rounds of saving data into the underlying partitioned
table (in s3). That's pretty slow.

Should I just set xcievers to 1M or will hadoop crash a result? Is each
xciever really a separate thread?

When will the spelling be corrected? :)

Thanks a bunch!

Mime
View raw message