cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Oberman <>
Subject cassandra/hadoop/pig
Date Wed, 06 Jul 2011 18:48:07 GMT
I have a few cassandra/hadoop/pig questions.  I currently have things set up
in a test environment, and for the most part everything works.  But, before
I start to roll things out to production, I wanted to check on/confirm some

When I originally set things up, I used:

One difference I noticed between the two guides, which I ignored at the
time, was how "datanodes" are treated.  The wiki said "At least one node in
your cluster will also need to be a datanode. That's because Hadoop uses
HDFS to store information like jar dependencies for your job, static data
(like stop words for a word count), and things like that - it's the
distributed cache. It's a very small amount of data but the Hadoop cluster
needs it to run properly".  But, the hadoop guide (if you follow it blindly
like I did), creates a datanode on all TaskTracker nodes.  I _think_ that is
controlled by the conf/slaves file, but I haven't proved that yet.  Is there
any good reason to run datanodes on only the JobTracker vs. on all nodes?
If I should only run it on the JobTracker, how do I properly stop the
datanodes from starting automatically (when both start-dfs and start-mapred
seem to draw from the same slaves file)?

I noticed a second issue/oddness with datanodes, in that the HDFS data isn't
always small.  The other day I ran out of disk running my pig script.  I
checked, and by default, hadoop creates HDFS in /tmp, and I'm using EC2 (and
/tmp is on the boot device) which is only 10G by default.  Do other people
put HDFS on a different disk?  If yes, I'll really want to only run one
datanode, as I don't want to re-template all of my cassandra nodes to have
HDFS disks vs. one new JobTracker node.

In terms of hardware, I am running small instances (32bit, 2GB) in the test
cluster, while my production cluster is larges (64bit, 7 or 8GB).  I was
going to check the performance impact there, but even on smalls in test I
was able to run hadoop jobs while serving web requests.  I am wondering if
smalls are causing the high HDFS usage though (I think data might "spill"
more, if I'm understanding things correctly).

If these are more hadoop then cassandra questions, let me know and I'll move
my questions around.

I did want to mention that these are small details compared to the amount of
complicated things that worked like a charm during my configuration and
testing of the combination of cassandra/hadoop/pig.  It was impressive :-)



View raw message