Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.210.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAjbL_kmTSmYXyHUaZysB9YZcDVDYOQM=HV7rCMwvqHuMF1z-Q@mail.gmail.com>
References: 
 <CAAjbL_kmTSmYXyHUaZysB9YZcDVDYOQM=HV7rCMwvqHuMF1z-Q@mail.gmail.com>
Date: Wed, 6 Jul 2011 15:29:35 -0400
Message-ID: 
 <CAENxBwzg68abo_BvOeqTEKAg0Nv-QMJr4xucTH=iF_BnRwqjPw@mail.gmail.com>
Subject: Re: cassandra/hadoop/pig
From: Edward Capriolo <edlinuxguru@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=20cf303dd95ada83ce04a76b9edc

--20cf303dd95ada83ce04a76b9edc
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jul 6, 2011 at 2:48 PM, William Oberman <oberman@civicscience.com>wrote:

> I have a few cassandra/hadoop/pig questions.  I currently have things set
> up in a test environment, and for the most part everything works.  But,
> before I start to roll things out to production, I wanted to check
> on/confirm some things.
>
> When I originally set things up, I used:
> http://wiki.apache.org/cassandra/HadoopSupport
> http://hadoop.apache.org/common/docs/r0.20.203.0/cluster_setup.html
>
> One difference I noticed between the two guides, which I ignored at the
> time, was how "datanodes" are treated.  The wiki said "At least one node in
> your cluster will also need to be a datanode. That's because Hadoop uses
> HDFS to store information like jar dependencies for your job, static data
> (like stop words for a word count), and things like that - it's the
> distributed cache. It's a very small amount of data but the Hadoop cluster
> needs it to run properly".  But, the hadoop guide (if you follow it blindly
> like I did), creates a datanode on all TaskTracker nodes.  I _think_ that is
> controlled by the conf/slaves file, but I haven't proved that yet.  Is there
> any good reason to run datanodes on only the JobTracker vs. on all nodes?
> If I should only run it on the JobTracker, how do I properly stop the
> datanodes from starting automatically (when both start-dfs and start-mapred
> seem to draw from the same slaves file)?
>
> I noticed a second issue/oddness with datanodes, in that the HDFS data
> isn't always small.  The other day I ran out of disk running my pig script.
> I checked, and by default, hadoop creates HDFS in /tmp, and I'm using EC2
> (and /tmp is on the boot device) which is only 10G by default.  Do other
> people put HDFS on a different disk?  If yes, I'll really want to only run
> one datanode, as I don't want to re-template all of my cassandra nodes to
> have HDFS disks vs. one new JobTracker node.
>
> In terms of hardware, I am running small instances (32bit, 2GB) in the test
> cluster, while my production cluster is larges (64bit, 7 or 8GB).  I was
> going to check the performance impact there, but even on smalls in test I
> was able to run hadoop jobs while serving web requests.  I am wondering if
> smalls are causing the high HDFS usage though (I think data might "spill"
> more, if I'm understanding things correctly).
>
> If these are more hadoop then cassandra questions, let me know and I'll
> move my questions around.
>
> I did want to mention that these are small details compared to the amount
> of complicated things that worked like a charm during my configuration and
> testing of the combination of cassandra/hadoop/pig.  It was impressive :-)
>
> Thanks!
>
> will
>
>
The logic that "only one datanode is needed" is not an absolute truth. If
your jobs use ColumnFamilyInputFormat to read and write to
ColumnFamilyOutputFormat then technically you only need one DataNode to hold
the distributed cache. However, if you have a large amount of intermediate
results or even a multiphase job that has to persist data between phases
(this is very very common) then that single DataNode is a bottleneck. Most
hadoop clusters run a DataNode and TaskTracker on each slave.

Most situations would use datanodes very heavily, for example suppose you
have 4 map/reduce jobs to run on the same Cassandra data. Ingesting the data
from Cassandra at the beginning of each job might would be wasteful. It
might be better to take the data into HDFS during the first job and then
save it. Your subsequent jobs could use that instead of re-acquiring it from
Cassandra.

--20cf303dd95ada83ce04a76b9edc
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<br><br><div class=3D"gmail_quote">On Wed, Jul 6, 2011 at 2:48 PM, William =
Oberman <span dir=3D"ltr">&lt;<a href=3D"mailto:oberman@civicscience.com">o=
berman@civicscience.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail=
_quote" style=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt=
 0pt 0.8ex; padding-left: 1ex;">
I have a few cassandra/hadoop/pig questions.=A0 I currently have things set=
 up in a test environment, and for the most part everything works.=A0 But, =
before I start to roll things out to production, I wanted to check on/confi=
rm some things.=A0 <br>


<br>When I originally set things up, I used:<br><a href=3D"http://wiki.apac=
he.org/cassandra/HadoopSupport" target=3D"_blank">http://wiki.apache.org/ca=
ssandra/HadoopSupport</a><br><a href=3D"http://hadoop.apache.org/common/doc=
s/r0.20.203.0/cluster_setup.html" target=3D"_blank">http://hadoop.apache.or=
g/common/docs/r0.20.203.0/cluster_setup.html</a><br>


<br>One difference I noticed between the two guides, which I ignored at the=
 time, was how &quot;datanodes&quot; are treated.=A0 The wiki said &quot;At=
 least one node in your cluster will also need to be a datanode. =20
That&#39;s because Hadoop uses HDFS to store information like jar=20
dependencies for your job, static data (like stop words for a word=20
count), and things like that - it&#39;s the distributed cache.  It&#39;s a =
very=20
small amount of data but the Hadoop cluster needs it to run properly&quot;.=
=A0 But, the hadoop guide (if you follow it blindly like I did), creates a =
datanode on all TaskTracker nodes.=A0 I _think_ that is controlled by the c=
onf/slaves file, but I haven&#39;t proved that yet.=A0 Is there any good re=
ason to run datanodes on only the JobTracker vs. on all nodes?=A0 If I shou=
ld only run it on the JobTracker, how do I properly stop the datanodes from=
 starting automatically (when both start-dfs and start-mapred seem to draw =
from the same slaves file)?<br>


<br>I noticed a second issue/oddness with datanodes, in that the HDFS data =
isn&#39;t always small.=A0 The other day I ran out of disk running my pig s=
cript.=A0 I checked, and by default, hadoop creates HDFS in /tmp, and I&#39=
;m using EC2 (and /tmp is on the boot device) which is only 10G by default.=
=A0 Do other people put HDFS on a different disk?=A0 If yes, I&#39;ll reall=
y want to only run one datanode, as I don&#39;t want to re-template all of =
my cassandra nodes to have HDFS disks vs. one new JobTracker node.=A0 <br>


<br>In terms of hardware, I am running small instances (32bit, 2GB) in the =
test cluster, while my production cluster is larges (64bit, 7 or 8GB).=A0 I=
 was going to check the performance impact there, but even on smalls in tes=
t I was able to run hadoop jobs while serving web requests.=A0 I am wonderi=
ng if smalls are causing the high HDFS usage though (I think data might &qu=
ot;spill&quot; more, if I&#39;m understanding things correctly).=A0 <br>


<br>If these are more hadoop then cassandra questions, let me know and I=
9;ll move my questions around.<br><br>I did want to mention that these are =
small details compared to the amount of complicated things that worked like=
 a charm during my configuration and testing of the combination of cassandr=
a/hadoop/pig.=A0 It was impressive :-)<br>


<br>Thanks!<br><br>will<br><br>
</blockquote></div><br>The logic that &quot;only one datanode is needed&quo=
t; is not an absolute truth. If your jobs use ColumnFamilyInputFormat to re=
ad and write to ColumnFamilyOutputFormat then technically you only need one=
 DataNode to hold the distributed cache. However, if you have a large amoun=
t of intermediate results or even a multiphase job that has to persist data=
 between phases (this is very very common) then that single DataNode is a b=
ottleneck. Most hadoop clusters run a DataNode and TaskTracker on each slav=
e. <br>
<br>Most situations would use datanodes very heavily, for example suppose y=
ou have 4 map/reduce jobs to run on the same Cassandra data. Ingesting the =
data from Cassandra at the beginning of each job might would be wasteful. I=
t might be better to take the data into HDFS during the first job and then =
save it. Your subsequent jobs could use that instead of re-acquiring it fro=
m Cassandra.<br>

--20cf303dd95ada83ce04a76b9edc--