Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of stevel@hortonworks.com
 designates 209.85.216.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1351193060.2547.15.camel@merkur>
References: <1351193060.2547.15.camel@merkur>
Date: Thu, 25 Oct 2012 22:10:27 +0100
Message-ID: 
 <CA+4kjVtFqpsSQ_SXupT9HGDLNAw0jpmj+P7A3HekZSB8euMW6A@mail.gmail.com>
Subject: Re: reference architecture
From: Steve Loughran <stevel@hortonworks.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7bdc12fce0201304cce8a188

--047d7bdc12fce0201304cce8a188
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 25 October 2012 20:24, Daniel K=C3=A4fer <d.kaefer@hs-furtwangen.de> wro=
te:

> Hello all,
>
> I'm looking for a reference architecture for hadoop. The only result I
> found is Lambda architecture from Nathan Marz[0].
>

I quite like the new Hadoop in Practice for a lot of that, especially the
answer to #2, "how to store the data", where he looks at all the options.
Joining is the other big issue.

http://steveloughran.blogspot.co.uk/2012/10/hadoop-in-practice-applied-hado=
op.html

Regarding storing DB data, HBase-on-HDFS is where people keep it; Pig and
Hive can work with that as well as rawer data kept in HDFS directly


> With architecture I mean answers to question like:
> - How should I store the data? CSV, Thirft, ProtoBuf
> - How should I model the data? ER-Model, Starschema, something new?
> - normalized or denormalized or both (master data normalized, then
> transformation to denormalized, like ETL)
> - How should i combine database and HDFS-Files?
>
> Are there any other documented architectures for hadoop?
>
> Regards
> Daniel K=C3=A4fer
>
>
> [0] http://www.manning.com/marz/ just a preprint yet, not completed
>
>

--047d7bdc12fce0201304cce8a188
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<br><br><div class=3D"gmail_quote">On 25 October 2012 20:24, Daniel K=C3=A4=
fer <span dir=3D"ltr">&lt;<a href=3D"mailto:d.kaefer@hs-furtwangen.de" targ=
et=3D"_blank">d.kaefer@hs-furtwangen.de</a>&gt;</span> wrote:<br><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex">
Hello all,<br>
<br>
I&#39;m looking for a reference architecture for hadoop. The only result I<=
br>
found is Lambda architecture from Nathan Marz[0].<br></blockquote><div><br>=
</div><div>I quite like the new Hadoop in=C2=A0Practice=C2=A0for a lot of t=
hat, especially the answer to #2, &quot;how to store the data&quot;, where =
he looks at all the options. Joining is the other big issue.</div>
<div><br></div><div><a href=3D"http://steveloughran.blogspot.co.uk/2012/10/=
hadoop-in-practice-applied-hadoop.html">http://steveloughran.blogspot.co.uk=
/2012/10/hadoop-in-practice-applied-hadoop.html</a></div><div>=C2=A0=C2=A0<=
/div><div>
Regarding storing DB data, HBase-on-HDFS is where people keep it; Pig and H=
ive can work with that as well as rawer data kept in HDFS directly</div><di=
v><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo=
rder-left:1px #ccc solid;padding-left:1ex">

<br>
With architecture I mean answers to question like:<br>
- How should I store the data? CSV, Thirft, ProtoBuf<br>
- How should I model the data? ER-Model, Starschema, something new?<br>
- normalized or denormalized or both (master data normalized, then<br>
transformation to denormalized, like ETL)<br>
- How should i combine database and HDFS-Files?<br>
<br>
Are there any other documented architectures for hadoop?<br>
<br>
Regards<br>
Daniel K=C3=A4fer<br>
<br>
<br>
[0] <a href=3D"http://www.manning.com/marz/" target=3D"_blank">http://www.m=
anning.com/marz/</a> just a preprint yet, not completed<br>
<br>
</blockquote></div><br>

--047d7bdc12fce0201304cce8a188--