Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of vbagare@me.com designates
 17.172.204.239 as permitted sender)
Content-type: text/plain; charset=iso-8859-1
MIME-version: 1.0 (Mac OS X Mail 7.0 \(1822\))
Subject: Re: from relational to bigger data
From: Vinay Bagare <vbagare@me.com>
In-reply-to: 
 <CADokdcPaVRDLJnpAdAyifwjGqKFBnC4tG-HAZ0LRg4H6yXEoKg@mail.gmail.com>
Date: Thu, 19 Dec 2013 13:59:25 -0800
Content-transfer-encoding: quoted-printable
Message-id: <B99159A7-8EC3-4090-983A-35A8CF25B758@me.com>
References: 
 <CADq_mb-vbBj2VveNDGet6dj+WpDGMkcWfr76bqOSSoWHn_Xx8Q@mail.gmail.com>
 <CADokdcPaVRDLJnpAdAyifwjGqKFBnC4tG-HAZ0LRg4H6yXEoKg@mail.gmail.com>
To: user@hadoop.apache.org, chris@embree.us

I would also look at current setup.
I agree with Chris that 500 GB is fairly insignificant.=20


Best,
Vinay Bagare


On Dec 19, 2013, at 12:51 PM, Chris Embree <cembree@gmail.com> wrote:

> In big data terms, 500G isn't big.  But, moving that much data around
> every night is not trivial either.  I'm going to guess at a lot here,
> but at a very high level.
>=20
> 1. Sqoop the data required to build the summary tables into Hadoop.
> 2. Crunch the summaries into new tables (really just files on Hadoop)
> 3. Sqoop the summarized data back out to Oracle
> 4. Build Indices as needed.
>=20
> Depending on the size of the data being sqoop'd, this might help.  It
> might also take longer.  A real solution would require more details
> and analysis.
>=20
> Chris
>=20
> On 12/19/13, Jay Vee <jvsrvcs@gmail.com> wrote:
>> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>>=20
>> We have summary tables that we rebuild from scratch each night that =
takes
>> about 10 hours.
>> =46rom these summary tables, we have a web interface that accesses =
the
>> summary tables to build reports.
>>=20
>> There is a business reason for doing a complete rebuild of the =
summary
>> tables each night, and using
>> views (as in the sense of Oracle views) is not an option at this =
time.
>>=20
>> If I wanted to leverage Big Data technologies to speed up the summary =
table
>> rebuild, what would be the first step into getting all data into some =
big
>> data storage technology?
>>=20
>> Ideally in the end, we want to retain the summary tables in a =
relational
>> database and have reporting work the same without modifications.
>>=20
>> It's just the crunching of the data and building these relational =
summary
>> tables where we need a significant performance increase.
>>=20