hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Kousiouris <gkous...@mail.ntua.gr>
Subject Re: Utilize Hadoop over GPGPUs
Date Sun, 09 Oct 2011 19:18:38 GMT
On 10/9/2011 7:56 PM, Amindri Udugala wrote:
> Also I'm interested on the topic "Ability to dynamically increase replicas
> of data in HDFS based on access patterns". I saw that it is reported as an
> issue at jira https://issues.apache.org/jira/browse/HDFS-782 .  The
> resolution is still marked as unresolved. I would like to know if any work
> is currently carried out about this.
>

Regarding this, we are currently investigating some models (based on a 
Matlab prototype that utilizes geneticaly optimized artificial neural 
networks) that have the goal of performing time series analysis of the 
access patterns. What we do is that we take the access logs, extract the 
average bytes/sec reads and writes in x-minute intervals (may be 
whatever one desires, we used 15-minute intervals) thus creating two 
time series, one for read operations and one for write operations with 
these average values. Then by utilizing the time series prediction model 
we can look into the future and decide proactively which replication 
factors to change.

You can find the paper referring to this technique here:
http://users.ntua.gr/gkousiou/publications/PID2095917.pdf

We began this approach for a slightly different  reason (choosing the 
fittest service for federation in a federated cloud scenario), but also 
for adjusting the replication factor (which is a simple HDFS command).

At this stage it is just a matlab prototype (however it can be used 
online for example by using the following approach: 
http://www.computer.org/portal/web/csdl/doi/10.1109/SCC.2010.37
accessible also here 
http://users.ntua.gr/gkousiou/publications/SCC2010.pdf )

However we are planning in the future to create the models through 
Apache Mahout project, in order to fully exploit the capabilities of 
Hadoop and MapReduce.

BR,
George

-- 

---------------------------

George Kousiouris
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: gkousiou@mail.ntua.gr
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece


Mime
View raw message