hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davide Spataro <davide90.spat...@gmail.com>
Subject Fwd: Parallel scan on regions only
Date Tue, 12 Apr 2016 10:15:19 GMT
Hi everyone,

I recently joined a project that uses hbase to store large amount of
granular material simulation data (e.g.  thousands of particles's position
and velocity over time). What I need to do is to efficiently gather the
data from Hbase and perform parallel rendering of them.

The parallel rendering framework I'm using is written in C/C++ and uses MPI
to coordinate the parallel job (ICET library). Each rendering process
should fetch a portion of data from hbase independently and then coordinate
to perform the final rendered image.

What I would like to do is to perform the same scan operation on all
regions in parallel and fetch their result to the corresponding renderer.

I'm trying to implement this mechanism using endpoint coprocessors .
Basically each coprocessor receives the same scan query, performs it
locally (at  region level) and send data  to a specific  rendering process
over network (using sockets).

Do you have guys  any better advice on how  to implement this in hbase?



P.S. I'm running hbase 1.2 on top of hadoop 2.7

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message