hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4606) HDFS API to move file replicas to caller's location
Date Sat, 16 Mar 2013 19:50:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604405#comment-13604405
] 

Bikas Saha commented on HDFS-4606:
----------------------------------

There are several such application hint proposals floating around which makes it clear that
there is need for HDFS to support new API's wrt block placement. The number of proposals also
seems to suggest that the HDFS community needs to abstract the problem space and figure out
the correct way forward (while balancing HDFS core principles and application complexity).
It would help if we try to identify the scenarios/use-cases being desired instead of solutions
for those scenarios/use-cases.
e.g.
HDFS-2121 - By creating replicas on the fly (when read off-switch), it looks like we are trying
to solve the problem of hot data locality for data processing applications. When todays logs
come in, almost every daily job wants to read them but gets stuck on 3 replicas. Being able
to allow hot-data to be highly replicated on demand would help latency and locality. HDFS
needs to understand that such over-replication needs to be tuned down by deleting excess replicas
as demand falls. Persistance is not necessary here. This feature needs to be automatic to
be useful.
HDFS-2576 - This seems to solve the use case when the application knows a priori that certain
files need to be co-located with each other. Its not clear whether all replicas of blocks
of those file needs co-location or not. By specifying the locations, the proposal solves the
problem of getting a good starting point without persisting any co-location state. And thus,
for stable clusters its good solution.
On the same jira, there is an alternate proposal to have co-location of files be a first class
feature that persisted and HDFS can continue to co-locate them across machine failures and
re-balancing. The application could potentially query the co-located machines and use that
to assign its own failover services. This feature is also useful for data-processing applications
that want to colocate frequently joined pre-partitioned data to avoid unnecessary re-partitioning.
I have seen this scenario work at scale at a different Hadoop like system at very large scale.
So its useful.
However, this does not prevent HDFS from co-locating 2 different Hbase region server data
on the same machine. HDFS-4606 addresses that problem by letting client specify that they
want to copy the data locally. But like someone suggested somewhere else, moving data to code
is kind of going against the basic grain of hadoop. So designing such an API needs to be careful
about doing the right thing and preventing abuse.

IMO, letting applications start managing their own blocks needs to be carefully thought out.
It might be enticing at first but we might soon end up with issues on who owns the blocks
and takes actions that HDFS currently takes on blocks for snaphotting, replicating, fault
tolerance and every future HDFS feature. Also, does this mean applications might have to develop
a host of namenode features themselves as they needs to fix more issues down the line. Or
ask HDFS for API's to control all such HDFS actions. And how does HDFS manage all these API's
while doing the right thing for the data in the cluster. Mistakes in data management are too
critical to make.
                
> HDFS API to move file replicas to caller's location
> ---------------------------------------------------
>
>                 Key: HDFS-4606
>                 URL: https://issues.apache.org/jira/browse/HDFS-4606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message