hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangmeng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5996) hadoop 1.1.2. hdfs write bug
Date Sat, 22 Feb 2014 13:29:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909368#comment-13909368

wangmeng commented on HDFS-5996:

These  days  I'm  doing a reserach on Spark and  shark  and   compared    with  mapreduce.
  I  find  spark/shark   have some advantages   over  mapreduce/Hive   for  spark's  im-menory
computing . As  Hegel  says : What is rational is actual and what is actual is rational. 
   I  think  just now  spark  can not  completely replace  mapreduce Frame ,  mapreduce  has
its addvantage.
 But  now I can not  find  an  advantage   that   mapreduce   has  over  spark.  Expect  for
 that  spark  is a  new  system and is not  mature   than mapreduce ,  does  the mapreduce's
  fault  tolerance  is  better than RDD's lineage?      So ,can you   analysis it   ?   thanks.


Best      Regards
Name:    Wang Meng (Boy)
Major:     Software Engineering ---Java ,Shell ,Python , Linux ,Big Data ,Hadoop ,Hive, Sql
On Hadoop , Warehouse
Degree:  Master
E-mail:   sjtufighter@163.com   sjtufighter@sjtu.edu.cn
Tel:         13141202303(BeiJing)   18818272832(ShangHai)
GitHub:    https://github.com/sjtufighter

> hadoop 1.1.2.  hdfs  write bug 
> -------------------------------
>                 Key: HDFS-5996
>                 URL: https://issues.apache.org/jira/browse/HDFS-5996
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: fuse-dfs
>    Affects Versions: 1.1.2
>         Environment: one master and  three slave ,all  of  them are normal
>            Reporter: wangmeng
>             Fix For: 1.1.2
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>       I am  a student from China ,my research  is Hive  data storage on hadoop .There
is a  hdfs-write bug  when I used  sql : insert overwrite table  wangmeng  select  *    from
testTable (this  sql  is  translated   into N map( no Reduce)  jobs,each map .corresponding
to  a  HDFS  file  output On disk. )  No  matter  what value N is , there will  always  exists
 some   DfsdataoutputStream buffer  can  not  write to disk at  last ,such as N=160 files
,then  there my  be  about 5  write-faliure  files .,the  write-failured  hdfs--file size
on disk  is always 0 bytes  rather than a value which  is between 0  and zhe correct  size.
.There does   not  have  any exceptions to throw . and the  HDFS WRITTEN  statistical data
  is  absolutely correct .
>        When  I  debug , I find  those  write-failed DFS-buffer   own  absolutely  correct
values on   its  buffer ,but the  buffer  can  not write to disk  at last although I use Dfsdataoutputstream.flush()
 , Dfsdataoutputstream close() .
>        .I can not find the reason those  dfs-buffer  can not  write success.  Now I choose
 a method to avoide this problem   by using  a temporary  file : for  example , if  the  DFS-buffer
 will write to its destination FINAL, now I will let this DFS-buffer  write to a temporary
file TEM  first ,and  then  I   move  the  TEM  data  to the destination just  by change the
  hdfs-- file path.  This method can avoid  the DFS-buffer  write -failure .Now   I   want
 to   fix this problem  radically ,  so How can I patch  my codes about  this  problem  and
 is  there  anything I  can do ? Many  Thanks.

This message was sent by Atlassian JIRA

View raw message