incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Friedrich <m...@mafr.de>
Subject Re: Hadoop 1.0.3 vs HBase 0.90.5
Date Mon, 06 Aug 2012 16:43:08 GMT
Hi Rahul,

ah, I see. Thanks for the explanation and for the patch!

Regards,
  Matthias

On Monday, 2012-08-06, Rahul wrote:
> The exceptions you see are warnings and an issue for the same is
> still open at HBASE-4709
> <https://issues.apache.org/jira/browse/HBASE-4709> . You would see a
> lot of exceptions but most of  these are going to be  warnings.
> The reason why  it did not work earlier is due to error in verify
> permissions.
 
> 3290 [main] WARN  org.apache.hadoop.hdfs.server.datanode.DataNode  - Invalid directory
in dfs.data.dir: Incorrect permission for /tmp/incubator-crunch/crunch/target/test-data/65210a78-e0a6-4435-97fd-a7e27173fafb/dfs/data/data1,
expected: rwxr-xr-x, while actual: rwxrwxr-x
> 3299 [main] WARN  org.apache.hadoop.hdfs.server.datanode.DataNode  - Invalid directory
in dfs.data.dir: Incorrect permission for /tmp/incubator-crunch/crunch/target/test-data/65210a78-e0a6-4435-97fd-a7e27173fafb/dfs/data/data2,
expected: rwxr-xr-x, while actual: rwxrwxr-x
> 3299 [main] ERROR org.apache.hadoop.hdfs.server.datanode.DataNode  - All directories
in dfs.data.dir are invalid.
 
 
> The same has been corrected in patchs of CRUNCH-4 by setting the
> correct set of permissions.
 
> conf.set("dfs.datanode.data.dir.perm", "775");
 
> regards
> Rahul
 
> On 05-08-2012 03:00, Matthias Friedrich wrote:
>> Good idea, that did it. Still lots of exceptions but it's not the
>> time of day to worry about that.
 
>> Thanks,
>> Matthias
 
>> On Saturday, 2012-08-04, Josh Wills wrote:
>>> Hrm. Probably requires the patch from CRUNCH-4?
 
>>> On Sat, Aug 4, 2012 at 2:04 PM, Matthias Friedrich <matt@mafr.de> wrote:
>>>> Nope, same thing with every single version starting from 0.90.0.
 
>>>> On Saturday, 2012-08-04, Josh Wills wrote:
>>>>> Try 0.90.4, it worked for me, IIRC.
 
>>>>> On Sat, Aug 4, 2012 at 1:50 PM, Matthias Friedrich <matt@mafr.de>
wrote:
>>>>>> Hi,
 
>>>>>> I tried getting the test suite running with Hadoop 1.0.3 and HBase
>>>>>> 0.90.5 for CRUNCH-16. All tests passed except for the HBase word
>>>>>> count (log attached). I'm no HBase expert, but perhaps someone else
>>>>>> knows what's going wrong here.
 
>>>>>> To reproduce simply change version numbers (see attached patch) and
>>>>>> run the following command:
 
>>>>>> mvn -Dit.test=org.apache.crunch.WordCountHBaseIT verify
 
>>>>>> BTW, HBase is really trying my patience today, it causes a dozen
>>>>>> dependency conflicts with Hadoop, half of which have no clean
>>>>>> resolution.
 
>>>>>> Thanks in advance,
>>>>>> Matthias
 

Mime
View raw message