incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom Hood (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BLUR-441) Row Mutation Bug
Date Fri, 28 Aug 2015 19:05:46 GMT

    [ https://issues.apache.org/jira/browse/BLUR-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720413#comment-14720413
] 

Tom Hood edited comment on BLUR-441 at 8/28/15 7:05 PM:
--------------------------------------------------------

I tried your random UUID version of the test modified slightly to use the blur cluster running
on my laptop (rather than MiniCluster) and it fails right away with the same kind of corruption.

Have you already tried the original code I uploaded with 0.2.4 or were you always using the
newest blur code?

I created the empty table from the blur shell as before with 1 shard and not specifying any
schema.

Laptop Environment
- Dell XPS L702X, 8GB ram, i7-2670QM CPU @ 2.2GHZ (4 cores, 8 threads), Windows 7 SP1
- CentOS 6.4 vm running with Oracle VirtualBox. VM is allocated 6 processors and 4GB ram
- hadoop version 0.20.205.0
- zookeeper version 3.4.5
- blur 0.2.4 (server setup to run with java 1.8.0_45; client/test setup to run with the same,
but also tried 1.6.0_45)

Work Environment
- blur 0.2.4 (server setup to run with some flavor of java 1.7; client/test setup to run with
the same)
- CDH 5.1.0 hadoop stack
- RedHat 6.3 Santiago
- (since going from memory, can't remember anything else about it)

I tried building the blur code on my laptop with the tests enabled, but I had various issues
doing that.  I started with a git clone and was messing around with the build_release.sh script
(had to edit the script due to the version check it does since I am using mvn 3.3.3 which
wants java 1.7 or later and the build_release.sh check is forcing the version to 1.6 or it
fails, although the commentary in the build_release.sh suggests that 1.6 or later should be
allowed).  So I commented out the version check and attempted to build with java 1.8.0_45
(I don't have 1.7 on my laptop right now).  I also changed the hadoop1.versions to 0.20.205.0
and the release_profiles to just hadoop1.  I tried that, but got a compile error:

[ERROR] /srv/local/blur/incubator-blur/blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java:[44,29]
package org.apache.commons.io does not exist
[ERROR] /srv/local/blur/incubator-blur/blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java:[502,13]
cannot find symbol
  symbol:   variable IOUtils
  location: class org.apache.blur.kvs.HdfsKeyValueStore

Rather than look into this, however, I then realized your "Getting Started" webpage just suggests
doing a "mvn install -DskipTests -Dhadoop1" rather than using the build_release.sh.  So I
tried that without the -DskipTests, because I'd like to see if the random UUID test you added
works on my laptop as part of the build.  However, didn't get that far, because of a org.apache.blur.kvs.HdfsKeyValueStoreTest
test failure using a hadoop mini cluster.  Error looks like:

15/08/28 10:29:07 INFO kvs.HdfsKeyValueStore: Opening for writing [hdfs://localhost.localdomain:58684/test/000000000001].
15/08/28 10:29:07 INFO hdfs.StateChange: BLOCK* allocateBlock: /test/000000000001. blk_-7602275106474658451_1001
15/08/28 10:29:07 INFO hdfs.DFSClient: Exception in createBlockOutputStream <mylaptopIP>:53612
java.net.ConnectException: Connection refused
15/08/28 10:29:07 INFO hdfs.DFSClient: Abandoning blk_-7602275106474658451_1001
15/08/28 10:29:07 INFO hdfs.DFSClient: Excluding datanode <mylaptopIP>:53612
15/08/28 10:29:07 WARN namenode.FSNamesystem: Not able to place enough replicas, still in
need of 1 to reach 1
Not able to place enough replicas
15/08/28 10:29:07 ERROR security.UserGroupInformation: PriviledgedActionException as:hood
cause:java.io.IOException: File /test/000000000001 could only be replicated to 0 nodes, instead
of 1

No time to track this down right  now.  However, I added back the -DskipTests and it built
okay.  Now that I have the latest blur jars built, I can later try running from eclipse your
random UUID test with MiniCluster.





was (Author: tom hood):
I tried your random UUID version of the test modified slightly to use the blur cluster running
on my laptop (rather than MiniCluster) and it fails right away with the same kind of corruption.

Have you already tried the original code I uploaded with 0.2.4 or were you always using the
newest blur code?

The blur "cluster" on my laptop is just 1 shard server and 1 controller.   I created the empty
table from the blur shell as before with 1 shard and not specifying any schema.

java version didn't seem to matter: tried versions of 1.6 and 1.8 for the test program.  Only
tried the 1.8 for JAVA_HOME running of the shard+controller servers.  However, the test failure
on my dev machine at work ran with a 1.7 version (both client and server).

Laptop Environment
- Dell XPS L702X, 8GB ram, i7-2670QM CPU @ 2.2GHZ (4 cores, 8 threads), Windows 7 SP1
- CentOS 6.4 vm running with Oracle VirtualBox. VM is allocated 6 processors and 4GB ram
- hadoop version 0.20.205.0
- zookeeper version 3.4.5
- blur 0.2.4 (server setup to run with java 1.8.0_45; client/test setup to run with the same,
but also tried 1.6.0_45)

Work Environment
- blur 0.2.4 (server setup to run with some flavor of java 1.7; client/test setup to run with
the same)
- CDH 5.1.0 hadoop stack
- RedHat 6.3 Santiago
- (since going from memory, can't remember anything else about it)

I tried building the blur code on my laptop with the tests enabled, but I had various issues
doing that.  I started with a git clone and was messing around with the build_release.sh script
(had to edit the script due to the version check it does since I am using mvn 3.3.3 which
wants java 1.7 or later and the build_release.sh check is forcing the version to 1.6 or it
fails, although the commentary in the build_release.sh suggests that 1.6 or later should be
allowed).  So I commented out the version check and attempted to build with java 1.8.0_45
(I don't have 1.7 on my laptop right now).  I also changed the hadoop1.versions to 0.20.205.0
and the release_profiles to just hadoop1.  I tried that, but got a compile error:

[ERROR] /srv/local/blur/incubator-blur/blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java:[44,29]
package org.apache.commons.io does not exist
[ERROR] /srv/local/blur/incubator-blur/blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java:[502,13]
cannot find symbol
  symbol:   variable IOUtils
  location: class org.apache.blur.kvs.HdfsKeyValueStore

Rather than look into this, however, I then realized your "Getting Started" webpage just suggests
doing a "mvn install -DskipTests -Dhadoop1" rather than using the build_release.sh.  So I
tried that without the -DskipTests, because I'd like to see if the random UUID test you added
works on my laptop as part of the build.  However, didn't get that far, because of a org.apache.blur.kvs.HdfsKeyValueStoreTest
test failure using a hadoop mini cluster.  Error looks like:

15/08/28 10:29:07 INFO kvs.HdfsKeyValueStore: Opening for writing [hdfs://localhost.localdomain:58684/test/000000000001].
15/08/28 10:29:07 INFO hdfs.StateChange: BLOCK* allocateBlock: /test/000000000001. blk_-7602275106474658451_1001
15/08/28 10:29:07 INFO hdfs.DFSClient: Exception in createBlockOutputStream <mylaptopIP>:53612
java.net.ConnectException: Connection refused
15/08/28 10:29:07 INFO hdfs.DFSClient: Abandoning blk_-7602275106474658451_1001
15/08/28 10:29:07 INFO hdfs.DFSClient: Excluding datanode <mylaptopIP>:53612
15/08/28 10:29:07 WARN namenode.FSNamesystem: Not able to place enough replicas, still in
need of 1 to reach 1
Not able to place enough replicas
15/08/28 10:29:07 ERROR security.UserGroupInformation: PriviledgedActionException as:hood
cause:java.io.IOException: File /test/000000000001 could only be replicated to 0 nodes, instead
of 1

No time to track this down right  now.  However, I added back the -DskipTests and it built
okay.  Now that I have the latest blur jars built, I can later try running from eclipse your
random UUID test with MiniCluster.




> Row Mutation Bug
> ----------------
>
>                 Key: BLUR-441
>                 URL: https://issues.apache.org/jira/browse/BLUR-441
>             Project: Apache Blur
>          Issue Type: Bug
>          Components: Blur
>    Affects Versions: 0.2.4
>            Reporter: Aaron McCurry
>            Assignee: Aaron McCurry
>            Priority: Blocker
>             Fix For: 0.2.4
>
>         Attachments: BlurMutationBug.java
>
>
> Reported on blur-user mail list:
> http://mail-archives.apache.org/mod_mbox/incubator-blur-user/201508.mbox/<CABO0Qi6m6xR60R64XBpoq7T0Q4GxxAJDcetV_-6PAmg32FdNxg%40mail.gmail.com>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message