incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colton McInroy <col...@dosarrest.com>
Subject Re: Creating/Managing tables saying table already exists
Date Thu, 26 Sep 2013 15:00:09 GMT
Hello Arron,

     Hahaha, I just found the problem. It was a cloned virtual host. I 
went in and changed the hostname to the new hostname, but I did NOT 
change the entry in the /etc/hosts file. I just noticed this, fixed it, 
and then tried the create table command without any problems.

     Now that we know why it was occurring, hunting down the problem and 
fixing it in the code, or giving some different kind of error shouldn't 
be too hard to do. Problem is fixed for me now though.

     I do have a few question if you don't mind... I am still trying to 
wrap my head around how this works. In my current implementation for a 
logging system I create new indexes for each hour because I have a 
massive amount of data coming in. I take in live log data from syslog 
and parse/store it in hourly lucene indexes along with a facet index. I 
want to turn this into a distributed redundant system and blur appears 
to be the way to go. I tried elasticsearch but it is just too slow 
compared to my current implementation. Given I take in gigs of raw log 
data an hour, I need something that is robust and able to keep up with 
in flow of data.

     Does blur handle facet indexing as well? I looked at the javadocs 
and did notice some facet objects, but didn't see any examples on how to 
access that information.

     When taking in lots of data constantly, how is it recommended that 
it be stored? I mentioned above that I create a new index for each hour 
to keep data separated and quicker to search. If I want to look up a 
specific time frame, I only have to load the directories timestamped 
with the hours I want to look at. So instead of having to look at a huge 
index of like a years worth of data, i'm looking at a much smaller data 
set which results in faster query response times. Should a new table be 
created for each hour of data? When I typed in the create command into 
the shell, it takes about 6 seconds to create a table. If I have to 
create a table for each application each hour, this could create a lot 
of lag. Perhaps this is just in my test environment though. Any thoughts 
on this? I also didn't see any examples of how to create tables via code.

     From what I understand when developing  with blur, the code acts as 
a client to connect to a controller server. If you have more than one 
controller, do you have to specify them all? Does the client have to 
connect to all of them, or does it use the list and try to connect to 
each host in the list until it's successful?

     The controller server acts like the query handler right? Does the 
controller reach out to the shards, get the data, run the query again 
it, and return the results to the client, or does each shard process the 
query and return the results from the dataset it has, then the 
controller aggregates the results and responds to the client? Just 
trying to understand where the main processing of the query is done.

     Do shards contain the index data while the location (hdfs) contains 
the documents (what lucene referred to them as)? I read that the shard 
contains the index while the fs contains the data... I just wasn't quiet 
sure what the data was, because when I work with lucene, the index 
directory contains the data as a document.

Thanks,
Colton McInroy

  * Director of Security Engineering

	
Phone
(Toll Free) 	
_US_ 	(888)-818-1344 Press 2
_UK_ 	0-800-635-0551 Press 2

My Extension 	101
24/7 Support 	support@dosarrest.com <mailto:support@dosarrest.com>
Email 	colton@dosarrest.com <mailto:colton@dosarrest.com>
Website 	http://www.dosarrest.com

On 9/26/2013 6:56 AM, Aaron McCurry wrote:
> Colton,
>
> First off welcome, hope we can help you get started or at least past this
> issue.  The the binary artifact of Blur bundles hadoop to try and get
> people up and running as quickly as possible.  Obviously it has not done
> it's job in this case.  ;-)
>
> Could you please post the shard and controller logs?  Also are you running
> hadoop 1.x (CDH3) or hadoop 2.x (CHD4)?  Or are you just trying to get it
> to work with the internal embedded version of hadoop?  Or all of the
> above?  Also have you set anything in the blur-env.sh file or is it the
> vanilla version?  Just trying to cover all the bases here.  I just walked
> through the getting started guide on centos 6.x and last night I walked
> through it on ubuntu 12.03? (I think) the LTS version with success.
>
> I have found 2 issues with it, but neither seem related to the error you
> are getting.
>
> - One is an error with the default MaxDirectMemorySize setting in the
> blur-env.sh, if left at the default it will throw an OOM exception when it
> gets full (which shouldn't happen), but set larger it works as expected.
> - The other is related to some status jsp pages in the daemon processes.
>
> Let me know.
>
> Thanks,
> Aaron
>
>
> On Thu, Sep 26, 2013 at 8:01 AM, Colton McInroy <colton@dosarrest.com>wrote:
>
>> Hello All,
>>
>>      I am trying to get blur operating on a virtual environment using
>> gentoo as a base. I have compiled the latest version of blur as well as
>> tried the 0.2.0 tagged version. Each time I go to try and create a table, I
>> get an IOException saying the table already exists. Here is a basic example
>> of what I have done...
>>
>> hadoop@blur ~ $ tar xfz apache-blur-0.3.0-incubating-**SNAPSHOT-bin.tar.gz
>> hadoop@blur ~ $ mv apache-blur-0.3.0-incubating-**SNAPSHOT-bin blur
>> hadoop@blur ~ $ cd blur
>> hadoop@blur ~/blur $ bin/start-all.sh
>> localhost: ZooKeeper starting as process 9536.
>> localhost: Shard [0] starting as process 9598.
>> localhost: Controller [0] starting as process 9660.
>> hadoop@blur ~/blur $ bin/blur shell
>> blur (default)> create -t testtable -c 11 -l file:///tmp/testtable
>> java.io.IOException: Table [testtable] already exists.
>> blur (default)> enable testtable
>> java.io.IOException: Table [testtable] already enabled.
>> blur (default)> disable testtable
>> java.io.IOException: Table [testtable] already disabled.
>> blur (default)> remove testtable
>> blur (default)>
>>
>> As one can see from the above commands, this is a fresh attempt at
>> starting blur as the instructions on the site provide. I have tried the
>> following versions of jdk with the same problem...
>>
>>    [1]   icedtea-bin-6
>>    [2]   icedtea-bin-7
>>    [3]   oracle-jdk-bin-1.7
>>    [4]   sun-jdk-1.6
>>
>>      Between attempts I made sure to delete /tmp/zk_data to avoid any
>> possible problems it was causing. Each time I would experience the same
>> problem when trying to create a table as a test. I tried this on a ubuntu
>> virtualbox and it worked. I don't understand why there would be a
>> difference between the two considering I tried the same versions of java.
>> The ubuntu system was using openjdk 7 which is what gentoo calls
>> icedtea-bin-7.
>>
>>      When using debug and timed responses, here is what gets outputted when
>> typing the same commands...
>>
>> blur (default)> debug
>> debugging is now on
>> blur (default)> timed
>> timing of commands is now on
>> Last command took 0ms
>> blur (default)> create -t testtable -c 11 -l file:///tmp/testtable
>> TableDescriptor(enabled:true, shardCount:11, tableUri:file:///tmp/**testtable,
>> cluster:default, name:testtable, similarityClass:null, blockCaching:true,
>> blockCachingFileTypes:null, readOnly:false, preCacheCols:null,
>> tableProperties:null, strictTypes:false, defaultMissingFieldType:text,
>> defaultMissingFieldLessIndexin**g:true, defaultMissingFieldProps:null)
>> java.io.IOException: Table [testtable] already exists.
>> BlurException(message:java.io.**IOException: Table [testtable] already
>> exists., stackTraceStr:java.lang.**RuntimeException: java.io.IOException:
>> Table [testtable] already exists.
>>          at org.apache.blur.manager.**clusterstatus.**
>> ZookeeperClusterStatus.**createTable(**ZookeeperClusterStatus.java:**744)
>>          at org.apache.blur.thrift.**TableAdmin.createTable(**
>> TableAdmin.java:101)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>> NativeMethodAccessorImpl.java:**57)
>>          at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>> DelegatingMethodAccessorImpl.**java:43)
>>          at java.lang.reflect.Method.**invoke(Method.java:606)
>>          at org.apache.blur.utils.**BlurUtil$1.invoke(BlurUtil.**java:183)
>>          at com.sun.proxy.$Proxy0.**createTable(Unknown Source)
>>          at org.apache.blur.thrift.**generated.Blur$Processor$**
>> createTable.getResult(Blur.**java:2402)
>>          at org.apache.blur.thrift.**generated.Blur$Processor$**
>> createTable.getResult(Blur.**java:2386)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.ProcessFunction.**
>> process(ProcessFunction.java:**54)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.TBaseProcessor.**
>> process(TBaseProcessor.java:**57)
>>          at org.apache.blur.thrift.server.**AbstractNonblockingServer$**
>> FrameBuffer.invoke(**AbstractNonblockingServer.**java:515)
>>          at org.apache.blur.thrift.server.**Invocation.run(Invocation.**
>> java:34)
>>          at org.apache.blur.concurrent.**ThreadWatcher$**
>> ThreadWatcherExecutorService$**1.run(ThreadWatcher.java:127)
>>          at java.util.concurrent.**ThreadPoolExecutor.runWorker(**
>> ThreadPoolExecutor.java:1145)
>>          at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
>> ThreadPoolExecutor.java:615)
>>          at java.lang.Thread.run(Thread.**java:724)
>> Caused by: java.io.IOException: Table [testtable] already exists.
>>          at org.apache.blur.manager.**clusterstatus.**
>> ZookeeperClusterStatus.**createTable(**ZookeeperClusterStatus.java:**722)
>>          ... 17 more
>> , errorType:UNKNOWN)
>>          at org.apache.blur.thrift.**generated.Blur$createTable_**
>> result$createTable_**resultStandardScheme.read(**Blur.java:3818)
>>          at org.apache.blur.thrift.**generated.Blur$createTable_**
>> result$createTable_**resultStandardScheme.read(**Blur.java:3804)
>>          at org.apache.blur.thrift.**generated.Blur$createTable_**
>> result.read(Blur.java:3754)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.TServiceClient.**
>> receiveBase(TServiceClient.**java:78)
>>          at org.apache.blur.thrift.**generated.Blur$Client.recv_**
>> createTable(Blur.java:458)
>>          at org.apache.blur.thrift.**generated.Blur$Client.**
>> createTable(Blur.java:445)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>> NativeMethodAccessorImpl.java:**57)
>>          at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>> DelegatingMethodAccessorImpl.**java:43)
>>          at java.lang.reflect.Method.**invoke(Method.java:606)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler$1.**call(BlurClient.java:59)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler$1.**call(BlurClient.java:55)
>>          at org.apache.blur.thrift.**AbstractCommand.call(**
>> AbstractCommand.java:62)
>>          at org.apache.blur.thrift.**BlurClientManager.execute(**
>> BlurClientManager.java:167)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler.**invoke(BlurClient.java:55)
>>          at com.sun.proxy.$Proxy0.**createTable(Unknown Source)
>>          at org.apache.blur.shell.**CreateTableCommand.doit(**
>> CreateTableCommand.java:100)
>>          at org.apache.blur.shell.Main.**main(Main.java:471)
>> Last command took 61289ms
>> blur (default)> enable testtable
>> java.io.IOException: Table [testtable] already enabled.
>> BlurException(message:java.io.**IOException: Table [testtable] already
>> enabled., stackTraceStr:java.lang.**RuntimeException:
>> java.io.IOException: Table [testtable] already enabled.
>>          at org.apache.blur.manager.**clusterstatus.**
>> ZookeeperClusterStatus.**enableTable(**ZookeeperClusterStatus.java:**809)
>>          at org.apache.blur.thrift.**TableAdmin.enableTable(**
>> TableAdmin.java:137)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>> NativeMethodAccessorImpl.java:**57)
>>          at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>> DelegatingMethodAccessorImpl.**java:43)
>>          at java.lang.reflect.Method.**invoke(Method.java:606)
>>          at org.apache.blur.utils.**BlurUtil$1.invoke(BlurUtil.**java:183)
>>          at com.sun.proxy.$Proxy0.**enableTable(Unknown Source)
>>          at org.apache.blur.thrift.**generated.Blur$Processor$**
>> enableTable.getResult(Blur.**java:2426)
>>          at org.apache.blur.thrift.**generated.Blur$Processor$**
>> enableTable.getResult(Blur.**java:2410)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.ProcessFunction.**
>> process(ProcessFunction.java:**54)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.TBaseProcessor.**
>> process(TBaseProcessor.java:**57)
>>          at org.apache.blur.thrift.server.**AbstractNonblockingServer$**
>> FrameBuffer.invoke(**AbstractNonblockingServer.**java:515)
>>          at org.apache.blur.thrift.server.**Invocation.run(Invocation.**
>> java:34)
>>          at org.apache.blur.concurrent.**ThreadWatcher$**
>> ThreadWatcherExecutorService$**1.run(ThreadWatcher.java:127)
>>          at java.util.concurrent.**ThreadPoolExecutor.runWorker(**
>> ThreadPoolExecutor.java:1145)
>>          at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
>> ThreadPoolExecutor.java:615)
>>          at java.lang.Thread.run(Thread.**java:724)
>> Caused by: java.io.IOException: Table [testtable] already enabled.
>>          at org.apache.blur.manager.**clusterstatus.**
>> ZookeeperClusterStatus.**enableTable(**ZookeeperClusterStatus.java:**805)
>>          ... 17 more
>> , errorType:UNKNOWN)
>>          at org.apache.blur.thrift.**generated.Blur$enableTable_**
>> result$enableTable_**resultStandardScheme.read(**Blur.java:4540)
>>          at org.apache.blur.thrift.**generated.Blur$enableTable_**
>> result$enableTable_**resultStandardScheme.read(**Blur.java:4526)
>>          at org.apache.blur.thrift.**generated.Blur$enableTable_**
>> result.read(Blur.java:4476)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.TServiceClient.**
>> receiveBase(TServiceClient.**java:78)
>>          at org.apache.blur.thrift.**generated.Blur$Client.recv_**
>> enableTable(Blur.java:481)
>>          at org.apache.blur.thrift.**generated.Blur$Client.**
>> enableTable(Blur.java:468)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>> NativeMethodAccessorImpl.java:**57)
>>          at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>> DelegatingMethodAccessorImpl.**java:43)
>>          at java.lang.reflect.Method.**invoke(Method.java:606)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler$1.**call(BlurClient.java:59)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler$1.**call(BlurClient.java:55)
>>          at org.apache.blur.thrift.**AbstractCommand.call(**
>> AbstractCommand.java:62)
>>          at org.apache.blur.thrift.**BlurClientManager.execute(**
>> BlurClientManager.java:167)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler.**invoke(BlurClient.java:55)
>>          at com.sun.proxy.$Proxy0.**enableTable(Unknown Source)
>>          at org.apache.blur.shell.**EnableTableCommand.doit(**
>> EnableTableCommand.java:37)
>>          at org.apache.blur.shell.Main.**main(Main.java:471)
>> Last command took 3ms
>> blur (default)> disable testtable
>> java.io.IOException: Table [testtable] already disabled.
>> BlurException(message:java.io.**IOException: Table [testtable] already
>> disabled., stackTraceStr:java.lang.**RuntimeException:
>> java.io.IOException: Table [testtable] already disabled.
>>          at org.apache.blur.manager.**clusterstatus.**
>> ZookeeperClusterStatus.**disableTable(**ZookeeperClusterStatus.java:**784)
>>          at org.apache.blur.thrift.**TableAdmin.disableTable(**
>> TableAdmin.java:120)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>> NativeMethodAccessorImpl.java:**57)
>>          at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>> DelegatingMethodAccessorImpl.**java:43)
>>          at java.lang.reflect.Method.**invoke(Method.java:606)
>>          at org.apache.blur.utils.**BlurUtil$1.invoke(BlurUtil.**java:183)
>>          at com.sun.proxy.$Proxy0.**disableTable(Unknown Source)
>>          at org.apache.blur.thrift.**generated.Blur$Processor$**
>> disableTable.getResult(Blur.**java:2450)
>>          at org.apache.blur.thrift.**generated.Blur$Processor$**
>> disableTable.getResult(Blur.**java:2434)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.ProcessFunction.**
>> process(ProcessFunction.java:**54)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.TBaseProcessor.**
>> process(TBaseProcessor.java:**57)
>>          at org.apache.blur.thrift.server.**AbstractNonblockingServer$**
>> FrameBuffer.invoke(**AbstractNonblockingServer.**java:515)
>>          at org.apache.blur.thrift.server.**Invocation.run(Invocation.**
>> java:34)
>>          at org.apache.blur.concurrent.**ThreadWatcher$**
>> ThreadWatcherExecutorService$**1.run(ThreadWatcher.java:127)
>>          at java.util.concurrent.**ThreadPoolExecutor.runWorker(**
>> ThreadPoolExecutor.java:1145)
>>          at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
>> ThreadPoolExecutor.java:615)
>>          at java.lang.Thread.run(Thread.**java:724)
>> Caused by: java.io.IOException: Table [testtable] already disabled.
>>          at org.apache.blur.manager.**clusterstatus.**
>> ZookeeperClusterStatus.**disableTable(**ZookeeperClusterStatus.java:**780)
>>          ... 17 more
>> , errorType:UNKNOWN)
>>          at org.apache.blur.thrift.**generated.Blur$disableTable_**
>> result$disableTable_**resultStandardScheme.read(**Blur.java:5262)
>>          at org.apache.blur.thrift.**generated.Blur$disableTable_**
>> result$disableTable_**resultStandardScheme.read(**Blur.java:5248)
>>          at org.apache.blur.thrift.**generated.Blur$disableTable_**
>> result.read(Blur.java:5198)
>>          at org.apache.blur.thirdparty.**thrift_0_9_0.TServiceClient.**
>> receiveBase(TServiceClient.**java:78)
>>          at org.apache.blur.thrift.**generated.Blur$Client.recv_**
>> disableTable(Blur.java:504)
>>          at org.apache.blur.thrift.**generated.Blur$Client.**
>> disableTable(Blur.java:491)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>> NativeMethodAccessorImpl.java:**57)
>>          at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>> DelegatingMethodAccessorImpl.**java:43)
>>          at java.lang.reflect.Method.**invoke(Method.java:606)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler$1.**call(BlurClient.java:59)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler$1.**call(BlurClient.java:55)
>>          at org.apache.blur.thrift.**AbstractCommand.call(**
>> AbstractCommand.java:62)
>>          at org.apache.blur.thrift.**BlurClientManager.execute(**
>> BlurClientManager.java:167)
>>          at org.apache.blur.thrift.**BlurClient$**
>> BlurClientInvocationHandler.**invoke(BlurClient.java:55)
>>          at com.sun.proxy.$Proxy0.**disableTable(Unknown Source)
>>          at org.apache.blur.shell.**DisableTableCommand.doit(**
>> DisableTableCommand.java:35)
>>          at org.apache.blur.shell.Main.**main(Main.java:471)
>> Last command took 61285ms
>> blur (default)> remove testtable
>> Last command took 38ms
>> blur (default)>
>>
>>      I emailed Aaron directly who suggested I make sure I clear the
>> /tmp/zk_data directory, and also requested that I submit this to the
>> mailing list, so here it is.
>>
>>      Any help in resolving this would be greatly appreciated.
>>
>> --
>> Thanks,
>> Colton McInroy
>>
>>   * Director of Security Engineering
>>
>>
>> Phone
>> (Toll Free)
>> _US_    (888)-818-1344 Press 2
>> _UK_    0-800-635-0551 Press 2
>>
>> My Extension    101
>> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com>
>> Email   colton@dosarrest.com <mailto:colton@dosarrest.com>
>> Website         http://www.dosarrest.com
>>
>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message