accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mwa...@apache.org
Subject [1/2] accumulo-website git commit: ACCUMULO-4673 Improved documentation for proxy
Date Wed, 05 Jul 2017 18:47:29 GMT
Repository: accumulo-website
Updated Branches:
  refs/heads/asf-site cfab25205 -> 9e4e6e676
  refs/heads/master 74699ed77 -> f07e47343


ACCUMULO-4673 Improved documentation for proxy


Project: http://git-wip-us.apache.org/repos/asf/accumulo-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo-website/commit/f07e4734
Tree: http://git-wip-us.apache.org/repos/asf/accumulo-website/tree/f07e4734
Diff: http://git-wip-us.apache.org/repos/asf/accumulo-website/diff/f07e4734

Branch: refs/heads/master
Commit: f07e47343c746c06e8cb84c064fbd622d82a519e
Parents: 74699ed
Author: Mike Walch <mwalch@apache.org>
Authored: Mon Jul 3 12:10:58 2017 -0400
Committer: Mike Walch <mwalch@apache.org>
Committed: Mon Jul 3 16:08:17 2017 -0400

----------------------------------------------------------------------
 _docs-unreleased/development/proxy.md | 320 ++++++++++++++++++++++++++---
 1 file changed, 292 insertions(+), 28 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/f07e4734/_docs-unreleased/development/proxy.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/development/proxy.md b/_docs-unreleased/development/proxy.md
index c0f71dd..2cca885 100644
--- a/_docs-unreleased/development/proxy.md
+++ b/_docs-unreleased/development/proxy.md
@@ -16,49 +16,67 @@ The proxy server can live on any node in which the basic client API would
work.
 means it must be able to communicate with the Master, ZooKeepers, NameNode, and the
 DataNodes. A proxy client only needs the ability to communicate with the proxy server.
 
-## Configuration
+## Running the Proxy Server
 
-The configuration options for the proxy server live inside of a properties file. At
-the very least, you need to supply the following properties:
+The proxy server is included in the Accumulo tarball distribution and can be run using
+the `accumulo` or `accumulo-service` command. A sample proxy configuration file can be found
at
+`conf/templates/proxy.properties`. Create a copy of this file and edit it for your environment:
+
+    cp ./conf/templates/proxy.properties ./conf/
+    vim ./conf/proxy.properties
+
+At the very least, you need to configure the following properties:
 
-    protocolFactory=org.apache.thrift.protocol.TCompactProtocol$Factory
-    tokenClass=org.apache.accumulo.core.client.security.tokens.PasswordToken
-    port=42424
     instance=test
     zookeepers=localhost:2181
+    port=42424
+    protocolFactory=org.apache.thrift.protocol.TCompactProtocol$Factory
+    tokenClass=org.apache.accumulo.core.client.security.tokens.PasswordToken
 
-You can find a sample configuration file in your distribution at `proxy/proxy.properties`.
+After `proxy.properties` is configured, the proxy server can be started using the `accumulo`
+or `accumulo-service` commands:
 
-This sample configuration file further demonstrates an ability to back the proxy server
-by MiniAccumuloCluster.
+To start the proxy in the foreground and log to the console, use the `accumulo` command:
 
-## Running the Proxy Server
+    accumulo proxy -p /path/to/proxy.properties
+
+To background the process and redirect logs, use the `accumulo-service` command (a `proxy.properties`
+file must exist in `conf/` if using this method):
+
+    accumulo-service proxy start
 
-After the properties file holding the configuration is created, the proxy server
-can be started using the following command in the Accumulo distribution (assuming
-your properties file is named `config.properties`):
+## Prerequisites for Proxy Clients
 
-    accumulo proxy -p config.properties
+Before you can run a proxy client, you will need the following:
 
-## Creating a Proxy Client
+1. Proxy client code generated for your language
+2. Thrift library installed for your language
 
-Aside from installing the Thrift compiler, you will also need the language-specific library
-for Thrift installed to generate client code in that language. Typically, your operating
-system's package manager will be able to automatically install these for you in an expected
-location such as `/usr/lib/python/site-packages/thrift`.
+These requirements are described in detail below.
 
-You can find the thrift file for generating the client at `proxy/proxy.thrift`.
+### Proxy client code
 
-After a client is generated, the port specified in the configuration properties above will
be
-used to connect to the server.
+The Accumulo tarball distribution ships with pre-generated client code for Python, Ruby,
and C++ in
+`lib/proxy`.
 
-## Using a Proxy Client
+If you want to write a proxy in another language, you will need to install Thrift and generate
+client code for you language using `lib/proxy/thrift/proxy.thrift`.  See the [Thrift Tutorial][tutorial]
+to how generate source from a thrift file.
 
-The following examples have been written in Java and the method signatures may be
-slightly different depending on the language specified when generating client with
-the Thrift compiler. After initiating a connection to the Proxy (see Apache Thrift's
-documentation for examples of connecting to a Thrift service), the methods on the
-proxy client will be available. The first thing to do is log in:
+### Thrift library
+
+Language-specific Thrift libraries can be installed using an OS or language package manager
(i.e gem, pip, etc).
+For example, `pip install thrift` will install Python-specific thrift libaries on your machine.
+
+## Proxy Client Examples
+
+The following examples show proxy clients written in Java, Ruby, and Python.
+
+### Java
+
+After initiating a connection to the Proxy (see Apache Thrift's documentation for examples
+of connecting to a Thrift service), the methods on the proxy client will be available. The
+first thing to do is log in:
 
 ```java
 Map password = new HashMap<String,String>();
@@ -117,4 +135,250 @@ for(KeyValue keyValue : results.getResultsIterator()) {
 client.closeScanner(scanner);
 ```
 
+### Ruby
+
+The example ruby code below can be run using the following command (the -I option is needed
for ruby 1.9.x):
+
+    ruby -I . client.rb <host of server>
+
+**Warning:** The script will connect to Accumulo, create a table, and add some rows to it.
+
+```ruby
+require 'rubygems'
+require 'thrift'
+require 'accumulo_proxy'
+
+server = ARGV[0] || 'localhost'
+
+socket = Thrift::Socket.new(server, 42424, 9001)
+transport = Thrift::FramedTransport.new(socket)
+proto = Thrift::CompactProtocol.new(transport)
+proxy = Accumulo::AccumuloProxy::Client.new(proto)
+
+# open up the connect
+transport.open()
+
+# Test if the server is up
+login = proxy.login('root', {'password' => 'secret'})
+
+# print out a table list
+puts "List of tables: #{proxy.listTables(login).inspect}"
+
+testtable = "rubytest"
+proxy.createTable(login, testtable, true, Accumulo::TimeType::MILLIS) unless proxy.tableExists(login,testtable)

+
+update1 = Accumulo::ColumnUpdate.new({'colFamily' => "cf1", 'colQualifier' => "cq1",
'value'=> "a"})
+update2 = Accumulo::ColumnUpdate.new({'colFamily' => "cf2", 'colQualifier' => "cq2",
'value'=> "b"})
+proxy.updateAndFlush(login,testtable,{'row1' => [update1,update2]})
+
+cookie = proxy.createScanner(login,testtable,nil)
+result = proxy.nextK(cookie,10)
+result.results.each{ |keyvalue| puts "Key: #{keyvalue.key.inspect} Value: #{keyvalue.value}"
}
+```
+
+### Python
+
+The example python client code below (if saved to TestClient.py) can be run using the following
command:
+
+    PYTHONPATH=/path/to/accumulo-{{ page.latest_release }}/lib/proxy/gen-py python TestClient.py
+
+As a warning, this script will create a table in your Accumulo instance and add a few cells
to it.
+
+```python
+#! /usr/bin/env python
+
+import sys
+
+from thrift import Thrift
+from thrift.transport import TSocket
+from thrift.transport import TTransport
+from thrift.protocol import TCompactProtocol
+
+from accumulo import AccumuloProxy
+from accumulo.ttypes import *
+
+transport = TSocket.TSocket('localhost', 42424)
+transport = TTransport.TFramedTransport(transport)
+protocol = TCompactProtocol.TCompactProtocol(transport)
+client = AccumuloProxy.Client(protocol)
+transport.open()
+
+login = client.login('root', {'password':'secret'})
+
+print client.listTables(login)
+
+testtable = "pythontest"
+if not client.tableExists(login, testtable):
+    client.createTable(login, testtable, True, TimeType.MILLIS)
+
+row1 = {'a':[ColumnUpdate('a','a',value='value1'), ColumnUpdate('b','b',value='value2')]}
+client.updateAndFlush(login, testtable, row1)
+
+cookie = client.createScanner(login, testtable, None)
+for entry in client.nextK(cookie, 10).results:
+   print entry
+```
+
+The example code below shows proxy client code for managing namespaces:
+
+```python
+#! /usr/bin/env python
+
+from thrift.protocol import TCompactProtocol
+from thrift.transport import TSocket, TTransport
+
+from proxy import AccumuloProxy
+from proxy.ttypes import NamespacePermission, IteratorSetting, IteratorScope, AccumuloException
+
+def main():
+    transport = TSocket.TSocket('localhost', 42424)
+    transport = TTransport.TFramedTransport(transport)
+    protocol = TCompactProtocol.TCompactProtocol(transport)
+    client = AccumuloProxy.Client(protocol)
+    transport.open()
+    login = client.login('root', {'password': 'password'})
+
+    client.createLocalUser(login, 'user1', 'password1')
+
+    print client.listNamespaces(login)
+
+    # create a namespace and give the user1 all permissions
+    print 'creating namespace testing'
+    client.createNamespace(login, 'testing')
+    assert client.namespaceExists(login, 'testing')
+    print client.listNamespaces(login)
+
+    print 'testing namespace renaming'
+    client.renameNamespace(login, 'testing', 'testing2')
+    assert not client.namespaceExists(login, 'testing')
+    assert client.namespaceExists(login, 'testing2')
+    client.renameNamespace(login, 'testing2', 'testing')
+    assert not client.namespaceExists(login, 'testing2')
+    assert client.namespaceExists(login, 'testing')
+
+    print 'granting all namespace permissions to user1'
+    for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
+        client.grantNamespacePermission(login, 'user1', 'testing', k)
+
+    # make sure the last operation worked
+    for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
+        assert client.hasNamespacePermission(login, 'user1', 'testing', k), \
+            'user1 does\'nt have namespace permission %s' % v
+
+    print 'default namespace: ' + client.defaultNamespace()
+    print 'system namespace: ' + client.systemNamespace()
+
+    # grab the namespace properties
+    print 'retrieving namespace properties'
+    props = client.getNamespaceProperties(login, 'testing')
+    assert props and props['table.compaction.major.ratio'] == '3'
+
+    # update a property and verify it is good
+    print 'setting namespace property table.compaction.major.ratio = 4'
+    client.setNamespaceProperty(login, 'testing', 'table.compaction.major.ratio', '4')
+    props = client.getNamespaceProperties(login, 'testing')
+    assert props and props['table.compaction.major.ratio'] == '4'
+
+    print 'retrieving namespace ID map'
+    nsids = client.namespaceIdMap(login)
+    assert nsids and 'accumulo' in nsids
+
+    print 'attaching debug iterator to namespace testing'
+    setting = IteratorSetting(priority=40, name='DebugTheThings',
+                              iteratorClass='org.apache.accumulo.core.iterators.DebugIterator',
properties={})
+    client.attachNamespaceIterator(login, 'testing', setting, [IteratorScope.SCAN])
+    setting = client.getNamespaceIteratorSetting(login, 'testing', 'DebugTheThings', IteratorScope.SCAN)
+    assert setting and setting.name == 'DebugTheThings'
+
+    # make sure the iterator is in the list
+    iters = client.listNamespaceIterators(login, 'testing')
+    found = False
+    for name, scopes in iters.iteritems():
+        if name == 'DebugTheThings':
+            found = True
+            break
+    assert found
+
+    print 'checking for iterator conflicts'
+
+    # this next statment should be fine since we are on a different scope
+    client.checkNamespaceIteratorConflicts(login, 'testing', setting, [IteratorScope.MINC])
+
+    # this time it should throw an exception since we have already added the iterator with
this scope
+    try:
+        client.checkNamespaceIteratorConflicts(login, 'testing', setting, [IteratorScope.SCAN,
IteratorScope.MINC])
+    except AccumuloException:
+        pass
+    else:
+        assert False, 'There should have been a namespace iterator conflict!'
+
+    print 'removing debug iterator from namespace testing'
+    client.removeNamespaceIterator(login, 'testing', 'DebugTheThings', [IteratorScope.SCAN])
+
+    # make sure the iterator is NOT in the list anymore
+    iters = client.listNamespaceIterators(login, 'testing')
+    found = False
+    for name, scopes in iters.iteritems():
+        if name == 'DebugTheThings':
+            found = True
+            break
+    assert not found
+
+    print 'adding max mutation size namespace constraint'
+    constraintid = client.addNamespaceConstraint(login, 'testing',
+                                                 'org.apache.accumulo.test.constraints.MaxMutationSize')
+
+    print 'make sure constraint was added'
+    constraints = client.listNamespaceConstraints(login, 'testing')
+    found = False
+    for name, cid in constraints.iteritems():
+        if cid == constraintid and name == 'org.apache.accumulo.test.constraints.MaxMutationSize':
+            found = True
+            break
+    assert found
+
+    print 'remove max mutation size namespace constraint'
+    client.removeNamespaceConstraint(login, 'testing', constraintid)
+
+    print 'make sure constraint was removed'
+    constraints = client.listNamespaceConstraints(login, 'testing')
+    found = False
+    for name, cid in constraints.iteritems():
+        if cid == constraintid and name == 'org.apache.accumulo.test.constraints.MaxMutationSize':
+            found = True
+            break
+    assert not found
+
+    print 'test a namespace class load of the VersioningIterator'
+    res = client.testNamespaceClassLoad(login, 'testing', 'org.apache.accumulo.core.iterators.user.VersioningIterator',
+                                        'org.apache.accumulo.core.iterators.SortedKeyValueIterator')
+    assert res
+
+    print 'test a bad namespace class load of the VersioningIterator'
+    res = client.testNamespaceClassLoad(login, 'testing', 'org.apache.accumulo.core.iterators.user.VersioningIterator',
+                                        'dummy')
+    assert not res
+
+    # revoke the permissions
+    print 'revoking namespace permissions for user1'
+    for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
+        client.revokeNamespacePermission(login, 'user1', 'testing', k)
+
+    # make sure the last operation worked
+    for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
+        assert not client.hasNamespacePermission(login, 'user1', 'testing', k), \
+            'user1 does\'nt have namespace permission %s' % v
+
+    print 'deleting namespace testing'
+    client.deleteNamespace(login, 'testing')
+    assert not client.namespaceExists(login, 'testing')
+
+    print 'deleting user1'
+    client.dropLocalUser(login, 'user1')
+
+if __name__ == "__main__":
+    main()
+```
+
 [ZookeeperInstance]: {{ page.javadoc_core }}/org/apache/accumulo/core/client/ZooKeeperInstance.html
+[tutorial]: https://thrift.apache.org/tutorial/


Mime
View raw message