Return-Path: X-Original-To: apmail-gora-commits-archive@www.apache.org Delivered-To: apmail-gora-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 716A1D8C6 for ; Sat, 3 Nov 2012 21:22:38 +0000 (UTC) Received: (qmail 50534 invoked by uid 500); 3 Nov 2012 21:22:38 -0000 Delivered-To: apmail-gora-commits-archive@gora.apache.org Received: (qmail 50510 invoked by uid 500); 3 Nov 2012 21:22:38 -0000 Mailing-List: contact commits-help@gora.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@gora.apache.org Delivered-To: mailing list commits@gora.apache.org Received: (qmail 50502 invoked by uid 99); 3 Nov 2012 21:22:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Nov 2012 21:22:38 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Nov 2012 21:22:27 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id D465B2388A64; Sat, 3 Nov 2012 21:22:04 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1405419 [1/2] - in /gora/trunk/gora-dynamodb: ./ src/ src/examples/ src/examples/java/ src/examples/java/org/ src/examples/java/org/apache/ src/examples/java/org/apache/gora/ src/examples/java/org/apache/gora/examples/ src/examples/java/or... Date: Sat, 03 Nov 2012 21:22:04 -0000 To: commits@gora.apache.org From: lewismc@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20121103212204.D465B2388A64@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: lewismc Date: Sat Nov 3 21:22:02 2012 New Revision: 1405419 URL: http://svn.apache.org/viewvc?rev=1405419&view=rev Log: GORA-103 Datastore for gora-dynamodb Added: gora/trunk/gora-dynamodb/ gora/trunk/gora-dynamodb/pom.xml gora/trunk/gora-dynamodb/src/ gora/trunk/gora-dynamodb/src/examples/ gora/trunk/gora-dynamodb/src/examples/java/ gora/trunk/gora-dynamodb/src/examples/java/org/ gora/trunk/gora-dynamodb/src/examples/java/org/apache/ gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/ gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/examples/ gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/examples/generated/ gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/examples/generated/person.java gora/trunk/gora-dynamodb/src/main/ gora/trunk/gora-dynamodb/src/main/java/ gora/trunk/gora-dynamodb/src/main/java/org/ gora/trunk/gora-dynamodb/src/main/java/org/apache/ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/compiler/ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/compiler/GoraDynamoDBCompiler.java gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBKey.java gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBQuery.java gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBResult.java gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBMapping.java gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBStore.java gora/trunk/gora-dynamodb/src/test/ gora/trunk/gora-dynamodb/src/test/conf/ gora/trunk/gora-dynamodb/src/test/conf/AwsCredentials.properties gora/trunk/gora-dynamodb/src/test/conf/gora-dynamodb-mapping.xml gora/trunk/gora-dynamodb/src/test/conf/gora.properties gora/trunk/gora-dynamodb/src/test/java/ gora/trunk/gora-dynamodb/src/test/java/org/ gora/trunk/gora-dynamodb/src/test/java/org/apache/ gora/trunk/gora-dynamodb/src/test/java/org/apache/gora/ gora/trunk/gora-dynamodb/src/test/java/org/apache/gora/dynamodb/ gora/trunk/gora-dynamodb/src/test/java/org/apache/gora/dynamodb/GoraDynamoDBTestDriver.java gora/trunk/gora-dynamodb/src/test/java/org/apache/gora/dynamodb/TestDynamoDBStore.java Added: gora/trunk/gora-dynamodb/pom.xml URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/pom.xml?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/pom.xml (added) +++ gora/trunk/gora-dynamodb/pom.xml Sat Nov 3 21:22:02 2012 @@ -0,0 +1,150 @@ + + + + 4.0.0 + + org.apache.gora + gora + 0.3-SNAPSHOT + ../ + + gora-dynamodb + bundle + Apache Gora :: Dynamodb + http://gora.apache.org + The Apache Gora open source framework provides an in-memory data model and + persistence for big data. Gora supports persisting to column stores, key value stores, + document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce + support. + 2010 + + The Apache Software Foundation + http://www.apache.org/ + + + http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/ + scm:svn:http://svn.apache.org/repos/asf/gora/trunk/gora-dynamodb/ + scm:svn:https://svn.apache.org/repos/asf/gora/trunk/gora-dynamodb/ + + + JIRA + https://issues.apache.org/jira/browse/GORA + + + Jenkins + https://builds.apache.org/job/Gora-trunk/ + + + * + org.apache.gora.dynamodb*;version="${project.version}";-noimport:=true + + + target + target/classes + ${project.artifactId}-${project.version} + target/test-classes + src/test/java + src/main/java + + + ${basedir}/src/main/resources + + + ${basedir}/conf + + + + + src/test/conf/ + + ** + + + + + + org.codehaus.mojo + build-helper-maven-plugin + ${build-helper-maven-plugin.version} + + + generate-sources + + add-source + + + + src/examples/java + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + ${maven-surfire-plugin.version} + + true + + + + + + + + org.apache.gora + gora-core + + + org.apache.gora + gora-core + test-jar + test + + + + + com.amazonaws + aws-java-sdk + + + + + org.jdom + jdom + + + + + org.slf4j + slf4j-api + + + org.slf4j + slf4j-jdk14 + + + log4j + log4j + + + + junit + junit + + + Added: gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/examples/generated/person.java URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/examples/generated/person.java?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/examples/generated/person.java (added) +++ gora/trunk/gora-dynamodb/src/examples/java/org/apache/gora/examples/generated/person.java Sat Nov 3 21:22:02 2012 @@ -0,0 +1,97 @@ +package org.apache.gora.examples.generated; +import java.util.Set; +import org.apache.gora.persistency.Persistent; +import org.apache.gora.persistency.StateManager; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBAttribute; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBHashKey; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBRangeKey; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBTable; + +@DynamoDBTable(tableName = "person") +public class person implements Persistent { + private String ssn; + private String date; + + @DynamoDBHashKey(attributeName="ssn") + public String getHashKey() { return ssn; } + public void setHashKey(String pSsn){ this.ssn = pSsn; } + @DynamoDBRangeKey(attributeName="date") + public String getRangeKey() { return date; } + public void setRangeKey(String pDate){ this.date = pDate; } + + private String lastName; + @DynamoDBAttribute(attributeName = "LastName") + public String getLastName() { return lastName; } + public void setLastName(String pLastName) { this.lastName = pLastName; } + + private Set visitedplaces; + @DynamoDBAttribute(attributeName = "Visitedplaces") + public Set getVisitedplaces() { return visitedplaces; } + public void setVisitedplaces(Set pVisitedplaces) { this.visitedplaces = pVisitedplaces; } + + private double salary; + @DynamoDBAttribute(attributeName = "Salary") + public double getSalary() { return salary; } + public void setSalary(double pSalary) { this.salary = pSalary; } + + private String firstName; + @DynamoDBAttribute(attributeName = "FirstName") + public String getFirstName() { return firstName; } + public void setFirstName(String pFirstName) { this.firstName = pFirstName; } + + + public void setNew(boolean pNew){} + public void setDirty(boolean pDirty){} + @Override + public StateManager getStateManager() { return null; } + @Override + public Persistent newInstance(StateManager stateManager) { return null; } + @Override + public String[] getFields() { return null; } + @Override + public String getField(int index) { return null; } + @Override + public int getFieldIndex(String field) { return 0; } + @Override + public void clear() { } + @Override + public person clone() { return null; } + @Override + public boolean isNew() { return false; } + @Override + public void setNew() { } + @Override + public void clearNew() { } + @Override + public boolean isDirty() { return false; } + @Override + public boolean isDirty(int fieldIndex) { return false; } + @Override + public boolean isDirty(String field) { return false; } + @Override + public void setDirty() { } + @Override + public void setDirty(int fieldIndex) { } + @Override + public void setDirty(String field) { } + @Override + public void clearDirty(int fieldIndex) { } + @Override + public void clearDirty(String field) { } + @Override + public void clearDirty() { } + @Override + public boolean isReadable(int fieldIndex) { return false; } + @Override + public boolean isReadable(String field) { return false; } + @Override + public void setReadable(int fieldIndex) { } + @Override + public void setReadable(String field) { } + @Override + public void clearReadable(int fieldIndex) { } + @Override + public void clearReadable(String field) { } + @Override + public void clearReadable() { } +} \ No newline at end of file Added: gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/compiler/GoraDynamoDBCompiler.java URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/compiler/GoraDynamoDBCompiler.java?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/compiler/GoraDynamoDBCompiler.java (added) +++ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/compiler/GoraDynamoDBCompiler.java Sat Nov 3 21:22:02 2012 @@ -0,0 +1,405 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.dynamodb.compiler; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.Writer; +import java.util.List; +import java.util.Map; + +import org.apache.gora.dynamodb.store.DynamoDBMapping; +import org.apache.gora.dynamodb.store.DynamoDBMapping.DynamoDBMappingBuilder; +import org.jdom.Document; +import org.jdom.Element; +import org.jdom.input.SAXBuilder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.amazonaws.services.dynamodb.model.KeySchema; +import com.amazonaws.services.dynamodb.model.KeySchemaElement; + +/** Generate specific Java classes for defined schemas. */ +public class GoraDynamoDBCompiler { + private File dest; + private Writer out; + private static final Logger log = LoggerFactory.getLogger(GoraDynamoDBCompiler.class); + + private GoraDynamoDBCompiler(File dest) { + this.dest = dest; // root directory for output + } + + /** Generates Java classes for a schema. */ + public static void compileSchema(File src, File dest) throws IOException { + log.info("Compiling " + src + " to " + dest ); + GoraDynamoDBCompiler compiler = new GoraDynamoDBCompiler(dest); + DynamoDBMapping dynamoDBMap = compiler.readMapping(src); + if (dynamoDBMap.getTables().isEmpty()) throw new IllegalStateException("There are not tables defined."); + + for(String tableName : dynamoDBMap.getTables().keySet()){ + compiler.compile(tableName, dynamoDBMap.getKeySchema(tableName), dynamoDBMap.getItems(tableName)); + } + } + + /** + * Method in charge of compiling a specific table using a key schema and a set of attributes + * @param pTableName Table name + * @param pKeySchema Key schema used + * @param pItems List of items belonging to a specific table + */ + private void compile(String pTableName, KeySchema pKeySchema, List> pItems){ + // TODO define where the generated will go + try { + startFile(pTableName, pTableName); + setHeaders(null); + line(0, ""); + line(0, "@DynamoDBTable(tableName = \"" + pTableName + "\")"); + line(0, "public class " + pTableName + " implements Persistent {"); + setKeyAttributes(pKeySchema, 2); + setKeyMethods(pKeySchema, 2); + setItems(pItems, 2); + setDefaultMethods(2); + line(0, "}"); + out.flush(); + out.close(); + } catch (IOException e) { + log.error("Error while compiling table " + pTableName); + e.printStackTrace(); + } + } + + /** + * Receives a list of all items and creates getters and setters for them + * @param pItems The items belonging to the table + * @param pIden The number of spaces used for identation + * @throws IOException + */ + private void setItems(List> pItems, int pIden) throws IOException{ + for(Map item : pItems){ + for (String itemName : item.keySet()){ + String itemType = "String"; + if (item.get(itemName).toString().equals("N")) + itemType = "double"; + if (item.get(itemName).toString().equals("SS")) + itemType = "Set"; + if (item.get(itemName).toString().equals("SN")) + itemType = "Set"; + line(pIden, "private " + itemType + " " + itemName + ";"); + setItemMethods(itemName, itemType, pIden); + } + } + line(0, ""); + } + + /** + * Creates item getters and setters + * @param pItemName Item's name + * @param pItemType Item's type + * @param pIden Number of spaces used for indentation + * @throws IOException + */ + private void setItemMethods(String pItemName, String pItemType, int pIden) throws IOException{ + line(pIden, "@DynamoDBAttribute(attributeName = \"" + camelCasify(pItemName) + "\")"); + line(pIden, "public " + pItemType + " get" + camelCasify(pItemName) + "() { return " + pItemName + "; }"); + line(pIden, "public void set" + camelCasify(pItemName) + "(" + pItemType + " p" + camelCasify(pItemName) + ") { this." + pItemName + " = p"+ camelCasify(pItemName) +"; }"); + line(0, ""); + } + + /** + * Creates key getters and setters + * @param pKeySchema The key schema for a specific table + * @param pIden Number of spaces used for indentation + * @throws IOException + */ + private void setKeyMethods(KeySchema pKeySchema, int pIden) throws IOException{ + KeySchemaElement hashKey = pKeySchema.getHashKeyElement(); + KeySchemaElement rangeKey = pKeySchema.getRangeKeyElement(); + StringBuilder strBuilder = new StringBuilder(); + // hash key + if(hashKey != null){ + strBuilder.append("@DynamoDBHashKey(attributeName=\"" + hashKey.getAttributeName() + "\") \n"); + strBuilder.append(" public String getHashKey() { return " + hashKey.getAttributeName() + "; } \n"); + strBuilder.append(" public void setHashKey(" + (hashKey.getAttributeType().equals("S")?"String ":"double ")); + strBuilder.append("p" + camelCasify(hashKey.getAttributeName()) + "){ this." + hashKey.getAttributeName()); + strBuilder.append(" = p" + camelCasify(hashKey.getAttributeName()) + "; }"); + line(pIden, strBuilder.toString()); + } + strBuilder.delete(0, strBuilder.length()); + // range key + if(rangeKey != null){ + strBuilder.append("@DynamoDBRangeKey(attributeName=\"" + rangeKey.getAttributeName() + "\") \n"); + strBuilder.append(" public String getRangeKey() { return " + rangeKey.getAttributeName() + "; } \n"); + strBuilder.append(" public void setRangeKey(" + (rangeKey.getAttributeType().equals("S")?"String ":"double ")); + strBuilder.append("p" + camelCasify(rangeKey.getAttributeName()) + "){ this." + rangeKey.getAttributeName()); + strBuilder.append(" = p" + camelCasify(rangeKey.getAttributeName()) + "; }"); + line(pIden, strBuilder.toString()); + } + line(0, ""); + } + + /** + * Creates the key attributes within the generated class + * @param pKeySchema Key schema + * @param pIden Number of spaces used for indentation + * @throws IOException + */ + private void setKeyAttributes(KeySchema pKeySchema, int pIden) throws IOException{ + KeySchemaElement hashKey = pKeySchema.getHashKeyElement(); + KeySchemaElement rangeKey = pKeySchema.getRangeKeyElement(); + StringBuilder strBuilder = new StringBuilder(); + // hash key + if(hashKey != null){ + strBuilder.append("private " + (hashKey.getAttributeType().equals("S")?"String ":"double ")); + strBuilder.append(hashKey.getAttributeName() + ";"); + line(pIden, strBuilder.toString()); + } + strBuilder.delete(0, strBuilder.length()); + // range key + if(rangeKey != null){ + strBuilder.append("private " + (rangeKey.getAttributeType().equals("S")?"String ":"double ")); + strBuilder.append(rangeKey.getAttributeName() + ";"); + line(pIden, strBuilder.toString()); + } + line(0, ""); + } + + /** + * Returns camel case version of a string + * @param s String to be camelcasified + * @return + */ + private static String camelCasify(String s) { + return s.substring(0, 1).toUpperCase() + s.substring(1); + } + + /** Recognizes camel case */ + private static String toUpperCase(String s) { + StringBuilder builder = new StringBuilder(); + for(int i=0; i 0) { + if(Character.isUpperCase(s.charAt(i)) + && Character.isLowerCase(s.charAt(i-1)) + && Character.isLetter(s.charAt(i))) { + builder.append("_"); + } + } + builder.append(Character.toUpperCase(s.charAt(i))); + } + return builder.toString(); + } + + /** + * Starts the java generated class file + * @param name Class name + * @param space + * @throws IOException + */ + private void startFile(String name, String space) throws IOException { + File dir = new File(dest, space.replace('.', File.separatorChar)); + if (!dir.exists()) + if (!dir.mkdirs()) + throw new IOException("Unable to create " + dir); + name = cap(name) + ".java"; + out = new OutputStreamWriter(new FileOutputStream(new File(dir, name))); + + } + + /** + * Sets the necessary imports for the generated java class to work + * @param namespace + * @throws IOException + */ + private void setHeaders(String namespace) throws IOException { + if(namespace != null) { + line(0, "package "+namespace+";\n"); + } + line(0, "import java.util.Set;"); + line(0, "import org.apache.gora.persistency.Persistent;"); + line(0, "import org.apache.gora.persistency.StateManager;"); + line(0, "import com.amazonaws.services.dynamodb.datamodeling.DynamoDBAttribute;"); + line(0, "import com.amazonaws.services.dynamodb.datamodeling.DynamoDBHashKey;"); + line(0, "import com.amazonaws.services.dynamodb.datamodeling.DynamoDBRangeKey;"); + line(0, "import com.amazonaws.services.dynamodb.datamodeling.DynamoDBTable;"); + } + + /** + * Creates default methods inherited from upper classes + * @param pIden Number of spaces used for indentation + * @throws IOException + */ + private void setDefaultMethods(int pIden) throws IOException { + line(pIden, "public void setNew(boolean pNew){}"); + line(pIden, "public void setDirty(boolean pDirty){}"); + line(pIden, "@Override"); + line(pIden, "public StateManager getStateManager() { return null; }"); + line(pIden, "@Override"); + line(pIden, "public Persistent newInstance(StateManager stateManager) { return null; }"); + line(pIden, "@Override"); + line(pIden, "public String[] getFields() { return null; }"); + line(pIden, "@Override"); + line(pIden, "public String getField(int index) { return null; }"); + line(pIden, "@Override"); + line(pIden, "public int getFieldIndex(String field) { return 0; }"); + line(pIden, "@Override"); + line(pIden, "public void clear() { }"); + line(pIden, "@Override"); + line(pIden, "public person clone() { return null; }"); + line(pIden, "@Override"); + line(pIden, "public boolean isNew() { return false; }"); + line(pIden, "@Override"); + line(pIden, "public void setNew() { }"); + line(pIden, "@Override"); + line(pIden, "public void clearNew() { }"); + line(pIden, "@Override"); + line(pIden, "public boolean isDirty() { return false; }"); + line(pIden, "@Override"); + line(pIden, "public boolean isDirty(int fieldIndex) { return false; }"); + line(pIden, "@Override"); + line(pIden, "public boolean isDirty(String field) { return false; }"); + line(pIden, "@Override"); + line(pIden, "public void setDirty() { }"); + line(pIden, "@Override"); + line(pIden, "public void setDirty(int fieldIndex) { }"); + line(pIden, "@Override"); + line(pIden, "public void setDirty(String field) { }"); + line(pIden, "@Override"); + line(pIden, "public void clearDirty(int fieldIndex) { }"); + line(pIden, "@Override"); + line(pIden, "public void clearDirty(String field) { }"); + line(pIden, "@Override"); + line(pIden, "public void clearDirty() { }"); + line(pIden, "@Override"); + line(pIden, "public boolean isReadable(int fieldIndex) { return false; }"); + line(pIden, "@Override"); + line(pIden, "public boolean isReadable(String field) { return false; }"); + line(pIden, "@Override"); + line(pIden, "public void setReadable(int fieldIndex) { }"); + line(pIden, "@Override"); + line(pIden, "public void setReadable(String field) { }"); + line(pIden, "@Override"); + line(pIden, "public void clearReadable(int fieldIndex) { }"); + line(pIden, "@Override"); + line(pIden, "public void clearReadable(String field) { }"); + line(pIden, "@Override"); + line(pIden, "public void clearReadable() { }"); + } + + /** + * Writes a line within the output stream + * @param indent Number of spaces used for indentation + * @param text Text to be written + * @throws IOException + */ + private void line(int indent, String text) throws IOException { + for (int i = 0; i < indent; i ++) { + out.append(" "); + } + out.append(text); + out.append("\n"); + } + + /** + * Returns the string received with the first letter in uppercase + * @param name String to be converted + * @return + */ + static String cap(String name) { + return name.substring(0,1).toUpperCase()+name.substring(1,name.length()); + } + + /** + * Start point of the compiler program + * @param args Receives the schema file to be compiled and where this should be written + * @throws Exception + */ + public static void main(String[] args) throws Exception { + if (args.length < 2) { + System.err.println("Usage: Compiler "); + System.exit(1); + } + compileSchema(new File(args[0]), new File(args[1])); + } + + /** + * Reads the schema file and converts it into a data structure to be used + * @param pMapFile The schema file to be mapped into a table + * @return + * @throws IOException + */ + @SuppressWarnings("unchecked") + private DynamoDBMapping readMapping(File pMapFile) throws IOException { + + DynamoDBMappingBuilder mappingBuilder = new DynamoDBMappingBuilder(); + + try { + SAXBuilder builder = new SAXBuilder(); + Document doc = builder.build(pMapFile); + + Element root = doc.getRootElement(); + + List tableElements = root.getChildren("table"); + for(Element tableElement : tableElements) { + + String tableName = tableElement.getAttributeValue("name"); + long readCapacUnits = Long.parseLong(tableElement.getAttributeValue("readcunit")); + long writeCapacUnits = Long.parseLong(tableElement.getAttributeValue("readcunit")); + + mappingBuilder.setTableName(tableName); + mappingBuilder.setProvisionedThroughput(tableName, readCapacUnits, writeCapacUnits); + log.debug("Basic table properties have been set: Name, and Provisioned throughput."); + + // Retrieving key's features + List fieldElements = tableElement.getChildren("key"); + for(Element fieldElement : fieldElements) { + String keyName = fieldElement.getAttributeValue("name"); + String keyType = fieldElement.getAttributeValue("type"); + String keyAttrType = fieldElement.getAttributeValue("att-type"); + if(keyType.equals("hash")) + mappingBuilder.setHashKeySchema(tableName, keyName, keyAttrType); + else if(keyType.equals("hashrange")) + mappingBuilder.setHashRangeKeySchema(tableName, keyName, keyAttrType); + } + log.debug("Table key schemas have been set."); + + // Retrieving attributes + fieldElements = tableElement.getChildren("attribute"); + for(Element fieldElement : fieldElements) { + String attributeName = fieldElement.getAttributeValue("name"); + String attributeType = fieldElement.getAttributeValue("type"); + mappingBuilder.addAttribute(tableName, attributeName, attributeType, 0); + } + log.info("Table attributes have been read."); + } + + } catch(IOException ex) { + log.info("Error while performing xml mapping."); + ex.printStackTrace(); + throw ex; + + } catch(Exception ex) { + ex.printStackTrace(); + throw new IOException(ex); + } + + return mappingBuilder.build(); + } +} + Added: gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBKey.java URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBKey.java?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBKey.java (added) +++ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBKey.java Sat Nov 3 21:22:02 2012 @@ -0,0 +1,64 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.dynamodb.query; + +public class DynamoDBKey { + + /** + * Hash key used for a specific table + */ + private H hashKey; + + /** + * Range key used for a specific table + */ + private R rangeKey; + + /** + * Gets hash key + * @return + */ + public H getHashKey() { + return hashKey; + } + + /** + * Sets hash key + * @param hashKey + */ + public void setHashKey(H hashKey) { + this.hashKey = hashKey; + } + + /** + * Gets range key + * @return + */ + public R getRangeKey() { + return rangeKey; + } + + /** + * Sets range key + * @param rangeKey + */ + public void setRangeKey(R rangeKey) { + this.rangeKey = rangeKey; + } +} Added: gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBQuery.java URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBQuery.java?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBQuery.java (added) +++ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBQuery.java Sat Nov 3 21:22:02 2012 @@ -0,0 +1,380 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.dynamodb.query; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; + +import org.apache.gora.persistency.Persistent; +import org.apache.gora.query.Query; +import org.apache.gora.query.ws.impl.QueryWSBase; +import org.apache.gora.store.DataStore; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBQueryExpression; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBScanExpression; +import com.amazonaws.services.dynamodb.model.AttributeValue; +import com.amazonaws.services.dynamodb.model.ComparisonOperator; +import com.amazonaws.services.dynamodb.model.Condition; +import com.amazonaws.services.dynamodb.model.KeySchema; +import com.amazonaws.services.dynamodb.model.KeySchemaElement; + +public class DynamoDBQuery extends QueryWSBase { + + /** + * Helper to write useful information into the logs + */ + public static final Logger LOG = LoggerFactory.getLogger(DynamoDBQuery.class); + + /** + * Reads consistency level + */ + private boolean consistencyReadLevel; + + /** + * Range comparator operator + */ + private static ComparisonOperator rangeCompOp; + + /** + * Scan comparator operator + */ + private static ComparisonOperator scanCompOp; + + /** + * Range query type property + */ + public static final String RANGE_QUERY = "range"; + + /** + * Scan query type property + */ + public static final String SCAN_QUERY = "scan"; + + /** + * Query type property + */ + private static String type; + + /** + * Generic query + */ + private Query query; + + /** + * DynamoDB Expression to be used. + * This could be a range or a scan DynamoDB Expression + */ + private Object dynamoDBExpression; + + /** + * Key schema used for the query + */ + private KeySchema keySchema; + + /** + * Hash key used for the query + */ + private K hashKey; + + /** + * Default Constructor + */ + public DynamoDBQuery(){ + super(null); + } + + /** + * Constructor + * @param dataStore + */ + public DynamoDBQuery(DataStore dataStore) { + super(dataStore); + } + + /** + * Sets hash key + */ + @Override + public void setKey(K key) { + this.hashKey = key; + } + + /** + * Gets hash key + */ + @Override + public K getKey() { + return this.hashKey; + } + + /** + * Builds query expression depending on query type (range or scan) + */ + public void buildExpression(){ + AttributeValue hashAttrValue = buildKeyHashAttribute(); + if (hashAttrValue == null) + throw new IllegalStateException("There is not a key schema defined."); + if (DynamoDBQuery.getType().equals(RANGE_QUERY)){ + Condition newCondition = buildRangeCondition(); + buildQueryExpression(newCondition, hashAttrValue); + } + if (DynamoDBQuery.getType().equals(SCAN_QUERY)) + buildScanExpression(hashAttrValue); + } + + /** + * Builds scan query expression using a hash attribute value where to start + * @param pHashAttrValue Hash attribute value where to start scanning + */ + public void buildScanExpression(AttributeValue pHashAttrValue){ + DynamoDBScanExpression newScanExpression = new DynamoDBScanExpression(); + // TODO right now we only support scanning using the key, but we should support other types of scans + newScanExpression.addFilterCondition(getKeySchema().getHashKeyElement().getAttributeName(), buildKeyScanCondition()); + dynamoDBExpression = newScanExpression; + } + + /** + * Builds range query expression + * @param pNewCondition Condition for querying + * @param pHashAttrValue Hash attribute value where to start + */ + public void buildQueryExpression(Condition pNewCondition, AttributeValue pHashAttrValue) { + DynamoDBQueryExpression newQueryExpression = new DynamoDBQueryExpression(pHashAttrValue); + newQueryExpression.setConsistentRead(getConsistencyReadLevel()); + newQueryExpression.setRangeKeyCondition(pNewCondition); + dynamoDBExpression = newQueryExpression; + } + + /** + * Builds hash key attribute from generic query received + * @return AttributeValue build from query + */ + private AttributeValue buildKeyHashAttribute(){ + String pAttrType = getKeySchema().getHashKeyElement().getAttributeType(); + if(pAttrType.equals("S")) + return new AttributeValue().withS(getHashKey(query.getKey()).toString()); + else if(pAttrType.equals("N")) + return new AttributeValue().withN(getHashKey(query.getKey()).toString()); + return null; + } + + /** + * Gets hash key for querying + * @param key + * @return + */ + private Object getHashKey(K key){ + Object hashKey = null; + try { + // Our key may be have hash and range keys + for (Method met :key.getClass().getDeclaredMethods()){ + if(met.getName().equals("getHashKey")){ + Object [] params = null; + hashKey = met.invoke(key, params); + break; + } + } + } catch (IllegalArgumentException e) { + LOG.info("DynamoDBStore: Error while trying to fetch range key."); + e.printStackTrace(); + } catch (IllegalAccessException e) { + LOG.info("DynamoDBStore: Error while trying to fetch range key."); + e.printStackTrace(); + } catch (InvocationTargetException e) { + LOG.info("DynamoDBStore: Error while trying to fetch range key."); + e.printStackTrace(); + } + return hashKey; + } + + /** + * Gets range key for querying from generic query object received + * @param key + * @return + */ + private Object getRangeKey(K key){ + Object rangeKey = null; + try { + // Our key may be have hash and range keys + for (Method met :key.getClass().getDeclaredMethods()){ + if(met.getName().equals("getRangeKey")){ + Object [] params = null; + rangeKey = met.invoke(key, params); + break; + } + } + } catch (IllegalArgumentException e) { + LOG.info("DynamoDBStore: Error while trying to fetch range key."); + e.printStackTrace(); + } catch (IllegalAccessException e) { + LOG.info("DynamoDBStore: Error while trying to fetch range key."); + e.printStackTrace(); + } catch (InvocationTargetException e) { + LOG.info("DynamoDBStore: Error while trying to fetch range key."); + e.printStackTrace(); + } + return rangeKey; + } + + /** + * Builds key scan condition using scan comparator, and hash key attribute + * @return + */ + private Condition buildKeyScanCondition(){ + Condition scanKeyCondition = new Condition(); + scanKeyCondition.setComparisonOperator(getScanCompOp()); + scanKeyCondition.withAttributeValueList(buildKeyHashAttribute()); + return scanKeyCondition; + } + + /** + * Builds range condition based on elements set + * @return + */ + private Condition buildRangeCondition(){ + KeySchemaElement kRangeSchema = getKeySchema().getRangeKeyElement(); + Condition rangeKeyCondition = null; + if(kRangeSchema != null){ + rangeKeyCondition = new Condition(); + rangeKeyCondition.setComparisonOperator(ComparisonOperator.BETWEEN.toString()); + AttributeValue startVal = null, endVal = null; + //startVal = buildKeyHashAttribute(); + if(kRangeSchema.getAttributeType().equals("S")){ + startVal = new AttributeValue().withS(getRangeKey(query.getStartKey()).toString()); + endVal = new AttributeValue().withS(getRangeKey(query.getEndKey()).toString()); + } + else if (kRangeSchema.getAttributeType().equals("N")){ + startVal = new AttributeValue().withN(getRangeKey(query.getStartKey()).toString()); + endVal = new AttributeValue().withN(getRangeKey(query.getEndKey()).toString()); + } + rangeKeyCondition.withAttributeValueList(startVal, endVal); + } + return rangeKeyCondition; + } + + /** + * Gets read consistency level + * @return + */ + public boolean getConsistencyReadLevel(){ + return consistencyReadLevel; + } + + /** + * Sets read consistency level + * @param pConsistencyReadLevel + */ + public void setConsistencyReadLevel(boolean pConsistencyReadLevel){ + this.consistencyReadLevel = pConsistencyReadLevel; + } + + /** + * Gets key schema + * @return + */ + public KeySchema getKeySchema(){ + return keySchema; + } + + /** + * Gets query expression for query + * @return + */ + public Object getQueryExpression(){ + return dynamoDBExpression; + } + + /** + * Sets query key schema used for queying + * @param pKeySchema + */ + public void setKeySchema(KeySchema pKeySchema){ + this.keySchema = pKeySchema; + } + + /** + * Sets query to be performed + * @param pQuery + */ + public void setQuery(Query pQuery){ + this.query = pQuery; + } + + /** + * Gets query performed + * @return + */ + public Query getQuery(){ + return this.query; + } + + /** + * Gets query type + * @return + */ + public static String getType() { + return type; + } + + /** + * Sets query type + * @param pType + */ + public static void setType(String pType) { + type = pType; + } + + /** + * Gets scan comparator operator + * @return + */ + public static ComparisonOperator getScanCompOp() { + if (scanCompOp == null) + scanCompOp = ComparisonOperator.GE; + return scanCompOp; + } + + /** + * Sets scan query comparator operator + * @param scanCompOp + */ + public static void setScanCompOp(ComparisonOperator scanCompOp) { + DynamoDBQuery.scanCompOp = scanCompOp; + } + + /** + * Gets range query comparator operator + * @return + */ + public static ComparisonOperator getRangeCompOp(){ + if (rangeCompOp == null) + rangeCompOp = ComparisonOperator.BETWEEN; + return rangeCompOp; + } + + /** + * Sets range query comparator operator + * @param pRangeCompOp + */ + public static void setRangeCompOp(ComparisonOperator pRangeCompOp){ + rangeCompOp = pRangeCompOp; + } +} Added: gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBResult.java URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBResult.java?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBResult.java (added) +++ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/query/DynamoDBResult.java Sat Nov 3 21:22:02 2012 @@ -0,0 +1,88 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.dynamodb.query; + +import java.io.IOException; +import java.util.List; + +import org.apache.gora.persistency.Persistent; +import org.apache.gora.query.Query; +import org.apache.gora.query.ws.impl.ResultWSBase; +import org.apache.gora.store.DataStore; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class DynamoDBResult extends ResultWSBase { + + /** + * Helper to write useful information into the logs + */ + public static final Logger LOG = LoggerFactory.getLogger(DynamoDBResult.class); + + /** + * Result set containing query results + */ + private List dynamoDBResultSet; + + /** + * Constructor for the result set + * @param dataStore Data store used + * @param query Query used + * @param objList Objects obtained from querying + */ + public DynamoDBResult(DataStore dataStore, Query query, List objList) { + super(dataStore, query); + LOG.debug("DynamoDB result created."); + this.setResultSet(objList); + } + + /** + * Sets the resulting objects within the class + * @param objList + */ + public void setResultSet(List objList) { + this.dynamoDBResultSet = objList; + this.limit = objList.size(); + } + + /** + * Gets the items reading progress + */ + public float getProgress() throws IOException, InterruptedException, Exception { + if (this.limit <= 0 || this.offset <= 0) + return 0; + return this.limit/this.offset; + } + + /** + * Gets the next item + */ + protected boolean nextInner() throws Exception { + if (offset < 0 || offset > ( dynamoDBResultSet.size() - 1)) + return false; + persistent = dynamoDBResultSet.get((int) this.offset); + return true; + } + + @Override + public void close() throws IOException { + // TODO Auto-generated method stub + } + +} Added: gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBMapping.java URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBMapping.java?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBMapping.java (added) +++ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBMapping.java Sat Nov 3 21:22:02 2012 @@ -0,0 +1,310 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.dynamodb.store; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.amazonaws.services.dynamodb.model.KeySchema; +import com.amazonaws.services.dynamodb.model.KeySchemaElement; +import com.amazonaws.services.dynamodb.model.ProvisionedThroughput; + +public class DynamoDBMapping { + + /** + * Helper to write useful information into the logs + */ + public static final Logger LOG = LoggerFactory.getLogger(DynamoDBMapping.class); + + /** + * a map from field name to attribute value + */ + private final Map>> tablesToItems; + + /** + * Maps tables to their own key schemas + */ + private final Map tablesToKeySchemas; + + /** + * Maps tables to their provisioned throughput + */ + private final Map tablesToPrTh; + + /** + * Constructor for DynamoDBMapping + * @param tables Tables mapped. + * @param tablesToKeySchemas KeySchemas used within tables mapped. + * @param provisionedThroughput Provisioned throughput used within tables mapped. + */ + public DynamoDBMapping(Map>> tables, + Map tablesToKeySchemas, + Map provisionedThroughput) { + + this.tablesToItems = tables; + this.tablesToKeySchemas = tablesToKeySchemas; + this.tablesToPrTh = provisionedThroughput; + } + + /** + * Gets the tables with their own items + * @return tablesToItem HashMap + */ + public Map>> getTables(){ + return tablesToItems; + } + + /** + * Gets items or attributes from a specific table + * @param tableName Table name to determine which attributes to get + * @return + */ + public List> getItems(String tableName){ + return tablesToItems.get(tableName); + } + + /** + * Gets the key schema from a specific table + * @param tableName Table name to determine which key schema to get + * @return + */ + public KeySchema getKeySchema(String tableName) { + return tablesToKeySchemas.get(tableName); + } + + /** + * Gets the provisioned throughput from a specific table + * @param tableName Table name to determine which provisioned throughput to get + * @return + */ + public ProvisionedThroughput getProvisionedThroughput(String tableName){ + return tablesToPrTh.get(tableName); + } + + /** + * A builder for creating the mapper. This will allow building a thread safe + * {@link DynamoDBMapping} using simple immutabilty. + * + */ + public static class DynamoDBMappingBuilder { + + /** + * Table name to be used to build the DynamoDBMapping object + */ + private String tableName; + + /** + * This data structure can hold several tables, with their own items. + * Map> + */ + private Map>> tablesToItems = + new HashMap>>(); + + /** + * Maps tables to key schemas + */ + private Map tablesToKeySchemas = new HashMap(); + + /** + * Maps tables to provisioned throughput + */ + private Map tablesToPrTh = new HashMap(); + + /** + * Sets table name + * @param tabName + */ + public void setTableName(String tabName){ + tableName = tabName; + } + + /** + * Gets the table name for which the table is being mapped + * @param tableName + * @return + */ + public String getTableName(String tableName){ + return tableName; + } + + /** + * Sets the provisioned throughput for the specified table + * @param tableName + * @param readCapUnits + * @param writeCapUnits + */ + public void setProvisionedThroughput(String tableName, long readCapUnits, long writeCapUnits){ + ProvisionedThroughput ptDesc = + new ProvisionedThroughput().withReadCapacityUnits(readCapUnits).withWriteCapacityUnits(writeCapUnits); + tablesToPrTh.put(tableName, ptDesc); + } + + /** + * Sets the hash range key schema for the specified table + * @param tableName + * @param rangeKeyName + * @param rangeKeyType + */ + public void setHashRangeKeySchema(String tableName, String rangeKeyName, String rangeKeyType){ + KeySchema kSchema = tablesToKeySchemas.get(tableName); + if ( kSchema == null) + kSchema = new KeySchema(); + + KeySchemaElement rangeKeyElement = + new KeySchemaElement().withAttributeName(rangeKeyName).withAttributeType(rangeKeyType); + kSchema.setRangeKeyElement(rangeKeyElement); + tablesToKeySchemas.put(tableName, kSchema); + } + + /** + * Sets the hash key schema for the specified table + * @param tableName + * @param keyName + * @param keyType + */ + public void setHashKeySchema(String tableName, String keyName, String keyType){ + KeySchema kSchema = tablesToKeySchemas.get(tableName); + if ( kSchema == null) + kSchema = new KeySchema(); + KeySchemaElement hashKey = + new KeySchemaElement().withAttributeName(keyName).withAttributeType(keyType); + kSchema.setHashKeyElement(hashKey); + tablesToKeySchemas.put(tableName, kSchema); + } + + /** + * Checks if a table exists, and if doesn't exist it creates the new table. + * @param tableName + * @return The table identified by the parameter + */ + private List> getOrCreateTable(String tableName) { + + List> items = tablesToItems.get(tableName); + if (items == null) { + items = new ArrayList>(); + tablesToItems.put(tableName, items); + } + return items; + } + + /** + * Gets the attribute for a specific item. The idea is to be able to get different items with different attributes. + * TODO This method is incomplete because the itemNumber might not be present and this would be a problem + * @param items + * @param itemNumber + * @return + */ + private HashMap getOrCreateItemAttribs(List> items, int itemNumber){ + HashMap itemAttribs; + + if (items.isEmpty()) + items.add(new HashMap()); + + itemAttribs = (HashMap) items.get(itemNumber); + if (itemAttribs == null) + items.add(new HashMap()); + return (HashMap) items.get(itemNumber); + } + + /** + * Adds an attribute to an specific item + * @param tableName + * @param attributeName + * @param attrType + * @param itemNumber + */ + public void addAttribute(String tableName, String attributeName, String attrType, int itemNumber) { + // selecting table + List> items = getOrCreateTable(tableName); + // add attribute to item + HashMap itemAttribs = getOrCreateItemAttribs(items, itemNumber); + itemAttribs.put(attributeName, attrType); + //items.add(itemAttribs); + // add item to table + //tablesToItems.put(tableName, items); + } + + /** + * Method to verify whether or not the schemas have been initialized + * @return + */ + private String verifyAllKeySchemas(){ + + String wrongTable = ""; + // if there are not tables defined + if (tablesToItems.isEmpty()) return ""; + for(String tableName : tablesToItems.keySet()){ + // if there are not schemas defined + if (tablesToKeySchemas.isEmpty()) return ""; + if (!verifyKeySchema(tableName)) return ""; + } + return wrongTable; + } + + /** + * Verifies is a table has a key schema defined + * @param tableName Table name to determine which key schema to obtain + * @return + */ + private boolean verifyKeySchema(String tableName){ + KeySchema kSchema = tablesToKeySchemas.get(tableName); + + if (kSchema == null) + return false; + + KeySchemaElement rangeKey = kSchema.getRangeKeyElement(); + KeySchemaElement hashKey = kSchema.getHashKeyElement(); + // A range key must have a hash key as well + if (rangeKey != null){ + if (hashKey != null) + return true; + else + return false; + } + // A hash key may exist by itself + if (hashKey != null) + return true; + return false; + } + + /** + * Constructs the DynamoDBMapping object + * @return A newly constructed mapping. + */ + public DynamoDBMapping build() { + + if (tableName == null) throw new IllegalStateException("tableName is not specified"); + + // verifying items for at least a table + if (tablesToItems.isEmpty()) throw new IllegalStateException("No tables"); + + // verifying if key schemas have been properly defined + String wrongTableName = verifyAllKeySchemas(); + if (!wrongTableName.equals("")) throw new IllegalStateException("no key schemas defined for table " + wrongTableName); + + // Return the tableDescription and all the attributes needed + return new DynamoDBMapping(tablesToItems,tablesToKeySchemas, tablesToPrTh); + } + } +} Added: gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBStore.java URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBStore.java?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBStore.java (added) +++ gora/trunk/gora-dynamodb/src/main/java/org/apache/gora/dynamodb/store/DynamoDBStore.java Sat Nov 3 21:22:02 2012 @@ -0,0 +1,827 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gora.dynamodb.store; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.List; +import java.util.Properties; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; + +import org.apache.gora.dynamodb.query.DynamoDBKey; +import org.apache.gora.dynamodb.query.DynamoDBQuery; +import org.apache.gora.dynamodb.query.DynamoDBResult; +import org.apache.gora.dynamodb.store.DynamoDBMapping.DynamoDBMappingBuilder; +import org.apache.gora.persistency.BeanFactory; +import org.apache.gora.persistency.Persistent; +import org.apache.gora.query.PartitionQuery; +import org.apache.gora.query.Query; +import org.apache.gora.query.Result; +import org.apache.gora.store.ws.impl.WSDataStoreBase; +import org.apache.gora.util.GoraException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.jdom.Document; +import org.jdom.Element; +import org.jdom.input.SAXBuilder; + +import com.amazonaws.AmazonServiceException; +import com.amazonaws.auth.AWSCredentials; +import com.amazonaws.auth.PropertiesCredentials; +import com.amazonaws.services.dynamodb.AmazonDynamoDB; +import com.amazonaws.services.dynamodb.AmazonDynamoDBAsyncClient; +import com.amazonaws.services.dynamodb.AmazonDynamoDBClient; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBMapper; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBQueryExpression; +import com.amazonaws.services.dynamodb.datamodeling.DynamoDBScanExpression; +import com.amazonaws.services.dynamodb.model.CreateTableRequest; +import com.amazonaws.services.dynamodb.model.DeleteTableRequest; +import com.amazonaws.services.dynamodb.model.DeleteTableResult; +import com.amazonaws.services.dynamodb.model.DescribeTableRequest; +import com.amazonaws.services.dynamodb.model.KeySchema; +import com.amazonaws.services.dynamodb.model.ProvisionedThroughput; +import com.amazonaws.services.dynamodb.model.ResourceNotFoundException; +import com.amazonaws.services.dynamodb.model.TableDescription; +import com.amazonaws.services.dynamodb.model.TableStatus; + + +public class DynamoDBStore extends WSDataStoreBase { + + /** + * Helper to write useful information into the logs + */ + public static final Logger LOG = LoggerFactory.getLogger(DynamoDBStore.class); + + /** + * Schema name which will be used from within the data store. + * If not set, all the available schemas from the mapping file will be used. + */ + private static String preferredSchema; + + /** + * The mapping file to create the tables from + */ + private static final String MAPPING_FILE = "gora-dynamodb-mapping.xml"; + + /** + * Default times to wait while requests are performed + */ + private static long waitTime = 10L * 60L * 1000L; + private static long sleepTime = 1000L * 20L; + private static long sleepDeleteTime = 1000L * 10L; + + /** + * AWS Credential file name. + */ + private static String awsCredentialsProperties = "AwsCredentials.properties"; + + /** + * Name of the cloud database provider. + */ + private static String wsProvider = "Amazon.Web.Services"; + + /** + * Parameter to decide what type of Amazon DynamoDB client to use + */ + private static String CLI_TYP_PROP = "gora.dynamodb.client"; + + /** + * Parameter to decide where the data store will make its computations + */ + private static String ENDPOINT_PROP = "gora.dynamodb.endpoint"; + + /** + * Parameter to decide which schema will be used + */ + private static String PREF_SCH_NAME = "preferred.schema.name"; + + /** + * Parameter to decide how reads will be made i.e. using strong consistency or eventual consistency. + */ + private static String CONSISTENCY_READS = "gora.dynamodb.consistent.reads"; + + /** + * The mapping object that contains the mapping file + */ + private DynamoDBMapping mapping; + + /** + * Amazon DynamoDB client which can be asynchronous or nor + */ + private AmazonDynamoDB dynamoDBClient; + + /** + * Contains the consistency level to be used + */ + private String consistency; + + /** + * TODO This would be useful for the batch read/write operations + * Contains the elements to be written or read from the data store + */ + //private Map buffer = new LinkedHashMap(); + + /** + * The class that will be persisted + */ + Class persistentClass; + + /** + * Constructor + */ + public DynamoDBStore(){ + } + + /** + * Initialize the data store by reading the credentials, setting the cloud provider, + * setting the client's properties up, setting the end point and reading the mapping file + */ + public void initialize(Class keyClass, Class pPersistentClass, + Properties properties) { + try { + LOG.debug("Initializing DynamoDB store"); + getCredentials(); + setWsProvider(wsProvider); + preferredSchema = properties.getProperty(PREF_SCH_NAME); + dynamoDBClient = getClient(properties.getProperty(CLI_TYP_PROP),(AWSCredentials)getConf()); + dynamoDBClient.setEndpoint(properties.getProperty(ENDPOINT_PROP)); + mapping = readMapping(); + consistency = properties.getProperty(CONSISTENCY_READS); + persistentClass = pPersistentClass; + } + catch (Exception e) { + LOG.error("Error while initializing DynamoDB store"); + LOG.error(e.getMessage()); + LOG.error(e.getStackTrace().toString()); + } + } + + /** + * Method to create the specific client to be used + * @param clientType + * @param credentials + * @return + */ + public AmazonDynamoDB getClient(String clientType, AWSCredentials credentials){ + if (clientType.equals("sync")) + return new AmazonDynamoDBClient(credentials); + if (clientType.equals("async")) + return new AmazonDynamoDBAsyncClient(credentials); + return null; + } + + /** + * Reads the schema file and converts it into a data structure to be used + * @param pMapFile The schema file to be mapped into a table + * @return DynamoDBMapping Object containing all necessary information to create tables + * @throws IOException + */ + @SuppressWarnings("unchecked") + private DynamoDBMapping readMapping() throws IOException { + + DynamoDBMappingBuilder mappingBuilder = new DynamoDBMappingBuilder(); + + try { + SAXBuilder builder = new SAXBuilder(); + Document doc = builder.build(getClass().getClassLoader().getResourceAsStream(MAPPING_FILE)); + + Element root = doc.getRootElement(); + + List tableElements = root.getChildren("table"); + for(Element tableElement : tableElements) { + + String tableName = tableElement.getAttributeValue("name"); + long readCapacUnits = Long.parseLong(tableElement.getAttributeValue("readcunit")); + long writeCapacUnits = Long.parseLong(tableElement.getAttributeValue("readcunit")); + + mappingBuilder.setTableName(tableName); + mappingBuilder.setProvisionedThroughput(tableName, readCapacUnits, writeCapacUnits); + LOG.debug("Basic table properties have been set: Name, and Provisioned throughput."); + + // Retrieving key's features + List fieldElements = tableElement.getChildren("key"); + for(Element fieldElement : fieldElements) { + String keyName = fieldElement.getAttributeValue("name"); + String keyType = fieldElement.getAttributeValue("type"); + String keyAttrType = fieldElement.getAttributeValue("att-type"); + if(keyType.equals("hash")) + mappingBuilder.setHashKeySchema(tableName, keyName, keyAttrType); + else if(keyType.equals("hashrange")) + mappingBuilder.setHashRangeKeySchema(tableName, keyName, keyAttrType); + } + LOG.debug("Table key schemas have been set."); + + // Retrieving attributes + fieldElements = tableElement.getChildren("attribute"); + for(Element fieldElement : fieldElements) { + String attributeName = fieldElement.getAttributeValue("name"); + String attributeType = fieldElement.getAttributeValue("type"); + mappingBuilder.addAttribute(tableName, attributeName, attributeType, 0); + } + LOG.debug("Table attributes have been read."); + } + + } catch(IOException ex) { + LOG.error("Error while performing xml mapping."); + ex.printStackTrace(); + throw ex; + + } catch(Exception ex) { + LOG.error("Error while performing xml mapping."); + ex.printStackTrace(); + throw new IOException(ex); + } + + return mappingBuilder.build(); + } + + /** + * Creates the AWSCredentials object based on the properties file. + * @return AWSCredentials object + * @throws FileNotFoundException + * @throws IllegalArgumentException + * @throws IOException + */ + private AWSCredentials getCredentials() throws FileNotFoundException, + IllegalArgumentException, IOException { + + if(authentication == null){ + InputStream awsCredInpStr = getClass().getClassLoader().getResourceAsStream(awsCredentialsProperties); + if (awsCredInpStr == null) + LOG.info("AWS Credentials File was not found on the classpath!"); + AWSCredentials credentials = new PropertiesCredentials(awsCredInpStr); + setConf(credentials); + } + return (AWSCredentials)authentication; + } + + /** + * Builds a DynamoDB query from a generic Query object + * @param query Generic query object + * @return DynamoDBQuery + */ + private DynamoDBQuery buildDynamoDBQuery(Query query){ + if(getSchemaName() == null) throw new IllegalStateException("There is not a preferred schema defined."); + + DynamoDBQuery dynamoDBQuery = new DynamoDBQuery(); + dynamoDBQuery.setKeySchema(mapping.getKeySchema(getSchemaName())); + dynamoDBQuery.setQuery(query); + dynamoDBQuery.setConsistencyReadLevel(getConsistencyReads()); + dynamoDBQuery.buildExpression(); + + return dynamoDBQuery; + } + + /** + * Gets consistency level for reads + * @return True for strong consistency or false for eventual consistent reads + */ + private boolean getConsistencyReads(){ + if(consistency != null) + if(consistency.equals("true")) + return true; + return false; + } + + /** + * Executes a query after building a DynamoDB specific query based on the received one + */ + @Override + public Result execute(Query query) { + DynamoDBQuery dynamoDBQuery = buildDynamoDBQuery(query); + DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient); + List objList = null; + if (DynamoDBQuery.getType().equals(DynamoDBQuery.RANGE_QUERY)) + objList = mapper.query(persistentClass, (DynamoDBQueryExpression)dynamoDBQuery.getQueryExpression()); + if (DynamoDBQuery.getType().equals(DynamoDBQuery.SCAN_QUERY)) + objList = mapper.scan(persistentClass, (DynamoDBScanExpression)dynamoDBQuery.getQueryExpression()); + return new DynamoDBResult(this, query, objList); + } + + @Override + public T get(K key, String[] fields) { + /* DynamoDBQuery query = new DynamoDBQuery(); + query.setDataStore(this); + //query.setKeyRange(key, key); + //query.setFields(fields); + //query.setLimit(1); + Result result = execute(query); + boolean hasResult = result.next(); + return hasResult ? result.get() : null;*/ + return null; + } + + @Override + /** + * Gets the object with the specific key + * @throws IOException + */ + public T get(K key) { + T object = null; + try { + Object rangeKey; + rangeKey = getRangeKey(key); + Object hashKey = getHashKey(key); + if (hashKey != null){ + DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient); + if (rangeKey != null) + object = mapper.load(persistentClass, hashKey, rangeKey); + else + object = mapper.load(persistentClass, hashKey); + } + else + throw new GoraException("Error while retrieving keys from object: " + key.toString()); + } catch (IllegalArgumentException e) { + e.printStackTrace(); + } catch (IllegalAccessException e) { + e.printStackTrace(); + } catch (InvocationTargetException e) { + e.printStackTrace(); + } catch (GoraException ge){ + LOG.error(ge.getMessage()); + LOG.error(ge.getStackTrace().toString()); + } + return object; + } + + /** + * Creates a new DynamoDBQuery + */ + public Query newQuery() { + Query query = new DynamoDBQuery(this); + //query.setFields(getFieldsToQuery(null)); + return query; + } + + /** + * Gets the preferred schema + */ + public String getSchemaName() { + if (preferredSchema != null) + return preferredSchema; + return null; + } + + /** + * Sets the preferred schema + * @param pSchemaName + */ + public void setSchemaName(String pSchemaName){ + preferredSchema = pSchemaName; + } + + /** + * Creates the table within the data store for a preferred schema or + * for a group of schemas defined withing the mapping file + * @throws IOException + */ + @Override + public void createSchema() { + LOG.info("Creating schema"); + if (mapping.getTables().isEmpty()) throw new IllegalStateException("There are not tables defined."); + if (preferredSchema == null){ + LOG.debug("create schemas"); + // read the mapping object + for(String tableName : mapping.getTables().keySet()) + executeCreateTableRequest(tableName); + LOG.debug("tables created successfully."); + } + else{ + LOG.debug("create schema " + preferredSchema); + executeCreateTableRequest(preferredSchema); + } + } + + /** + * Executes a create table request using the DynamoDB client and waits + * the default time until it's been created. + * @param tableName + */ + private void executeCreateTableRequest(String tableName){ + CreateTableRequest createTableRequest = getCreateTableRequest(tableName, + mapping.getKeySchema(tableName), mapping.getProvisionedThroughput(tableName)); + // use the client to perform the request + dynamoDBClient.createTable(createTableRequest).getTableDescription(); + // wait for table to become active + waitForTableToBecomeAvailable(tableName); + LOG.info(tableName + "Schema now available"); + } + + /** + * Builds the necessary requests to create tables + * @param tableName + * @param keySchema + * @param proThrou + * @return + */ + private CreateTableRequest getCreateTableRequest(String tableName, KeySchema keySchema, ProvisionedThroughput proThrou){ + CreateTableRequest createTableRequest = new CreateTableRequest(); + createTableRequest.setTableName(tableName); + createTableRequest.setKeySchema(keySchema); + createTableRequest.setProvisionedThroughput(proThrou); + return createTableRequest; + } + + /** + * Deletes all tables present in the mapping object. + * @throws IOException + */ + @Override + public void deleteSchema() { + if (mapping.getTables().isEmpty()) throw new IllegalStateException("There are not tables defined."); + if (preferredSchema == null){ + LOG.debug("Delete schemas"); + if (mapping.getTables().isEmpty()) throw new IllegalStateException("There are not tables defined."); + // read the mapping object + for(String tableName : mapping.getTables().keySet()) + executeDeleteTableRequest(tableName); + LOG.debug("All schemas deleted successfully."); + } + else{ + LOG.debug("create schema " + preferredSchema); + executeDeleteTableRequest(preferredSchema); + } + } + + /** + * Executes a delete table request using the DynamoDB client + * @param tableName + */ + public void executeDeleteTableRequest(String pTableName){ + try{ + DeleteTableRequest deleteTableRequest = new DeleteTableRequest().withTableName(pTableName); + DeleteTableResult result = dynamoDBClient.deleteTable(deleteTableRequest); + waitForTableToBeDeleted(pTableName); + LOG.debug("Schema: " + result.getTableDescription() + " deleted successfully."); + } + catch(Exception e){ + LOG.debug("Schema: " + pTableName + " deleted."); + e.printStackTrace(); + } + } + + /** + * Waits up to 6 minutes to confirm if a table has been deleted or not + * @param pTableName + */ + private void waitForTableToBeDeleted(String pTableName){ + LOG.debug("Waiting for " + pTableName + " to be deleted."); + long startTime = System.currentTimeMillis(); + long endTime = startTime + waitTime; + while (System.currentTimeMillis() < endTime) { + try {Thread.sleep(sleepDeleteTime);} catch (Exception e) {} + try { + DescribeTableRequest request = new DescribeTableRequest().withTableName(pTableName); + TableDescription tableDescription = dynamoDBClient.describeTable(request).getTable(); + String tableStatus = tableDescription.getTableStatus(); + LOG.debug(pTableName + " - current state: " + tableStatus); + } catch (AmazonServiceException ase) { + if (ase.getErrorCode().equalsIgnoreCase("ResourceNotFoundException") == true) + return; + ase.printStackTrace(); + } + } + LOG.debug(pTableName + " deleted."); + } + + /** + * Waits up to 6 minutes to confirm if a table has been created or not + * @param pTableName + */ + private void waitForTableToBecomeAvailable(String tableName) { + LOG.debug("Waiting for " + tableName + " to become available"); + long startTime = System.currentTimeMillis(); + long endTime = startTime + waitTime; + while (System.currentTimeMillis() < endTime) { + try {Thread.sleep(sleepTime);} catch (Exception e) {} + try { + DescribeTableRequest request = new DescribeTableRequest().withTableName(tableName); + TableDescription tableDescription = dynamoDBClient.describeTable(request).getTable(); + String tableStatus = tableDescription.getTableStatus(); + LOG.debug(tableName + " - current state: " + tableStatus); + if (tableStatus.equals(TableStatus.ACTIVE.toString())) return; + } catch (AmazonServiceException ase) { + if (ase.getErrorCode().equalsIgnoreCase("ResourceNotFoundException") == false) throw ase; + } + } + throw new RuntimeException("Table " + tableName + " never became active"); + } + + /** + * Verifies if the specified schemas exist + * @throws IOException + */ + @Override + public boolean schemaExists() { + LOG.info("Verifying schemas."); + TableDescription success = null; + if (mapping.getTables().isEmpty()) throw new IllegalStateException("There are not tables defined."); + if (preferredSchema == null){ + LOG.debug("Verifying schemas"); + if (mapping.getTables().isEmpty()) throw new IllegalStateException("There are not tables defined."); + // read the mapping object + for(String tableName : mapping.getTables().keySet()){ + success = getTableSchema(tableName); + if (success == null) return false; + } + } + else{ + LOG.info("Verifying schema " + preferredSchema); + success = getTableSchema(preferredSchema); + } + LOG.info("Finished verifying schemas."); + return (success != null)? true: false; + } + + /** + * Retrieves the table description for the specific resource name + * @param tableName + * @return + */ + private TableDescription getTableSchema(String tableName){ + TableDescription tableDescription = null; + try{ + DescribeTableRequest describeTableRequest = new DescribeTableRequest().withTableName(tableName); + tableDescription = dynamoDBClient.describeTable(describeTableRequest).getTable(); + } + catch(ResourceNotFoundException e){ + LOG.error("Error while getting table schema: " + tableName); + return tableDescription; + } + return tableDescription; + } + /** + * Returns a new instance of the key object. + * @throws IOException + */ + @Override + public K newKey() { + // TODO Auto-generated method stub + return null; + } + + /** + * Returns a new persistent object + * @throws IOException + */ + @Override + public T newPersistent() { + T obj = null; + try { + obj = persistentClass.newInstance(); + } catch (InstantiationException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } catch (IllegalAccessException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + return obj; + } + + /** + * Puts an object identified by a key + * @throws IOException + */ + @Override + public void put(K key, T obj) { + try{ + Object rangeKey = getRangeKey(key); + Object hashKey = getHashKey(key); + // if the key does not have these attributes then try to get them from the object + if (hashKey == null) + hashKey = getHashKey(obj); + if (rangeKey == null) + rangeKey = getRangeKey(obj); + if (hashKey != null){ + DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient); + if (rangeKey != null) + mapper.load(persistentClass, hashKey.toString(), rangeKey.toString()); + else + mapper.load(persistentClass, hashKey.toString()); + mapper.save(obj); + } + else + throw new GoraException("Error while retrieving keys from object: " + obj.toString()); + }catch(NullPointerException npe){ + LOG.error("Error while putting an item. " + npe.toString()); + npe.printStackTrace(); + }catch(Exception e){ + LOG.error("Error while putting an item. " + obj.toString()); + e.printStackTrace(); + } + } + + /** + * Deletes the object using key + * @return true for a successful process + * @throws IOException + */ + @Override + public boolean delete(K key) { + try{ + T object = null; + Object rangeKey = null, hashKey = null; + DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient); + for (Method met :key.getClass().getDeclaredMethods()){ + if(met.getName().equals("getRangeKey")){ + Object [] params = null; + rangeKey = met.invoke(key, params); + break; + } + } + for (Method met :key.getClass().getDeclaredMethods()){ + if(met.getName().equals("getHashKey")){ + Object [] params = null; + hashKey = met.invoke(key, params); + break; + } + } + if (hashKey == null) object = (T) mapper.load(persistentClass, key); + if (rangeKey == null) + object = (T) mapper.load(persistentClass, hashKey); + else + object = (T) mapper.load(persistentClass, hashKey, rangeKey); + + if (object == null) return false; + + // setting key for dynamodbMapper + mapper.delete(object); + return true; + }catch(Exception e){ + LOG.error("Error while deleting value with key " + key.toString()); + LOG.error(e.getMessage()); + return false; + } + } + + /** + * Deletes items using a specific query + * @throws IOException + */ + @Override + @SuppressWarnings("unchecked") + public long deleteByQuery(Query query) { + // TODO verify whether or not we are deleting a whole row + //String[] fields = getFieldsToQuery(query.getFields()); + //find whether all fields are queried, which means that complete + //rows will be deleted + //boolean isAllFields = Arrays.equals(fields + // , getBeanFactory().getCachedPersistent().getFields()); + Result result = execute(query); + ArrayList deletes = new ArrayList(); + try { + while(result.next()) { + T resultObj = result.get(); + deletes.add(resultObj); + + @SuppressWarnings("rawtypes") + DynamoDBKey dKey = new DynamoDBKey(); + + dKey.setHashKey(getHashKey(resultObj)); + + dKey.setRangeKey(getRangeKey(resultObj)); + delete((K)dKey); + } + } catch (IllegalArgumentException e) { + e.printStackTrace(); + } catch (IllegalAccessException e) { + e.printStackTrace(); + } catch (InvocationTargetException e) { + e.printStackTrace(); + } catch (Exception e) { + e.printStackTrace(); + } + return deletes.size(); + } + + /** + * Gets a hash key from an object of type T + * @param obj Object from which we will get a hash key + * @return + * @throws IllegalArgumentException + * @throws IllegalAccessException + * @throws InvocationTargetException + */ + private Object getHashKey(T obj) throws IllegalArgumentException, IllegalAccessException, InvocationTargetException{ + Object hashKey = null; + for (Method met : obj.getClass().getDeclaredMethods()){ + if(met.getName().equals("getHashKey")){ + Object [] params = null; + hashKey = met.invoke(obj, params); + break; + } + } + return hashKey; + } + + /** + * Gets a hash key from a key of type K + * @param obj Object from which we will get a hash key + * @return + * @throws IllegalArgumentException + * @throws IllegalAccessException + * @throws InvocationTargetException + */ + private Object getHashKey(K obj) throws IllegalArgumentException, IllegalAccessException, InvocationTargetException{ + Object hashKey = null; + for (Method met : obj.getClass().getDeclaredMethods()){ + if(met.getName().equals("getHashKey")){ + Object [] params = null; + hashKey = met.invoke(obj, params); + break; + } + } + return hashKey; + } + + /** + * Gets a range key from an object T + * @param obj Object from which a range key will be extracted + * @return + * @throws IllegalArgumentException + * @throws IllegalAccessException + * @throws InvocationTargetException + */ + private Object getRangeKey(T obj) throws IllegalArgumentException, IllegalAccessException, InvocationTargetException{ + Object rangeKey = null; + for (Method met : obj.getClass().getDeclaredMethods()){ + if(met.getName().equals("getRangeKey")){ + Object [] params = null; + rangeKey = met.invoke(obj, params); + break; + } + } + return rangeKey; + } + + /** + * Gets a range key from a key obj + * @param obj Object from which a range key will be extracted + * @return + * @throws IllegalArgumentException + * @throws IllegalAccessException + * @throws InvocationTargetException + */ + private Object getRangeKey(K obj) throws IllegalArgumentException, IllegalAccessException, InvocationTargetException{ + Object rangeKey = null; + for (Method met : obj.getClass().getDeclaredMethods()){ + if(met.getName().equals("getRangeKey")){ + Object [] params = null; + rangeKey = met.invoke(obj, params); + break; + } + } + return rangeKey; + } + + public List> getPartitions(Query query) throws IOException { + // TODO Auto-generated method stub + return null; + } + @Override + /** + * flushes objects to DynamoDB + * @throws IOException + */ + public void flush() { + // TODO Auto-generated method stub + } + + public void setBeanFactory(BeanFactory beanFactory) { + // TODO Auto-generated method stub + } + + public BeanFactory getBeanFactory() { + // TODO Auto-generated method stub + return null; + } + + @Override + /** + * Closes the data store. + */ + public void close() { + LOG.debug("Datastore closed."); + flush(); + } +} Added: gora/trunk/gora-dynamodb/src/test/conf/AwsCredentials.properties URL: http://svn.apache.org/viewvc/gora/trunk/gora-dynamodb/src/test/conf/AwsCredentials.properties?rev=1405419&view=auto ============================================================================== --- gora/trunk/gora-dynamodb/src/test/conf/AwsCredentials.properties (added) +++ gora/trunk/gora-dynamodb/src/test/conf/AwsCredentials.properties Sat Nov 3 21:22:02 2012 @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#Insert your AWS Credentials from http://aws.amazon.com/security-credentials +#The secretKey should contain 40 characters +#The accessKey should contain 20 characters + +secretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX +accessKey=XXXXXXXXXXXXXXXXXXXX