Return-Path: X-Original-To: apmail-oodt-dev-archive@www.apache.org Delivered-To: apmail-oodt-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2E46918BDE for ; Sat, 22 Aug 2015 16:38:05 +0000 (UTC) Received: (qmail 90508 invoked by uid 500); 22 Aug 2015 16:38:05 -0000 Delivered-To: apmail-oodt-dev-archive@oodt.apache.org Received: (qmail 90471 invoked by uid 500); 22 Aug 2015 16:38:05 -0000 Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@oodt.apache.org Delivered-To: mailing list dev@oodt.apache.org Received: (qmail 90450 invoked by uid 99); 22 Aug 2015 16:38:04 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2015 16:38:04 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 776C4269F15; Sat, 22 Aug 2015 16:38:04 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============1981374805060585624==" MIME-Version: 1.0 Subject: Re: Review Request 36953: Full implementation of AvroRPC alongside XMLRPC in filemgr module From: "Chris Mattmann" To: "Lewis McGibbney" , "Chris Mattmann" Cc: "oodt" , "Radu Manole" Date: Sat, 22 Aug 2015 16:38:04 -0000 Message-ID: <20150822163804.1483.19333@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Chris Mattmann" X-ReviewGroup: oodt X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/36953/ X-Sender: "Chris Mattmann" References: <20150821195338.1354.92334@reviews.apache.org> In-Reply-To: <20150821195338.1354.92334@reviews.apache.org> Reply-To: "Chris Mattmann" X-ReviewRequest-Repository: oodt --===============1981374805060585624== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36953/#review96119 ----------------------------------------------------------- This was a LOT better Radu. You are getting close. Some big things to remember: 1. If code is commented out, you should remove it. 2. We shouldn't disable tests e.g., Tika, by commenting it out. If the test isn't passing, then something is wrong (we have a regression) 3. You need to update RADIX and make sure this is working with OODT RADIX. - Chris Mattmann On Aug. 21, 2015, 7:53 p.m., Radu Manole wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36953/ > ----------------------------------------------------------- > > (Updated Aug. 21, 2015, 7:53 p.m.) > > > Review request for oodt, Lewis McGibbney and Chris Mattmann. > > > Repository: oodt > > > Description > ------- > > Separated the FileManager logic from rpc communication createing interfaces for server/client side (FileManagerCient, FileManagerServer).Refactor existing XMLRPC server/client so it would use communication interfaces. Created Avro schemas and protocol for ipc communication, (classes that have the prefixe Avro), with a Factory class AvroTypeFactory that transforms from oodt to avro and from avro to oodt classes. Implement avro server/client. For changeability I created a Factory (RpcCommunicationFactory) that can be configurable to use either avro or xmlrpc for IPC. Because of this, anywhere in project where on instantiation of server/client rewrite to use this factory. > > > Diffs > ----- > > trunk/app/fmbrowser/src/main/java/org/apache/oodt/cas/filemgr/browser/model/CasDB.java 1693501 > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/action/FilemgrUniquenessChecker.java 1693501 > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/comparator/FilemgrUniquenessCheckComparator.java 1693501 > trunk/crawler/src/site/xdoc/user/index.xml 1693501 > trunk/filemgr/pom.xml 1693501 > trunk/filemgr/src/main/avro/types/AvroComplexQuery.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroElement.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroExtractorSpec.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroFileTransferStatus.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroFilterAlgor.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroMetadata.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroMimeType.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroProduct.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroProductPage.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroProductType.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroQuery.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroQueryCriteria.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroQueryFilter.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroQueryResult.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroReference.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/AvroTypeHandler.avsc PRE-CREATION > trunk/filemgr/src/main/avro/types/protocol.avdl PRE-CREATION > trunk/filemgr/src/main/bin/filemgr 1693501 > trunk/filemgr/src/main/bin/filemgr-client 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AbstractDeleteProductCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/DeleteProductByIdCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/FileManagerCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetFirstPageCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetLastPageCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNextPageCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNumProductsCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetPrevPageCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetProductPercentTransferredCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/RetrieveFilesCliAction.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/AvroLocalDataTransferer.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/InPlaceDataTransferer.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/RemoteDataTransferer.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/LocalCache.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/StdIngester.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/ProductMetKeys.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerClient.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerServer.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManager.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClient.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClientMain.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServer.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServerMain.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerServer.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerClientFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerServerFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerClientFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerServerFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerClientFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerServerFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/CatalogSearch.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/DeleteProduct.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ExpImpCatalog.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataBasedProductMover.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataDumper.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ProductDumper.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/QueryTool.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/SolrIndexer.java 1693501 > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/AvroTypeFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/RpcCommunicationFactory.java PRE-CREATION > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java 1693501 > trunk/filemgr/src/main/resources/filemgr.properties 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/TestFileManagerCli.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/UseMockClientCmdLineActionStore.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/DummyFileManagerClient.java PRE-CREATION > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestAddProductTypeCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByIdCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByNameCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDumpMetadataCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransferCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransfersCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFilePercentTransferredCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFirstPageCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetLastPageCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNextPageCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNumProductsCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetPrevPageCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByIdCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByNameCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductPercentTransferredCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductTypeByNameCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestHasProductCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestIngestProductCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestLuceneQueryCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestCachedIngester.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestLocalCache.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestRmiCache.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestStdIngester.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/TestTikaAutoDetectExtractor.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/structs/type/TestTypeHandler.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/MockFileManagerClient.java PRE-CREATION > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManager.java PRE-CREATION > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManagerClient.java PRE-CREATION > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/rpc/TestServerClientFactories.java PRE-CREATION > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestExpImpCatalog.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestMetadataBasedProductMover.java 1693501 > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestAvroTypesFactory.java PRE-CREATION > trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestXmlRpcStructFactory.java 1693501 > trunk/filemgr/src/test/resources/createProduct/test.txt PRE-CREATION > trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/bin/filemgr 1693501 > trunk/pcs/core/src/main/java/org/apache/oodt/pcs/util/FileManagerUtils.java 1693501 > trunk/pge/src/main/java/org/apache/oodt/cas/pge/staging/FileStager.java 1693501 > trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1693501 > trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1693501 > trunk/profile/src/main/java/org/apache/oodt/profile/handlers/cas/CASProfileHandler.java 1693501 > trunk/webapp/components/src/main/java/org/apache/oodt/cas/webcomponents/filemgr/FileManagerConn.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/CASProductHandler.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DataDeliveryServlet.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DatasetDeliveryServlet.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/CasProductJaxrsService.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/servlets/CasProductJaxrsServlet.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFDatasetServlet.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFProductServlet.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductServlet.java 1693501 > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductTransferServlet.java 1693501 > > Diff: https://reviews.apache.org/r/36953/diff/ > > > Testing > ------- > > Used existing tests, applied them for Avro RPC. > > Installed filemanager module, and started it as described in the oodt documentation. Ingested a simple Generic file. Used querry-tool. > > > Thanks, > > Radu Manole > > --===============1981374805060585624==--