asterixdb-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "abdullah alamoudi (Code Review)" <do-not-re...@asterixdb.incubator.apache.org>
Subject Change in asterixdb[master]: Improve Error Handling in Local Directory Feeds
Date Sat, 26 Mar 2016 19:57:59 GMT
Hello Ian Maxon, Jenkins,

I'd like you to reexamine a change.  Please visit

    https://asterix-gerrit.ics.uci.edu/720

to look at the new patch set (#14).

Change subject: Improve Error Handling in Local Directory Feeds
......................................................................

Improve Error Handling in Local Directory Feeds

This change improves handling of two error types for filesystem
based feeds. The first one is the handling of IO Errors which
causes the input stream to be closed, and the second one is
reacting to missed filesystem events. In both cases, we scan the
directory and compare it with the history we have in order to
resume from where we last left off.

In addition, this change includes some refactoring in external
data. Particularly, we get rid of the stream provider layer and
instead, stream factories create input streams directly. This
is consistent with record reader factories which create readers
directly without reader providers.

Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5
---
M .gitattributes
M .gitignore
M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
M asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
M asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.1.ddl.aql
M asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.2.update.aql
M asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.3.query.aql
A asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.4.sleep.aql
M asterix-app/src/test/resources/runtimets/results/external-library/classad-parser/classad-parser.1.adm
M asterix-app/src/test/resources/runtimets/results/feeds/feed-with-external-parser/feed-with-external-parser.1.adm
M asterix-app/src/test/resources/runtimets/testsuite.xml
M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java
R asterix-external-data/src/main/java/org/apache/asterix/external/api/AsterixInputStream.java
M asterix-external-data/src/main/java/org/apache/asterix/external/api/IExternalIndexer.java
M asterix-external-data/src/main/java/org/apache/asterix/external/api/IIndexingDatasource.java
R asterix-external-data/src/main/java/org/apache/asterix/external/api/IInputStreamFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedRecordDataFlowController.java
M asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedStreamDataFlowController.java
M asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedTupleForwarder.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IFeedFrameHandler.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IFeedJoint.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IFeedLifecycleListener.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IMessageReceiver.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/DistributeFeedFrameWriter.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FeedCollectRuntimeInputHandler.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FeedFrameHandlers.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FeedRuntimeInputHandler.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FrameDistributor.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/management/FeedConnectionRequest.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/message/MessageReceiver.java
M asterix-external-data/src/main/java/org/apache/asterix/external/feed/runtime/IngestionRuntime.java
M asterix-external-data/src/main/java/org/apache/asterix/external/indexing/FileOffsetIndexer.java
M asterix-external-data/src/main/java/org/apache/asterix/external/indexing/RecordColumnarIndexer.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/CharArrayRecord.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/converter/DCPRequestToRecordWithMetadataAndPKConverter.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/TestAsterixMembersReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/AbstractCharRecordLookupReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/HDFSRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/kv/KVReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/kv/KVTestReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/rss/RSSRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/AbstractStreamRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/AbstractStreamRecordReaderFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReaderFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReaderFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/QuotedLineRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReaderFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/AsterixInputStreamReader.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/BasicInputStream.java
A asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/HDFSInputStream.java
R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/LocalFSInputStream.java
A asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/SocketClientInputStream.java
M asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/SocketServerInputStream.java
A asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/TwitterFirehoseInputStream.java
R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/LocalFSInputStreamFactory.java
R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/SocketClientInputStreamFactory.java
R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/SocketServerInputStreamFactory.java
R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/TwitterFirehoseStreamFactory.java
D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/HDFSInputStreamProvider.java
D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/LocalFSInputStreamProvider.java
D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/SocketClientInputStreamProvider.java
D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/SocketServerInputStreamProvider.java
D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/TwitterFirehoseInputStreamProvider.java
M asterix-external-data/src/main/java/org/apache/asterix/external/operators/FeedCollectOperatorDescriptor.java
M asterix-external-data/src/main/java/org/apache/asterix/external/operators/FeedCollectOperatorNodePushable.java
M asterix-external-data/src/main/java/org/apache/asterix/external/parser/AbstractDataParser.java
M asterix-external-data/src/main/java/org/apache/asterix/external/provider/DataflowControllerProvider.java
M asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/AttributeReference.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/BuiltinClassAdFunctions.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/CaseInsensitiveString.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAd.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAdFunc.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAdTime.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAdUnParser.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/EvalState.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ExprList.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ExprTree.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ExprTreeHolder.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/FunctionCall.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Lexer.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Literal.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Operation.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/PrettyPrint.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Value.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/AMutableCharArrayStringPool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/AttributeReferencePool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/CaseInsensitiveStringPool.java
A asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdObjectPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdParserPool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdTimePool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/DoublePool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/EvalStatePool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ExprHolderPool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ExprListPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/FunctionCallPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/HashMapPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/Int32Pool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/Int64Pool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/LiteralPool.java
R asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/MutableBooleanPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/MutableNumberFactorPool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/OperationPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/PrettyPrintPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/StringArrayListPool.java
C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/TreeSetPool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ValuePool.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdFunctionalTest.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdParserTest.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdToADMTest.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdUnitTest.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdUnitTester.java
M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/FunctionalTester.java
M asterix-external-data/src/test/java/org/apache/asterix/external/library/ClassAdParser.java
M asterix-external-data/src/test/java/org/apache/asterix/external/library/ClassAdParserFactory.java
M asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/RecordWithMetaTest.java
M asterix-external-data/src/test/resources/functional_tests.txt
M asterix-installer/src/test/resources/integrationts/library/results/library-parsers/record-parser/record-parser.1.adm
M asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java
M asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/FeedDataSource.java
M asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/AStringSerializerDeserializer.java
M asterix-om/src/main/java/org/apache/asterix/formats/nontagged/AqlSerializerDeserializerProvider.java
M asterix-om/src/test/java/org/apache/asterix/dataflow/data/nontagged/serde/SimpleSerializerDeserializerTest.java
136 files changed, 3,333 insertions(+), 2,851 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/20/720/14
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/720
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5
Gerrit-PatchSet: 14
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi <bamousaa@gmail.com>
Gerrit-Reviewer: Ian Maxon <imaxon@apache.org>
Gerrit-Reviewer: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hubailmor@gmail.com>
Gerrit-Reviewer: Till Westmann <tillw@apache.org>
Gerrit-Reviewer: abdullah alamoudi <bamousaa@gmail.com>

Mime
View raw message