Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E636185A7 for ; Mon, 13 Jul 2015 18:59:32 +0000 (UTC) Received: (qmail 31424 invoked by uid 500); 13 Jul 2015 18:59:15 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 30856 invoked by uid 500); 13 Jul 2015 18:59:15 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 29784 invoked by uid 99); 13 Jul 2015 18:59:14 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2015 18:59:14 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2330CE07F6; Mon, 13 Jul 2015 18:59:14 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: zjshen@apache.org To: common-commits@hadoop.apache.org Date: Mon, 13 Jul 2015 18:59:36 -0000 Message-Id: <76dc4a995f574bb3bcf96740ab09fa7e@git.apache.org> In-Reply-To: <33d989a3ccfe41de887186eacebf5660@git.apache.org> References: <33d989a3ccfe41de887186eacebf5660@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [24/48] hadoop git commit: HDFS-8726. Move protobuf files that define the client-sever protocols to hdfs-client. Contributed by Haohui Mai. HDFS-8726. Move protobuf files that define the client-sever protocols to hdfs-client. Contributed by Haohui Mai. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7a9b6ad0 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7a9b6ad0 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7a9b6ad0 Branch: refs/heads/YARN-2928 Commit: 7a9b6ad04a646658022c5f045dfce194f4ffe4b9 Parents: 5099a94 Author: Haohui Mai Authored: Wed Jul 8 10:37:10 2015 -0700 Committer: Zhijie Shen Committed: Mon Jul 13 11:43:29 2015 -0700 ---------------------------------------------------------------------- .../dev-support/findbugsExcludeFile.xml | 4 + hadoop-hdfs-project/hadoop-hdfs-client/pom.xml | 42 + .../src/main/proto/ClientDatanodeProtocol.proto | 247 ++++++ .../src/main/proto/ClientNamenodeProtocol.proto | 863 +++++++++++++++++++ .../hadoop-hdfs-client/src/main/proto/acl.proto | 108 +++ .../src/main/proto/datatransfer.proto | 304 +++++++ .../src/main/proto/encryption.proto | 67 ++ .../src/main/proto/hdfs.proto | 611 +++++++++++++ .../src/main/proto/inotify.proto | 126 +++ .../src/main/proto/xattr.proto | 75 ++ hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 + hadoop-hdfs-project/hadoop-hdfs/pom.xml | 10 +- .../hadoop-hdfs/src/contrib/bkjournal/pom.xml | 2 +- .../hdfs/server/namenode/FSEditLogOp.java | 4 +- .../src/main/proto/ClientDatanodeProtocol.proto | 247 ------ .../src/main/proto/ClientNamenodeProtocol.proto | 863 ------------------- .../hadoop-hdfs/src/main/proto/acl.proto | 113 --- .../src/main/proto/datatransfer.proto | 304 ------- .../hadoop-hdfs/src/main/proto/editlog.proto | 35 + .../hadoop-hdfs/src/main/proto/encryption.proto | 67 -- .../hadoop-hdfs/src/main/proto/hdfs.proto | 611 ------------- .../hadoop-hdfs/src/main/proto/inotify.proto | 126 --- .../hadoop-hdfs/src/main/proto/xattr.proto | 80 -- 23 files changed, 2490 insertions(+), 2422 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml b/hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml index be2911f..ba6453d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml @@ -17,4 +17,8 @@ + + + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml index 1b45095..aeaa980 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml @@ -63,6 +63,48 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> + + org.apache.hadoop + hadoop-maven-plugins + + + compile-protoc + generate-sources + + protoc + + + ${protobuf.version} + ${protoc.path} + + ${basedir}/../../hadoop-common-project/hadoop-common/src/main/proto + ${basedir}/src/main/proto + + + ${basedir}/src/main/proto + + ClientDatanodeProtocol.proto + ClientNamenodeProtocol.proto + acl.proto + xattr.proto + datatransfer.proto + hdfs.proto + encryption.proto + inotify.proto + + + ${project.build.directory}/generated-sources/java + + + + + + org.apache.maven.plugins + maven-javadoc-plugin + + org.apache.hadoop.hdfs.protocol.proto + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto new file mode 100644 index 0000000..e0d1f5f --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto @@ -0,0 +1,247 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +// This file contains protocol buffers that are used throughout HDFS -- i.e. +// by the client, server, and data transfer protocols. + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "ClientDatanodeProtocolProtos"; +option java_generic_services = true; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +import "Security.proto"; +import "hdfs.proto"; + +/** + * block - block for which visible length is requested + */ +message GetReplicaVisibleLengthRequestProto { + required ExtendedBlockProto block = 1; +} + +/** + * length - visible length of the block + */ +message GetReplicaVisibleLengthResponseProto { + required uint64 length = 1; +} + +/** + * void request + */ +message RefreshNamenodesRequestProto { +} + +/** + * void response + */ +message RefreshNamenodesResponseProto { +} + +/** + * blockPool - block pool to be deleted + * force - if false, delete the block pool only if it is empty. + * if true, delete the block pool even if it has blocks. + */ +message DeleteBlockPoolRequestProto { + required string blockPool = 1; + required bool force = 2; +} + +/** + * void response + */ +message DeleteBlockPoolResponseProto { +} + +/** + * Gets the file information where block and its metadata is stored + * block - block for which path information is being requested + * token - block token + * + * This message is deprecated in favor of file descriptor passing. + */ +message GetBlockLocalPathInfoRequestProto { + required ExtendedBlockProto block = 1; + required hadoop.common.TokenProto token = 2; +} + +/** + * block - block for which file path information is being returned + * localPath - file path where the block data is stored + * localMetaPath - file path where the block meta data is stored + * + * This message is deprecated in favor of file descriptor passing. + */ +message GetBlockLocalPathInfoResponseProto { + required ExtendedBlockProto block = 1; + required string localPath = 2; + required string localMetaPath = 3; +} + +/** + * Query for the disk locations of a number of blocks on this DN. + * blockPoolId - the pool to query + * blockIds - list of block IDs to query + * tokens - list of access tokens corresponding to list of block IDs + */ +message GetHdfsBlockLocationsRequestProto { + // Removed: HDFS-3969 + // repeated ExtendedBlockProto blocks = 1; + repeated hadoop.common.TokenProto tokens = 2; + + required string blockPoolId = 3; + repeated sfixed64 blockIds = 4 [ packed = true ]; +} + +/** + * volumeIds - id of each volume, potentially multiple bytes + * volumeIndexes - for each block, an index into volumeIds specifying the volume + * on which it is located. If block is not present on any volume, + * index is set to MAX_INT. + */ +message GetHdfsBlockLocationsResponseProto { + repeated bytes volumeIds = 1; + repeated uint32 volumeIndexes = 2 [ packed = true ]; +} + +/** + * forUpgrade - if true, clients are advised to wait for restart and quick + * upgrade restart is instrumented. Otherwise, datanode does + * the regular shutdown. + */ +message ShutdownDatanodeRequestProto { + required bool forUpgrade = 1; +} + +message ShutdownDatanodeResponseProto { +} + +/** + * Ping datanode for liveness and quick info + */ +message GetDatanodeInfoRequestProto { +} + +message GetDatanodeInfoResponseProto { + required DatanodeLocalInfoProto localInfo = 1; +} + +/** Asks DataNode to reload configuration file. */ +message StartReconfigurationRequestProto { +} + +message StartReconfigurationResponseProto { +} + +message TriggerBlockReportRequestProto { + required bool incremental = 1; +} + +message TriggerBlockReportResponseProto { +} + +/** Query the running status of reconfiguration process */ +message GetReconfigurationStatusRequestProto { +} + +message GetReconfigurationStatusConfigChangeProto { + required string name = 1; + required string oldValue = 2; + optional string newValue = 3; + optional string errorMessage = 4; // It is empty if success. +} + +message GetReconfigurationStatusResponseProto { + required int64 startTime = 1; + optional int64 endTime = 2; + repeated GetReconfigurationStatusConfigChangeProto changes = 3; +} + +message ListReconfigurablePropertiesRequestProto { +} + +/** Query the reconfigurable properties on DataNode. */ +message ListReconfigurablePropertiesResponseProto { + repeated string name = 1; +} + +/** + * Protocol used from client to the Datanode. + * See the request and response for details of rpc call. + */ +service ClientDatanodeProtocolService { + /** + * Returns the visible length of the replica + */ + rpc getReplicaVisibleLength(GetReplicaVisibleLengthRequestProto) + returns(GetReplicaVisibleLengthResponseProto); + + /** + * Refresh the list of federated namenodes from updated configuration. + * Adds new namenodes and stops the deleted namenodes. + */ + rpc refreshNamenodes(RefreshNamenodesRequestProto) + returns(RefreshNamenodesResponseProto); + + /** + * Delete the block pool from the datanode. + */ + rpc deleteBlockPool(DeleteBlockPoolRequestProto) + returns(DeleteBlockPoolResponseProto); + + /** + * Retrieves the path names of the block file and metadata file stored on the + * local file system. + */ + rpc getBlockLocalPathInfo(GetBlockLocalPathInfoRequestProto) + returns(GetBlockLocalPathInfoResponseProto); + + /** + * Retrieve additional HDFS-specific metadata about a set of blocks stored + * on the local file system. + */ + rpc getHdfsBlockLocations(GetHdfsBlockLocationsRequestProto) + returns(GetHdfsBlockLocationsResponseProto); + + rpc shutdownDatanode(ShutdownDatanodeRequestProto) + returns(ShutdownDatanodeResponseProto); + + rpc getDatanodeInfo(GetDatanodeInfoRequestProto) + returns(GetDatanodeInfoResponseProto); + + rpc getReconfigurationStatus(GetReconfigurationStatusRequestProto) + returns(GetReconfigurationStatusResponseProto); + + rpc startReconfiguration(StartReconfigurationRequestProto) + returns(StartReconfigurationResponseProto); + + rpc listReconfigurableProperties( + ListReconfigurablePropertiesRequestProto) + returns(ListReconfigurablePropertiesResponseProto); + + rpc triggerBlockReport(TriggerBlockReportRequestProto) + returns(TriggerBlockReportResponseProto); +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto new file mode 100644 index 0000000..b44c556 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto @@ -0,0 +1,863 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "ClientNamenodeProtocolProtos"; +option java_generic_services = true; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +import "Security.proto"; +import "hdfs.proto"; +import "acl.proto"; +import "xattr.proto"; +import "encryption.proto"; +import "inotify.proto"; + +/** + * The ClientNamenodeProtocol Service defines the interface between a client + * (as runnign inside a MR Task) and the Namenode. + * See org.apache.hadoop.hdfs.protocol.ClientProtocol for the javadoc + * for each of the methods. + * The exceptions declared in the above class also apply to this protocol. + * Exceptions are unwrapped and thrown by the PB libraries. + */ + +message GetBlockLocationsRequestProto { + required string src = 1; // file name + required uint64 offset = 2; // range start offset + required uint64 length = 3; // range length +} + +message GetBlockLocationsResponseProto { + optional LocatedBlocksProto locations = 1; +} + +message GetServerDefaultsRequestProto { // No parameters +} + +message GetServerDefaultsResponseProto { + required FsServerDefaultsProto serverDefaults = 1; +} + +enum CreateFlagProto { + CREATE = 0x01; // Create a file + OVERWRITE = 0x02; // Truncate/overwrite a file. Same as POSIX O_TRUNC + APPEND = 0x04; // Append to a file + LAZY_PERSIST = 0x10; // File with reduced durability guarantees. + NEW_BLOCK = 0x20; // Write data to a new block when appending +} + +message CreateRequestProto { + required string src = 1; + required FsPermissionProto masked = 2; + required string clientName = 3; + required uint32 createFlag = 4; // bits set using CreateFlag + required bool createParent = 5; + required uint32 replication = 6; // Short: Only 16 bits used + required uint64 blockSize = 7; + repeated CryptoProtocolVersionProto cryptoProtocolVersion = 8; +} + +message CreateResponseProto { + optional HdfsFileStatusProto fs = 1; +} + +message AppendRequestProto { + required string src = 1; + required string clientName = 2; + optional uint32 flag = 3; // bits set using CreateFlag +} + +message AppendResponseProto { + optional LocatedBlockProto block = 1; + optional HdfsFileStatusProto stat = 2; +} + +message SetReplicationRequestProto { + required string src = 1; + required uint32 replication = 2; // Short: Only 16 bits used +} + +message SetReplicationResponseProto { + required bool result = 1; +} + +message SetStoragePolicyRequestProto { + required string src = 1; + required string policyName = 2; +} + +message SetStoragePolicyResponseProto { // void response +} + +message GetStoragePoliciesRequestProto { // void request +} + +message GetStoragePoliciesResponseProto { + repeated BlockStoragePolicyProto policies = 1; +} + +message SetPermissionRequestProto { + required string src = 1; + required FsPermissionProto permission = 2; +} + +message SetPermissionResponseProto { // void response +} + +message SetOwnerRequestProto { + required string src = 1; + optional string username = 2; + optional string groupname = 3; +} + +message SetOwnerResponseProto { // void response +} + +message AbandonBlockRequestProto { + required ExtendedBlockProto b = 1; + required string src = 2; + required string holder = 3; + optional uint64 fileId = 4 [default = 0]; // default to GRANDFATHER_INODE_ID +} + +message AbandonBlockResponseProto { // void response +} + +message AddBlockRequestProto { + required string src = 1; + required string clientName = 2; + optional ExtendedBlockProto previous = 3; + repeated DatanodeInfoProto excludeNodes = 4; + optional uint64 fileId = 5 [default = 0]; // default as a bogus id + repeated string favoredNodes = 6; //the set of datanodes to use for the block +} + +message AddBlockResponseProto { + required LocatedBlockProto block = 1; +} + +message GetAdditionalDatanodeRequestProto { + required string src = 1; + required ExtendedBlockProto blk = 2; + repeated DatanodeInfoProto existings = 3; + repeated DatanodeInfoProto excludes = 4; + required uint32 numAdditionalNodes = 5; + required string clientName = 6; + repeated string existingStorageUuids = 7; + optional uint64 fileId = 8 [default = 0]; // default to GRANDFATHER_INODE_ID +} + +message GetAdditionalDatanodeResponseProto { + required LocatedBlockProto block = 1; +} + +message CompleteRequestProto { + required string src = 1; + required string clientName = 2; + optional ExtendedBlockProto last = 3; + optional uint64 fileId = 4 [default = 0]; // default to GRANDFATHER_INODE_ID +} + +message CompleteResponseProto { + required bool result = 1; +} + +message ReportBadBlocksRequestProto { + repeated LocatedBlockProto blocks = 1; +} + +message ReportBadBlocksResponseProto { // void response +} + +message ConcatRequestProto { + required string trg = 1; + repeated string srcs = 2; +} + +message ConcatResponseProto { // void response +} + +message TruncateRequestProto { + required string src = 1; + required uint64 newLength = 2; + required string clientName = 3; +} + +message TruncateResponseProto { + required bool result = 1; +} + +message RenameRequestProto { + required string src = 1; + required string dst = 2; +} + +message RenameResponseProto { + required bool result = 1; +} + + +message Rename2RequestProto { + required string src = 1; + required string dst = 2; + required bool overwriteDest = 3; +} + +message Rename2ResponseProto { // void response +} + +message DeleteRequestProto { + required string src = 1; + required bool recursive = 2; +} + +message DeleteResponseProto { + required bool result = 1; +} + +message MkdirsRequestProto { + required string src = 1; + required FsPermissionProto masked = 2; + required bool createParent = 3; +} +message MkdirsResponseProto { + required bool result = 1; +} + +message GetListingRequestProto { + required string src = 1; + required bytes startAfter = 2; + required bool needLocation = 3; +} +message GetListingResponseProto { + optional DirectoryListingProto dirList = 1; +} + +message GetSnapshottableDirListingRequestProto { // no input parameters +} +message GetSnapshottableDirListingResponseProto { + optional SnapshottableDirectoryListingProto snapshottableDirList = 1; +} + +message GetSnapshotDiffReportRequestProto { + required string snapshotRoot = 1; + required string fromSnapshot = 2; + required string toSnapshot = 3; +} +message GetSnapshotDiffReportResponseProto { + required SnapshotDiffReportProto diffReport = 1; +} + +message RenewLeaseRequestProto { + required string clientName = 1; +} + +message RenewLeaseResponseProto { //void response +} + +message RecoverLeaseRequestProto { + required string src = 1; + required string clientName = 2; +} +message RecoverLeaseResponseProto { + required bool result = 1; +} + +message GetFsStatusRequestProto { // no input paramters +} + +message GetFsStatsResponseProto { + required uint64 capacity = 1; + required uint64 used = 2; + required uint64 remaining = 3; + required uint64 under_replicated = 4; + required uint64 corrupt_blocks = 5; + required uint64 missing_blocks = 6; + optional uint64 missing_repl_one_blocks = 7; +} + +enum DatanodeReportTypeProto { // type of the datanode report + ALL = 1; + LIVE = 2; + DEAD = 3; + DECOMMISSIONING = 4; +} + +message GetDatanodeReportRequestProto { + required DatanodeReportTypeProto type = 1; +} + +message GetDatanodeReportResponseProto { + repeated DatanodeInfoProto di = 1; +} + +message GetDatanodeStorageReportRequestProto { + required DatanodeReportTypeProto type = 1; +} + +message DatanodeStorageReportProto { + required DatanodeInfoProto datanodeInfo = 1; + repeated StorageReportProto storageReports = 2; +} + +message GetDatanodeStorageReportResponseProto { + repeated DatanodeStorageReportProto datanodeStorageReports = 1; +} + +message GetPreferredBlockSizeRequestProto { + required string filename = 1; +} + +message GetPreferredBlockSizeResponseProto { + required uint64 bsize = 1; +} + +enum SafeModeActionProto { + SAFEMODE_LEAVE = 1; + SAFEMODE_ENTER = 2; + SAFEMODE_GET = 3; +} + +message SetSafeModeRequestProto { + required SafeModeActionProto action = 1; + optional bool checked = 2 [default = false]; +} + +message SetSafeModeResponseProto { + required bool result = 1; +} + +message SaveNamespaceRequestProto { + optional uint64 timeWindow = 1 [default = 0]; + optional uint64 txGap = 2 [default = 0]; +} + +message SaveNamespaceResponseProto { // void response + optional bool saved = 1 [default = true]; +} + +message RollEditsRequestProto { // no parameters +} + +message RollEditsResponseProto { // response + required uint64 newSegmentTxId = 1; +} + +message RestoreFailedStorageRequestProto { + required string arg = 1; +} + +message RestoreFailedStorageResponseProto { + required bool result = 1; +} + +message RefreshNodesRequestProto { // no parameters +} + +message RefreshNodesResponseProto { // void response +} + +message FinalizeUpgradeRequestProto { // no parameters +} + +message FinalizeUpgradeResponseProto { // void response +} + +enum RollingUpgradeActionProto { + QUERY = 1; + START = 2; + FINALIZE = 3; +} + +message RollingUpgradeRequestProto { + required RollingUpgradeActionProto action = 1; +} + +message RollingUpgradeInfoProto { + required RollingUpgradeStatusProto status = 1; + required uint64 startTime = 2; + required uint64 finalizeTime = 3; + required bool createdRollbackImages = 4; +} + +message RollingUpgradeResponseProto { + optional RollingUpgradeInfoProto rollingUpgradeInfo= 1; +} + +message ListCorruptFileBlocksRequestProto { + required string path = 1; + optional string cookie = 2; +} + +message ListCorruptFileBlocksResponseProto { + required CorruptFileBlocksProto corrupt = 1; +} + +message MetaSaveRequestProto { + required string filename = 1; +} + +message MetaSaveResponseProto { // void response +} + +message GetFileInfoRequestProto { + required string src = 1; +} + +message GetFileInfoResponseProto { + optional HdfsFileStatusProto fs = 1; +} + +message IsFileClosedRequestProto { + required string src = 1; +} + +message IsFileClosedResponseProto { + required bool result = 1; +} + +message CacheDirectiveInfoProto { + optional int64 id = 1; + optional string path = 2; + optional uint32 replication = 3; + optional string pool = 4; + optional CacheDirectiveInfoExpirationProto expiration = 5; +} + +message CacheDirectiveInfoExpirationProto { + required int64 millis = 1; + required bool isRelative = 2; +} + +message CacheDirectiveStatsProto { + required int64 bytesNeeded = 1; + required int64 bytesCached = 2; + required int64 filesNeeded = 3; + required int64 filesCached = 4; + required bool hasExpired = 5; +} + +enum CacheFlagProto { + FORCE = 0x01; // Ignore pool resource limits +} + +message AddCacheDirectiveRequestProto { + required CacheDirectiveInfoProto info = 1; + optional uint32 cacheFlags = 2; // bits set using CacheFlag +} + +message AddCacheDirectiveResponseProto { + required int64 id = 1; +} + +message ModifyCacheDirectiveRequestProto { + required CacheDirectiveInfoProto info = 1; + optional uint32 cacheFlags = 2; // bits set using CacheFlag +} + +message ModifyCacheDirectiveResponseProto { +} + +message RemoveCacheDirectiveRequestProto { + required int64 id = 1; +} + +message RemoveCacheDirectiveResponseProto { +} + +message ListCacheDirectivesRequestProto { + required int64 prevId = 1; + required CacheDirectiveInfoProto filter = 2; +} + +message CacheDirectiveEntryProto { + required CacheDirectiveInfoProto info = 1; + required CacheDirectiveStatsProto stats = 2; +} + +message ListCacheDirectivesResponseProto { + repeated CacheDirectiveEntryProto elements = 1; + required bool hasMore = 2; +} + +message CachePoolInfoProto { + optional string poolName = 1; + optional string ownerName = 2; + optional string groupName = 3; + optional int32 mode = 4; + optional int64 limit = 5; + optional int64 maxRelativeExpiry = 6; +} + +message CachePoolStatsProto { + required int64 bytesNeeded = 1; + required int64 bytesCached = 2; + required int64 bytesOverlimit = 3; + required int64 filesNeeded = 4; + required int64 filesCached = 5; +} + +message AddCachePoolRequestProto { + required CachePoolInfoProto info = 1; +} + +message AddCachePoolResponseProto { // void response +} + +message ModifyCachePoolRequestProto { + required CachePoolInfoProto info = 1; +} + +message ModifyCachePoolResponseProto { // void response +} + +message RemoveCachePoolRequestProto { + required string poolName = 1; +} + +message RemoveCachePoolResponseProto { // void response +} + +message ListCachePoolsRequestProto { + required string prevPoolName = 1; +} + +message ListCachePoolsResponseProto { + repeated CachePoolEntryProto entries = 1; + required bool hasMore = 2; +} + +message CachePoolEntryProto { + required CachePoolInfoProto info = 1; + required CachePoolStatsProto stats = 2; +} + +message GetFileLinkInfoRequestProto { + required string src = 1; +} + +message GetFileLinkInfoResponseProto { + optional HdfsFileStatusProto fs = 1; +} + +message GetContentSummaryRequestProto { + required string path = 1; +} + +message GetContentSummaryResponseProto { + required ContentSummaryProto summary = 1; +} + +message SetQuotaRequestProto { + required string path = 1; + required uint64 namespaceQuota = 2; + required uint64 storagespaceQuota = 3; + optional StorageTypeProto storageType = 4; +} + +message SetQuotaResponseProto { // void response +} + +message FsyncRequestProto { + required string src = 1; + required string client = 2; + optional sint64 lastBlockLength = 3 [default = -1]; + optional uint64 fileId = 4 [default = 0]; // default to GRANDFATHER_INODE_ID +} + +message FsyncResponseProto { // void response +} + +message SetTimesRequestProto { + required string src = 1; + required uint64 mtime = 2; + required uint64 atime = 3; +} + +message SetTimesResponseProto { // void response +} + +message CreateSymlinkRequestProto { + required string target = 1; + required string link = 2; + required FsPermissionProto dirPerm = 3; + required bool createParent = 4; +} + +message CreateSymlinkResponseProto { // void response +} + +message GetLinkTargetRequestProto { + required string path = 1; +} +message GetLinkTargetResponseProto { + optional string targetPath = 1; +} + +message UpdateBlockForPipelineRequestProto { + required ExtendedBlockProto block = 1; + required string clientName = 2; +} + +message UpdateBlockForPipelineResponseProto { + required LocatedBlockProto block = 1; +} + +message UpdatePipelineRequestProto { + required string clientName = 1; + required ExtendedBlockProto oldBlock = 2; + required ExtendedBlockProto newBlock = 3; + repeated DatanodeIDProto newNodes = 4; + repeated string storageIDs = 5; +} + +message UpdatePipelineResponseProto { // void response +} + +message SetBalancerBandwidthRequestProto { + required int64 bandwidth = 1; +} + +message SetBalancerBandwidthResponseProto { // void response +} + +message GetDataEncryptionKeyRequestProto { // no parameters +} + +message GetDataEncryptionKeyResponseProto { + optional DataEncryptionKeyProto dataEncryptionKey = 1; +} + +message CreateSnapshotRequestProto { + required string snapshotRoot = 1; + optional string snapshotName = 2; +} + +message CreateSnapshotResponseProto { + required string snapshotPath = 1; +} + +message RenameSnapshotRequestProto { + required string snapshotRoot = 1; + required string snapshotOldName = 2; + required string snapshotNewName = 3; +} + +message RenameSnapshotResponseProto { // void response +} + +message AllowSnapshotRequestProto { + required string snapshotRoot = 1; +} + +message AllowSnapshotResponseProto { +} + +message DisallowSnapshotRequestProto { + required string snapshotRoot = 1; +} + +message DisallowSnapshotResponseProto { +} + +message DeleteSnapshotRequestProto { + required string snapshotRoot = 1; + required string snapshotName = 2; +} + +message DeleteSnapshotResponseProto { // void response +} + +message CheckAccessRequestProto { + required string path = 1; + required AclEntryProto.FsActionProto mode = 2; +} + +message CheckAccessResponseProto { // void response +} + +message GetCurrentEditLogTxidRequestProto { +} + +message GetCurrentEditLogTxidResponseProto { + required int64 txid = 1; +} + +message GetEditsFromTxidRequestProto { + required int64 txid = 1; +} + +message GetEditsFromTxidResponseProto { + required EventsListProto eventsList = 1; +} + +service ClientNamenodeProtocol { + rpc getBlockLocations(GetBlockLocationsRequestProto) + returns(GetBlockLocationsResponseProto); + rpc getServerDefaults(GetServerDefaultsRequestProto) + returns(GetServerDefaultsResponseProto); + rpc create(CreateRequestProto)returns(CreateResponseProto); + rpc append(AppendRequestProto) returns(AppendResponseProto); + rpc setReplication(SetReplicationRequestProto) + returns(SetReplicationResponseProto); + rpc setStoragePolicy(SetStoragePolicyRequestProto) + returns(SetStoragePolicyResponseProto); + rpc getStoragePolicies(GetStoragePoliciesRequestProto) + returns(GetStoragePoliciesResponseProto); + rpc setPermission(SetPermissionRequestProto) + returns(SetPermissionResponseProto); + rpc setOwner(SetOwnerRequestProto) returns(SetOwnerResponseProto); + rpc abandonBlock(AbandonBlockRequestProto) returns(AbandonBlockResponseProto); + rpc addBlock(AddBlockRequestProto) returns(AddBlockResponseProto); + rpc getAdditionalDatanode(GetAdditionalDatanodeRequestProto) + returns(GetAdditionalDatanodeResponseProto); + rpc complete(CompleteRequestProto) returns(CompleteResponseProto); + rpc reportBadBlocks(ReportBadBlocksRequestProto) + returns(ReportBadBlocksResponseProto); + rpc concat(ConcatRequestProto) returns(ConcatResponseProto); + rpc truncate(TruncateRequestProto) returns(TruncateResponseProto); + rpc rename(RenameRequestProto) returns(RenameResponseProto); + rpc rename2(Rename2RequestProto) returns(Rename2ResponseProto); + rpc delete(DeleteRequestProto) returns(DeleteResponseProto); + rpc mkdirs(MkdirsRequestProto) returns(MkdirsResponseProto); + rpc getListing(GetListingRequestProto) returns(GetListingResponseProto); + rpc renewLease(RenewLeaseRequestProto) returns(RenewLeaseResponseProto); + rpc recoverLease(RecoverLeaseRequestProto) + returns(RecoverLeaseResponseProto); + rpc getFsStats(GetFsStatusRequestProto) returns(GetFsStatsResponseProto); + rpc getDatanodeReport(GetDatanodeReportRequestProto) + returns(GetDatanodeReportResponseProto); + rpc getDatanodeStorageReport(GetDatanodeStorageReportRequestProto) + returns(GetDatanodeStorageReportResponseProto); + rpc getPreferredBlockSize(GetPreferredBlockSizeRequestProto) + returns(GetPreferredBlockSizeResponseProto); + rpc setSafeMode(SetSafeModeRequestProto) + returns(SetSafeModeResponseProto); + rpc saveNamespace(SaveNamespaceRequestProto) + returns(SaveNamespaceResponseProto); + rpc rollEdits(RollEditsRequestProto) + returns(RollEditsResponseProto); + rpc restoreFailedStorage(RestoreFailedStorageRequestProto) + returns(RestoreFailedStorageResponseProto); + rpc refreshNodes(RefreshNodesRequestProto) returns(RefreshNodesResponseProto); + rpc finalizeUpgrade(FinalizeUpgradeRequestProto) + returns(FinalizeUpgradeResponseProto); + rpc rollingUpgrade(RollingUpgradeRequestProto) + returns(RollingUpgradeResponseProto); + rpc listCorruptFileBlocks(ListCorruptFileBlocksRequestProto) + returns(ListCorruptFileBlocksResponseProto); + rpc metaSave(MetaSaveRequestProto) returns(MetaSaveResponseProto); + rpc getFileInfo(GetFileInfoRequestProto) returns(GetFileInfoResponseProto); + rpc addCacheDirective(AddCacheDirectiveRequestProto) + returns (AddCacheDirectiveResponseProto); + rpc modifyCacheDirective(ModifyCacheDirectiveRequestProto) + returns (ModifyCacheDirectiveResponseProto); + rpc removeCacheDirective(RemoveCacheDirectiveRequestProto) + returns (RemoveCacheDirectiveResponseProto); + rpc listCacheDirectives(ListCacheDirectivesRequestProto) + returns (ListCacheDirectivesResponseProto); + rpc addCachePool(AddCachePoolRequestProto) + returns(AddCachePoolResponseProto); + rpc modifyCachePool(ModifyCachePoolRequestProto) + returns(ModifyCachePoolResponseProto); + rpc removeCachePool(RemoveCachePoolRequestProto) + returns(RemoveCachePoolResponseProto); + rpc listCachePools(ListCachePoolsRequestProto) + returns(ListCachePoolsResponseProto); + rpc getFileLinkInfo(GetFileLinkInfoRequestProto) + returns(GetFileLinkInfoResponseProto); + rpc getContentSummary(GetContentSummaryRequestProto) + returns(GetContentSummaryResponseProto); + rpc setQuota(SetQuotaRequestProto) returns(SetQuotaResponseProto); + rpc fsync(FsyncRequestProto) returns(FsyncResponseProto); + rpc setTimes(SetTimesRequestProto) returns(SetTimesResponseProto); + rpc createSymlink(CreateSymlinkRequestProto) + returns(CreateSymlinkResponseProto); + rpc getLinkTarget(GetLinkTargetRequestProto) + returns(GetLinkTargetResponseProto); + rpc updateBlockForPipeline(UpdateBlockForPipelineRequestProto) + returns(UpdateBlockForPipelineResponseProto); + rpc updatePipeline(UpdatePipelineRequestProto) + returns(UpdatePipelineResponseProto); + rpc getDelegationToken(hadoop.common.GetDelegationTokenRequestProto) + returns(hadoop.common.GetDelegationTokenResponseProto); + rpc renewDelegationToken(hadoop.common.RenewDelegationTokenRequestProto) + returns(hadoop.common.RenewDelegationTokenResponseProto); + rpc cancelDelegationToken(hadoop.common.CancelDelegationTokenRequestProto) + returns(hadoop.common.CancelDelegationTokenResponseProto); + rpc setBalancerBandwidth(SetBalancerBandwidthRequestProto) + returns(SetBalancerBandwidthResponseProto); + rpc getDataEncryptionKey(GetDataEncryptionKeyRequestProto) + returns(GetDataEncryptionKeyResponseProto); + rpc createSnapshot(CreateSnapshotRequestProto) + returns(CreateSnapshotResponseProto); + rpc renameSnapshot(RenameSnapshotRequestProto) + returns(RenameSnapshotResponseProto); + rpc allowSnapshot(AllowSnapshotRequestProto) + returns(AllowSnapshotResponseProto); + rpc disallowSnapshot(DisallowSnapshotRequestProto) + returns(DisallowSnapshotResponseProto); + rpc getSnapshottableDirListing(GetSnapshottableDirListingRequestProto) + returns(GetSnapshottableDirListingResponseProto); + rpc deleteSnapshot(DeleteSnapshotRequestProto) + returns(DeleteSnapshotResponseProto); + rpc getSnapshotDiffReport(GetSnapshotDiffReportRequestProto) + returns(GetSnapshotDiffReportResponseProto); + rpc isFileClosed(IsFileClosedRequestProto) + returns(IsFileClosedResponseProto); + rpc modifyAclEntries(ModifyAclEntriesRequestProto) + returns(ModifyAclEntriesResponseProto); + rpc removeAclEntries(RemoveAclEntriesRequestProto) + returns(RemoveAclEntriesResponseProto); + rpc removeDefaultAcl(RemoveDefaultAclRequestProto) + returns(RemoveDefaultAclResponseProto); + rpc removeAcl(RemoveAclRequestProto) + returns(RemoveAclResponseProto); + rpc setAcl(SetAclRequestProto) + returns(SetAclResponseProto); + rpc getAclStatus(GetAclStatusRequestProto) + returns(GetAclStatusResponseProto); + rpc setXAttr(SetXAttrRequestProto) + returns(SetXAttrResponseProto); + rpc getXAttrs(GetXAttrsRequestProto) + returns(GetXAttrsResponseProto); + rpc listXAttrs(ListXAttrsRequestProto) + returns(ListXAttrsResponseProto); + rpc removeXAttr(RemoveXAttrRequestProto) + returns(RemoveXAttrResponseProto); + rpc checkAccess(CheckAccessRequestProto) + returns(CheckAccessResponseProto); + rpc createEncryptionZone(CreateEncryptionZoneRequestProto) + returns(CreateEncryptionZoneResponseProto); + rpc listEncryptionZones(ListEncryptionZonesRequestProto) + returns(ListEncryptionZonesResponseProto); + rpc getEZForPath(GetEZForPathRequestProto) + returns(GetEZForPathResponseProto); + rpc getCurrentEditLogTxid(GetCurrentEditLogTxidRequestProto) + returns(GetCurrentEditLogTxidResponseProto); + rpc getEditsFromTxid(GetEditsFromTxidRequestProto) + returns(GetEditsFromTxidResponseProto); +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/acl.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/acl.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/acl.proto new file mode 100644 index 0000000..bb7fdb0 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/acl.proto @@ -0,0 +1,108 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "AclProtos"; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +import "hdfs.proto"; + +message AclEntryProto { + enum AclEntryScopeProto { + ACCESS = 0x0; + DEFAULT = 0x1; + } + + enum AclEntryTypeProto { + USER = 0x0; + GROUP = 0x1; + MASK = 0x2; + OTHER = 0x3; + } + + enum FsActionProto { + NONE = 0x0; + EXECUTE = 0x1; + WRITE = 0x2; + WRITE_EXECUTE = 0x3; + READ = 0x4; + READ_EXECUTE = 0x5; + READ_WRITE = 0x6; + PERM_ALL = 0x7; + } + + required AclEntryTypeProto type = 1; + required AclEntryScopeProto scope = 2; + required FsActionProto permissions = 3; + optional string name = 4; +} + +message AclStatusProto { + required string owner = 1; + required string group = 2; + required bool sticky = 3; + repeated AclEntryProto entries = 4; + optional FsPermissionProto permission = 5; +} + +message ModifyAclEntriesRequestProto { + required string src = 1; + repeated AclEntryProto aclSpec = 2; +} + +message ModifyAclEntriesResponseProto { +} + +message RemoveAclRequestProto { + required string src = 1; +} + +message RemoveAclResponseProto { +} + +message RemoveAclEntriesRequestProto { + required string src = 1; + repeated AclEntryProto aclSpec = 2; +} + +message RemoveAclEntriesResponseProto { +} + +message RemoveDefaultAclRequestProto { + required string src = 1; +} + +message RemoveDefaultAclResponseProto { +} + +message SetAclRequestProto { + required string src = 1; + repeated AclEntryProto aclSpec = 2; +} + +message SetAclResponseProto { +} + +message GetAclStatusRequestProto { + required string src = 1; +} + +message GetAclStatusResponseProto { + required AclStatusProto result = 1; +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto new file mode 100644 index 0000000..5071d15 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto @@ -0,0 +1,304 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +// This file contains protocol buffers that are used to transfer data +// to and from the datanode, as well as between datanodes. + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "DataTransferProtos"; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +import "Security.proto"; +import "hdfs.proto"; + +message DataTransferEncryptorMessageProto { + enum DataTransferEncryptorStatus { + SUCCESS = 0; + ERROR_UNKNOWN_KEY = 1; + ERROR = 2; + } + required DataTransferEncryptorStatus status = 1; + optional bytes payload = 2; + optional string message = 3; + repeated CipherOptionProto cipherOption = 4; +} + +message BaseHeaderProto { + required ExtendedBlockProto block = 1; + optional hadoop.common.TokenProto token = 2; + optional DataTransferTraceInfoProto traceInfo = 3; +} + +message DataTransferTraceInfoProto { + required uint64 traceId = 1; + required uint64 parentId = 2; +} + +message ClientOperationHeaderProto { + required BaseHeaderProto baseHeader = 1; + required string clientName = 2; +} + +message CachingStrategyProto { + optional bool dropBehind = 1; + optional int64 readahead = 2; +} + +message OpReadBlockProto { + required ClientOperationHeaderProto header = 1; + required uint64 offset = 2; + required uint64 len = 3; + optional bool sendChecksums = 4 [default = true]; + optional CachingStrategyProto cachingStrategy = 5; +} + + +message ChecksumProto { + required ChecksumTypeProto type = 1; + required uint32 bytesPerChecksum = 2; +} + +message OpWriteBlockProto { + required ClientOperationHeaderProto header = 1; + repeated DatanodeInfoProto targets = 2; + optional DatanodeInfoProto source = 3; + enum BlockConstructionStage { + PIPELINE_SETUP_APPEND = 0; + // pipeline set up for failed PIPELINE_SETUP_APPEND recovery + PIPELINE_SETUP_APPEND_RECOVERY = 1; + // data streaming + DATA_STREAMING = 2; + // pipeline setup for failed data streaming recovery + PIPELINE_SETUP_STREAMING_RECOVERY = 3; + // close the block and pipeline + PIPELINE_CLOSE = 4; + // Recover a failed PIPELINE_CLOSE + PIPELINE_CLOSE_RECOVERY = 5; + // pipeline set up for block creation + PIPELINE_SETUP_CREATE = 6; + // transfer RBW for adding datanodes + TRANSFER_RBW = 7; + // transfer Finalized for adding datanodes + TRANSFER_FINALIZED = 8; + } + required BlockConstructionStage stage = 4; + required uint32 pipelineSize = 5; + required uint64 minBytesRcvd = 6; + required uint64 maxBytesRcvd = 7; + required uint64 latestGenerationStamp = 8; + + /** + * The requested checksum mechanism for this block write. + */ + required ChecksumProto requestedChecksum = 9; + optional CachingStrategyProto cachingStrategy = 10; + optional StorageTypeProto storageType = 11 [default = DISK]; + repeated StorageTypeProto targetStorageTypes = 12; + + /** + * Hint to the DataNode that the block can be allocated on transient + * storage i.e. memory and written to disk lazily. The DataNode is free + * to ignore this hint. + */ + optional bool allowLazyPersist = 13 [default = false]; + //whether to pin the block, so Balancer won't move it. + optional bool pinning = 14 [default = false]; + repeated bool targetPinnings = 15; +} + +message OpTransferBlockProto { + required ClientOperationHeaderProto header = 1; + repeated DatanodeInfoProto targets = 2; + repeated StorageTypeProto targetStorageTypes = 3; +} + +message OpReplaceBlockProto { + required BaseHeaderProto header = 1; + required string delHint = 2; + required DatanodeInfoProto source = 3; + optional StorageTypeProto storageType = 4 [default = DISK]; +} + +message OpCopyBlockProto { + required BaseHeaderProto header = 1; +} + +message OpBlockChecksumProto { + required BaseHeaderProto header = 1; +} + +/** + * An ID uniquely identifying a shared memory segment. + */ +message ShortCircuitShmIdProto { + required int64 hi = 1; + required int64 lo = 2; +} + +/** + * An ID uniquely identifying a slot within a shared memory segment. + */ +message ShortCircuitShmSlotProto { + required ShortCircuitShmIdProto shmId = 1; + required int32 slotIdx = 2; +} + +message OpRequestShortCircuitAccessProto { + required BaseHeaderProto header = 1; + + /** In order to get short-circuit access to block data, clients must set this + * to the highest version of the block data that they can understand. + * Currently 1 is the only version, but more versions may exist in the future + * if the on-disk format changes. + */ + required uint32 maxVersion = 2; + + /** + * The shared memory slot to use, if we are using one. + */ + optional ShortCircuitShmSlotProto slotId = 3; + + /** + * True if the client supports verifying that the file descriptor has been + * sent successfully. + */ + optional bool supportsReceiptVerification = 4 [default = false]; +} + +message ReleaseShortCircuitAccessRequestProto { + required ShortCircuitShmSlotProto slotId = 1; + optional DataTransferTraceInfoProto traceInfo = 2; +} + +message ReleaseShortCircuitAccessResponseProto { + required Status status = 1; + optional string error = 2; +} + +message ShortCircuitShmRequestProto { + // The name of the client requesting the shared memory segment. This is + // purely for logging / debugging purposes. + required string clientName = 1; + optional DataTransferTraceInfoProto traceInfo = 2; +} + +message ShortCircuitShmResponseProto { + required Status status = 1; + optional string error = 2; + optional ShortCircuitShmIdProto id = 3; +} + +message PacketHeaderProto { + // All fields must be fixed-length! + required sfixed64 offsetInBlock = 1; + required sfixed64 seqno = 2; + required bool lastPacketInBlock = 3; + required sfixed32 dataLen = 4; + optional bool syncBlock = 5 [default = false]; +} + +// Status is a 4-bit enum +enum Status { + SUCCESS = 0; + ERROR = 1; + ERROR_CHECKSUM = 2; + ERROR_INVALID = 3; + ERROR_EXISTS = 4; + ERROR_ACCESS_TOKEN = 5; + CHECKSUM_OK = 6; + ERROR_UNSUPPORTED = 7; + OOB_RESTART = 8; // Quick restart + OOB_RESERVED1 = 9; // Reserved + OOB_RESERVED2 = 10; // Reserved + OOB_RESERVED3 = 11; // Reserved + IN_PROGRESS = 12; +} + +enum ShortCircuitFdResponse { + DO_NOT_USE_RECEIPT_VERIFICATION = 0; + USE_RECEIPT_VERIFICATION = 1; +} + +message PipelineAckProto { + required sint64 seqno = 1; + repeated Status reply = 2; + optional uint64 downstreamAckTimeNanos = 3 [default = 0]; + repeated uint32 flag = 4 [packed=true]; +} + +/** + * Sent as part of the BlockOpResponseProto + * for READ_BLOCK and COPY_BLOCK operations. + */ +message ReadOpChecksumInfoProto { + required ChecksumProto checksum = 1; + + /** + * The offset into the block at which the first packet + * will start. This is necessary since reads will align + * backwards to a checksum chunk boundary. + */ + required uint64 chunkOffset = 2; +} + +message BlockOpResponseProto { + required Status status = 1; + + optional string firstBadLink = 2; + optional OpBlockChecksumResponseProto checksumResponse = 3; + optional ReadOpChecksumInfoProto readOpChecksumInfo = 4; + + /** explanatory text which may be useful to log on the client side */ + optional string message = 5; + + /** If the server chooses to agree to the request of a client for + * short-circuit access, it will send a response message with the relevant + * file descriptors attached. + * + * In the body of the message, this version number will be set to the + * specific version number of the block data that the client is about to + * read. + */ + optional uint32 shortCircuitAccessVersion = 6; +} + +/** + * Message sent from the client to the DN after reading the entire + * read request. + */ +message ClientReadStatusProto { + required Status status = 1; +} + +message DNTransferAckProto { + required Status status = 1; +} + +message OpBlockChecksumResponseProto { + required uint32 bytesPerCrc = 1; + required uint64 crcPerBlock = 2; + required bytes md5 = 3; + optional ChecksumTypeProto crcType = 4; +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/encryption.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/encryption.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/encryption.proto new file mode 100644 index 0000000..68b2f3a --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/encryption.proto @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +// This file contains protocol buffers that are used throughout HDFS -- i.e. +// by the client, server, and data transfer protocols. + + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "EncryptionZonesProtos"; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +import "hdfs.proto"; + +message CreateEncryptionZoneRequestProto { + required string src = 1; + optional string keyName = 2; +} + +message CreateEncryptionZoneResponseProto { +} + +message ListEncryptionZonesRequestProto { + required int64 id = 1; +} + +message EncryptionZoneProto { + required int64 id = 1; + required string path = 2; + required CipherSuiteProto suite = 3; + required CryptoProtocolVersionProto cryptoProtocolVersion = 4; + required string keyName = 5; +} + +message ListEncryptionZonesResponseProto { + repeated EncryptionZoneProto zones = 1; + required bool hasMore = 2; +} + +message GetEZForPathRequestProto { + required string src = 1; +} + +message GetEZForPathResponseProto { + optional EncryptionZoneProto zone = 1; +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto new file mode 100644 index 0000000..86fb462 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto @@ -0,0 +1,611 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +// This file contains protocol buffers that are used throughout HDFS -- i.e. +// by the client, server, and data transfer protocols. + + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "HdfsProtos"; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +import "Security.proto"; + +/** + * Extended block idenfies a block + */ +message ExtendedBlockProto { + required string poolId = 1; // Block pool id - gloablly unique across clusters + required uint64 blockId = 2; // the local id within a pool + required uint64 generationStamp = 3; + optional uint64 numBytes = 4 [default = 0]; // len does not belong in ebid + // here for historical reasons +} + +/** + * Identifies a Datanode + */ +message DatanodeIDProto { + required string ipAddr = 1; // IP address + required string hostName = 2; // hostname + required string datanodeUuid = 3; // UUID assigned to the Datanode. For + // upgraded clusters this is the same + // as the original StorageID of the + // Datanode. + required uint32 xferPort = 4; // data streaming port + required uint32 infoPort = 5; // datanode http port + required uint32 ipcPort = 6; // ipc server port + optional uint32 infoSecurePort = 7 [default = 0]; // datanode https port +} + +/** + * Datanode local information + */ +message DatanodeLocalInfoProto { + required string softwareVersion = 1; + required string configVersion = 2; + required uint64 uptime = 3; +} + +/** + * DatanodeInfo array + */ +message DatanodeInfosProto { + repeated DatanodeInfoProto datanodes = 1; +} + +/** + * The status of a Datanode + */ +message DatanodeInfoProto { + required DatanodeIDProto id = 1; + optional uint64 capacity = 2 [default = 0]; + optional uint64 dfsUsed = 3 [default = 0]; + optional uint64 remaining = 4 [default = 0]; + optional uint64 blockPoolUsed = 5 [default = 0]; + optional uint64 lastUpdate = 6 [default = 0]; + optional uint32 xceiverCount = 7 [default = 0]; + optional string location = 8; + enum AdminState { + NORMAL = 0; + DECOMMISSION_INPROGRESS = 1; + DECOMMISSIONED = 2; + } + + optional AdminState adminState = 10 [default = NORMAL]; + optional uint64 cacheCapacity = 11 [default = 0]; + optional uint64 cacheUsed = 12 [default = 0]; + optional uint64 lastUpdateMonotonic = 13 [default = 0]; +} + +/** + * Represents a storage available on the datanode + */ +message DatanodeStorageProto { + enum StorageState { + NORMAL = 0; + READ_ONLY_SHARED = 1; + } + + required string storageUuid = 1; + optional StorageState state = 2 [default = NORMAL]; + optional StorageTypeProto storageType = 3 [default = DISK]; +} + +message StorageReportProto { + required string storageUuid = 1 [ deprecated = true ]; + optional bool failed = 2 [ default = false ]; + optional uint64 capacity = 3 [ default = 0 ]; + optional uint64 dfsUsed = 4 [ default = 0 ]; + optional uint64 remaining = 5 [ default = 0 ]; + optional uint64 blockPoolUsed = 6 [ default = 0 ]; + optional DatanodeStorageProto storage = 7; // supersedes StorageUuid +} + +/** + * Summary of a file or directory + */ +message ContentSummaryProto { + required uint64 length = 1; + required uint64 fileCount = 2; + required uint64 directoryCount = 3; + required uint64 quota = 4; + required uint64 spaceConsumed = 5; + required uint64 spaceQuota = 6; + optional StorageTypeQuotaInfosProto typeQuotaInfos = 7; +} + +/** + * Storage type quota and usage information of a file or directory + */ +message StorageTypeQuotaInfosProto { + repeated StorageTypeQuotaInfoProto typeQuotaInfo = 1; +} + +message StorageTypeQuotaInfoProto { + required StorageTypeProto type = 1; + required uint64 quota = 2; + required uint64 consumed = 3; +} + +/** + * Contains a list of paths corresponding to corrupt files and a cookie + * used for iterative calls to NameNode.listCorruptFileBlocks. + * + */ +message CorruptFileBlocksProto { + repeated string files = 1; + required string cookie = 2; +} + +/** + * File or Directory permision - same spec as posix + */ +message FsPermissionProto { + required uint32 perm = 1; // Actually a short - only 16bits used +} + +/** + * Types of recognized storage media. + */ +enum StorageTypeProto { + DISK = 1; + SSD = 2; + ARCHIVE = 3; + RAM_DISK = 4; +} + +/** + * A list of storage types. + */ +message StorageTypesProto { + repeated StorageTypeProto storageTypes = 1; +} + +/** + * Block replica storage policy. + */ +message BlockStoragePolicyProto { + required uint32 policyId = 1; + required string name = 2; + // a list of storage types for storing the block replicas when creating a + // block. + required StorageTypesProto creationPolicy = 3; + // A list of storage types for creation fallback storage. + optional StorageTypesProto creationFallbackPolicy = 4; + optional StorageTypesProto replicationFallbackPolicy = 5; +} + +/** + * A list of storage IDs. + */ +message StorageUuidsProto { + repeated string storageUuids = 1; +} + +/** + * A LocatedBlock gives information about a block and its location. + */ +message LocatedBlockProto { + required ExtendedBlockProto b = 1; + required uint64 offset = 2; // offset of first byte of block in the file + repeated DatanodeInfoProto locs = 3; // Locations ordered by proximity to client ip + required bool corrupt = 4; // true if all replicas of a block are corrupt, else false + // If block has few corrupt replicas, they are filtered and + // their locations are not part of this object + + required hadoop.common.TokenProto blockToken = 5; + repeated bool isCached = 6 [packed=true]; // if a location in locs is cached + repeated StorageTypeProto storageTypes = 7; + repeated string storageIDs = 8; +} + +message DataEncryptionKeyProto { + required uint32 keyId = 1; + required string blockPoolId = 2; + required bytes nonce = 3; + required bytes encryptionKey = 4; + required uint64 expiryDate = 5; + optional string encryptionAlgorithm = 6; +} + +/** + * Cipher suite. + */ +enum CipherSuiteProto { + UNKNOWN = 1; + AES_CTR_NOPADDING = 2; +} + +/** + * Crypto protocol version used to access encrypted files. + */ +enum CryptoProtocolVersionProto { + UNKNOWN_PROTOCOL_VERSION = 1; + ENCRYPTION_ZONES = 2; +} + +/** + * Encryption information for a file. + */ +message FileEncryptionInfoProto { + required CipherSuiteProto suite = 1; + required CryptoProtocolVersionProto cryptoProtocolVersion = 2; + required bytes key = 3; + required bytes iv = 4; + required string keyName = 5; + required string ezKeyVersionName = 6; +} + +/** + * Encryption information for an individual + * file within an encryption zone + */ +message PerFileEncryptionInfoProto { + required bytes key = 1; + required bytes iv = 2; + required string ezKeyVersionName = 3; +} + +/** + * Encryption information for an encryption + * zone + */ +message ZoneEncryptionInfoProto { + required CipherSuiteProto suite = 1; + required CryptoProtocolVersionProto cryptoProtocolVersion = 2; + required string keyName = 3; +} + +/** + * Cipher option + */ +message CipherOptionProto { + required CipherSuiteProto suite = 1; + optional bytes inKey = 2; + optional bytes inIv = 3; + optional bytes outKey = 4; + optional bytes outIv = 5; +} + +/** + * A set of file blocks and their locations. + */ +message LocatedBlocksProto { + required uint64 fileLength = 1; + repeated LocatedBlockProto blocks = 2; + required bool underConstruction = 3; + optional LocatedBlockProto lastBlock = 4; + required bool isLastBlockComplete = 5; + optional FileEncryptionInfoProto fileEncryptionInfo = 6; +} + +/** + * Status of a file, directory or symlink + * Optionally includes a file's block locations if requested by client on the rpc call. + */ +message HdfsFileStatusProto { + enum FileType { + IS_DIR = 1; + IS_FILE = 2; + IS_SYMLINK = 3; + } + required FileType fileType = 1; + required bytes path = 2; // local name of inode encoded java UTF8 + required uint64 length = 3; + required FsPermissionProto permission = 4; + required string owner = 5; + required string group = 6; + required uint64 modification_time = 7; + required uint64 access_time = 8; + + // Optional fields for symlink + optional bytes symlink = 9; // if symlink, target encoded java UTF8 + + // Optional fields for file + optional uint32 block_replication = 10 [default = 0]; // only 16bits used + optional uint64 blocksize = 11 [default = 0]; + optional LocatedBlocksProto locations = 12; // suppled only if asked by client + + // Optional field for fileId + optional uint64 fileId = 13 [default = 0]; // default as an invalid id + optional int32 childrenNum = 14 [default = -1]; + // Optional field for file encryption + optional FileEncryptionInfoProto fileEncryptionInfo = 15; + + optional uint32 storagePolicy = 16 [default = 0]; // block storage policy id +} + +/** + * Checksum algorithms/types used in HDFS + * Make sure this enum's integer values match enum values' id properties defined + * in org.apache.hadoop.util.DataChecksum.Type + */ +enum ChecksumTypeProto { + CHECKSUM_NULL = 0; + CHECKSUM_CRC32 = 1; + CHECKSUM_CRC32C = 2; +} + +/** + * HDFS Server Defaults + */ +message FsServerDefaultsProto { + required uint64 blockSize = 1; + required uint32 bytesPerChecksum = 2; + required uint32 writePacketSize = 3; + required uint32 replication = 4; // Actually a short - only 16 bits used + required uint32 fileBufferSize = 5; + optional bool encryptDataTransfer = 6 [default = false]; + optional uint64 trashInterval = 7 [default = 0]; + optional ChecksumTypeProto checksumType = 8 [default = CHECKSUM_CRC32]; +} + + +/** + * Directory listing + */ +message DirectoryListingProto { + repeated HdfsFileStatusProto partialListing = 1; + required uint32 remainingEntries = 2; +} + +/** + * Status of a snapshottable directory: besides the normal information for + * a directory status, also include snapshot quota, number of snapshots, and + * the full path of the parent directory. + */ +message SnapshottableDirectoryStatusProto { + required HdfsFileStatusProto dirStatus = 1; + + // Fields specific for snapshottable directory + required uint32 snapshot_quota = 2; + required uint32 snapshot_number = 3; + required bytes parent_fullpath = 4; +} + +/** + * Snapshottable directory listing + */ +message SnapshottableDirectoryListingProto { + repeated SnapshottableDirectoryStatusProto snapshottableDirListing = 1; +} + +/** + * Snapshot diff report entry + */ +message SnapshotDiffReportEntryProto { + required bytes fullpath = 1; + required string modificationLabel = 2; + optional bytes targetPath = 3; +} + +/** + * Snapshot diff report + */ +message SnapshotDiffReportProto { + // full path of the directory where snapshots were taken + required string snapshotRoot = 1; + required string fromSnapshot = 2; + required string toSnapshot = 3; + repeated SnapshotDiffReportEntryProto diffReportEntries = 4; +} + +/** + * Common node information shared by all the nodes in the cluster + */ +message StorageInfoProto { + required uint32 layoutVersion = 1; // Layout version of the file system + required uint32 namespceID = 2; // File system namespace ID + required string clusterID = 3; // ID of the cluster + required uint64 cTime = 4; // File system creation time +} + +/** + * Information sent by a namenode to identify itself to the primary namenode. + */ +message NamenodeRegistrationProto { + required string rpcAddress = 1; // host:port of the namenode RPC address + required string httpAddress = 2; // host:port of the namenode http server + enum NamenodeRoleProto { + NAMENODE = 1; + BACKUP = 2; + CHECKPOINT = 3; + } + required StorageInfoProto storageInfo = 3; // Node information + optional NamenodeRoleProto role = 4 [default = NAMENODE]; // Namenode role +} + +/** + * Unique signature to identify checkpoint transactions. + */ +message CheckpointSignatureProto { + required string blockPoolId = 1; + required uint64 mostRecentCheckpointTxId = 2; + required uint64 curSegmentTxId = 3; + required StorageInfoProto storageInfo = 4; +} + +/** + * Command sent from one namenode to another namenode. + */ +message NamenodeCommandProto { + enum Type { + NamenodeCommand = 0; // Base command + CheckPointCommand = 1; // Check point command + } + required uint32 action = 1; + required Type type = 2; + optional CheckpointCommandProto checkpointCmd = 3; +} + +/** + * Command returned from primary to checkpointing namenode. + * This command has checkpoint signature that identifies + * checkpoint transaction and is needed for further + * communication related to checkpointing. + */ +message CheckpointCommandProto { + // Unique signature to identify checkpoint transation + required CheckpointSignatureProto signature = 1; + + // If true, return transfer image to primary upon the completion of checkpoint + required bool needToReturnImage = 2; +} + +/** + * Block information + * + * Please be wary of adding additional fields here, since INodeFiles + * need to fit in PB's default max message size of 64MB. + * We restrict the max # of blocks per file + * (dfs.namenode.fs-limits.max-blocks-per-file), but it's better + * to avoid changing this. + */ +message BlockProto { + required uint64 blockId = 1; + required uint64 genStamp = 2; + optional uint64 numBytes = 3 [default = 0]; +} + +/** + * Block and datanodes where is it located + */ +message BlockWithLocationsProto { + required BlockProto block = 1; // Block + repeated string datanodeUuids = 2; // Datanodes with replicas of the block + repeated string storageUuids = 3; // Storages with replicas of the block + repeated StorageTypeProto storageTypes = 4; +} + +/** + * List of block with locations + */ +message BlocksWithLocationsProto { + repeated BlockWithLocationsProto blocks = 1; +} + +/** + * Editlog information with available transactions + */ +message RemoteEditLogProto { + required uint64 startTxId = 1; // Starting available edit log transaction + required uint64 endTxId = 2; // Ending available edit log transaction + optional bool isInProgress = 3 [default = false]; +} + +/** + * Enumeration of editlogs available on a remote namenode + */ +message RemoteEditLogManifestProto { + repeated RemoteEditLogProto logs = 1; +} + +/** + * Namespace information that describes namespace on a namenode + */ +message NamespaceInfoProto { + required string buildVersion = 1; // Software revision version (e.g. an svn or git revision) + required uint32 unused = 2; // Retained for backward compatibility + required string blockPoolID = 3; // block pool used by the namespace + required StorageInfoProto storageInfo = 4;// Node information + required string softwareVersion = 5; // Software version number (e.g. 2.0.0) + optional uint64 capabilities = 6 [default = 0]; // feature flags +} + +/** + * Block access token information + */ +message BlockKeyProto { + required uint32 keyId = 1; // Key identifier + required uint64 expiryDate = 2; // Expiry time in milliseconds + optional bytes keyBytes = 3; // Key secret +} + +/** + * Current key and set of block keys at the namenode. + */ +message ExportedBlockKeysProto { + required bool isBlockTokenEnabled = 1; + required uint64 keyUpdateInterval = 2; + required uint64 tokenLifeTime = 3; + required BlockKeyProto currentKey = 4; + repeated BlockKeyProto allKeys = 5; +} + +/** + * State of a block replica at a datanode + */ +enum ReplicaStateProto { + FINALIZED = 0; // State of a replica when it is not modified + RBW = 1; // State of replica that is being written to + RWR = 2; // State of replica that is waiting to be recovered + RUR = 3; // State of replica that is under recovery + TEMPORARY = 4; // State of replica that is created for replication +} + +/** + * Block that needs to be recovered with at a given location + */ +message RecoveringBlockProto { + required uint64 newGenStamp = 1; // New genstamp post recovery + required LocatedBlockProto block = 2; // Block to be recovered + optional BlockProto truncateBlock = 3; // New block for recovery (truncate) +} + +/** + * void request + */ +message VersionRequestProto { +} + +/** + * Version response from namenode. + */ +message VersionResponseProto { + required NamespaceInfoProto info = 1; +} + +/** + * Information related to a snapshot + * TODO: add more information + */ +message SnapshotInfoProto { + required string snapshotName = 1; + required string snapshotRoot = 2; + required FsPermissionProto permission = 3; + required string owner = 4; + required string group = 5; + required string createTime = 6; + // TODO: do we need access time? +} + +/** + * Rolling upgrade status + */ +message RollingUpgradeStatusProto { + required string blockPoolId = 1; + optional bool finalized = 2 [default = false]; +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/inotify.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/inotify.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/inotify.proto new file mode 100644 index 0000000..5b78fe6 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/inotify.proto @@ -0,0 +1,126 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +// This file contains protocol buffers used to communicate edits to clients +// as part of the inotify system. + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "InotifyProtos"; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +import "acl.proto"; +import "xattr.proto"; +import "hdfs.proto"; + +enum EventType { + EVENT_CREATE = 0x0; + EVENT_CLOSE = 0x1; + EVENT_APPEND = 0x2; + EVENT_RENAME = 0x3; + EVENT_METADATA = 0x4; + EVENT_UNLINK = 0x5; +} + +message EventProto { + required EventType type = 1; + required bytes contents = 2; +} + +message EventBatchProto { + required int64 txid = 1; + repeated EventProto events = 2; +} + +enum INodeType { + I_TYPE_FILE = 0x0; + I_TYPE_DIRECTORY = 0x1; + I_TYPE_SYMLINK = 0x2; +} + +enum MetadataUpdateType { + META_TYPE_TIMES = 0x0; + META_TYPE_REPLICATION = 0x1; + META_TYPE_OWNER = 0x2; + META_TYPE_PERMS = 0x3; + META_TYPE_ACLS = 0x4; + META_TYPE_XATTRS = 0x5; +} + +message CreateEventProto { + required INodeType type = 1; + required string path = 2; + required int64 ctime = 3; + required string ownerName = 4; + required string groupName = 5; + required FsPermissionProto perms = 6; + optional int32 replication = 7; + optional string symlinkTarget = 8; + optional bool overwrite = 9; + optional int64 defaultBlockSize = 10 [default=0]; +} + +message CloseEventProto { + required string path = 1; + required int64 fileSize = 2; + required int64 timestamp = 3; +} + +message AppendEventProto { + required string path = 1; + optional bool newBlock = 2 [default = false]; +} + +message RenameEventProto { + required string srcPath = 1; + required string destPath = 2; + required int64 timestamp = 3; +} + +message MetadataUpdateEventProto { + required string path = 1; + required MetadataUpdateType type = 2; + optional int64 mtime = 3; + optional int64 atime = 4; + optional int32 replication = 5; + optional string ownerName = 6; + optional string groupName = 7; + optional FsPermissionProto perms = 8; + repeated AclEntryProto acls = 9; + repeated XAttrProto xAttrs = 10; + optional bool xAttrsRemoved = 11; +} + +message UnlinkEventProto { + required string path = 1; + required int64 timestamp = 2; +} + +message EventsListProto { + repeated EventProto events = 1; // deprecated + required int64 firstTxid = 2; + required int64 lastTxid = 3; + required int64 syncTxid = 4; + repeated EventBatchProto batch = 5; +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/xattr.proto ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/xattr.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/xattr.proto new file mode 100644 index 0000000..6c8b5eb --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/xattr.proto @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +option java_package = "org.apache.hadoop.hdfs.protocol.proto"; +option java_outer_classname = "XAttrProtos"; +option java_generate_equals_and_hash = true; +package hadoop.hdfs; + +message XAttrProto { + enum XAttrNamespaceProto { + USER = 0; + TRUSTED = 1; + SECURITY = 2; + SYSTEM = 3; + RAW = 4; + } + + required XAttrNamespaceProto namespace = 1; + required string name = 2; + optional bytes value = 3; +} + +enum XAttrSetFlagProto { + XATTR_CREATE = 0x01; + XATTR_REPLACE = 0x02; +} + +message SetXAttrRequestProto { + required string src = 1; + optional XAttrProto xAttr = 2; + optional uint32 flag = 3; //bits set using XAttrSetFlagProto +} + +message SetXAttrResponseProto { +} + +message GetXAttrsRequestProto { + required string src = 1; + repeated XAttrProto xAttrs = 2; +} + +message GetXAttrsResponseProto { + repeated XAttrProto xAttrs = 1; +} + +message ListXAttrsRequestProto { + required string src = 1; +} + +message ListXAttrsResponseProto { + repeated XAttrProto xAttrs = 1; +} + +message RemoveXAttrRequestProto { + required string src = 1; + optional XAttrProto xAttr = 2; +} + +message RemoveXAttrResponseProto { +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index a7c9e7c..f595751 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -710,6 +710,9 @@ Release 2.8.0 - UNRELEASED HDFS-8712. Remove 'public' and 'abstracta modifiers in FsVolumeSpi and FsDatasetSpi (Lei (Eddy) Xu via vinayakumarb) + HDFS-8726. Move protobuf files that define the client-sever protocols to + hdfs-client. (wheat9) + OPTIMIZATIONS HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs/pom.xml ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/pom.xml b/hadoop-hdfs-project/hadoop-hdfs/pom.xml index f90644c..db38851 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/pom.xml @@ -328,26 +328,20 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> ${protoc.path} ${basedir}/../../hadoop-common-project/hadoop-common/src/main/proto + ${basedir}/../hadoop-hdfs-client/src/main/proto ${basedir}/src/main/proto ${basedir}/src/main/proto - ClientDatanodeProtocol.proto - ClientNamenodeProtocol.proto DatanodeProtocol.proto HAZKInfo.proto InterDatanodeProtocol.proto JournalProtocol.proto NamenodeProtocol.proto QJournalProtocol.proto - acl.proto - xattr.proto - datatransfer.proto + editlog.proto fsimage.proto - hdfs.proto - encryption.proto - inotify.proto ${project.build.directory}/generated-sources/java http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml index 3b205e4..7e58606 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml @@ -112,7 +112,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> ${protoc.path} ${basedir}/../../../../../hadoop-common-project/hadoop-common/src/main/proto - ${basedir}/../../../../../hadoop-hdfs-project/hadoop-hdfs/src/main/proto + ${basedir}/../../../../../hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto ${basedir}/src/main/proto http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9b6ad0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java index c8e565e..ab36f17 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java @@ -103,8 +103,8 @@ import org.apache.hadoop.hdfs.protocol.ClientProtocol; import org.apache.hadoop.hdfs.protocol.HdfsConstants; import org.apache.hadoop.hdfs.protocol.LayoutVersion; import org.apache.hadoop.hdfs.protocol.LayoutVersion.Feature; -import org.apache.hadoop.hdfs.protocol.proto.AclProtos.AclEditLogProto; -import org.apache.hadoop.hdfs.protocol.proto.XAttrProtos.XAttrEditLogProto; +import org.apache.hadoop.hdfs.protocol.proto.EditLogProtos.AclEditLogProto; +import org.apache.hadoop.hdfs.protocol.proto.EditLogProtos.XAttrEditLogProto; import org.apache.hadoop.hdfs.protocolPB.PBHelper; import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;