hawq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hongxu Ma (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (HAWQ-1627) Support setting the max protocol message size when talking with HDFS
Date Tue, 10 Jul 2018 02:10:00 GMT

     [ https://issues.apache.org/jira/browse/HAWQ-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hongxu Ma closed HAWQ-1627.
---------------------------
    Resolution: Fixed

fixed

> Support setting the max protocol message size when talking with HDFS
> --------------------------------------------------------------------
>
>                 Key: HAWQ-1627
>                 URL: https://issues.apache.org/jira/browse/HAWQ-1627
>             Project: Apache HAWQ
>          Issue Type: Improvement
>          Components: libhdfs
>            Reporter: Hongxu Ma
>            Assignee: Hongxu Ma
>            Priority: Major
>             Fix For: 2.4.0.0-incubating
>
>
> Now, the max size of protocol message in libhdfs is 64MB, and it cannot be adjusted.
> When the max size is reached (accessing a very big HDFS table/file), will see the following
lines in hawq master log:
> {code}
> 2018-06-20 11:21:56.768003 CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd
party error log:
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:208] A protocol message was rejected
because it was too big (more than 67108864 bytes).  To increase the limit (or to
> disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.",,,,,,,,"SysLoggerMain","syslogger.c",518,
> 2018-06-20 11:21:56.771657 CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd
party error log:
> 2018-06-20 11:21:56.771492, p75751, th0x7fffcd7303c0, ERROR Failed to invoke RPC call
""getFsStats"" on server ""localhost:9000"":
> RpcChannel.cpp: 783: HdfsRpcException: RPC channel to ""localhost:9000"" got protocol
mismatch: RPC channel cannot parse response header.
>     @   Hdfs::Internal::RpcChannelImpl::readOneResponse(bool)
>     @   Hdfs::Internal::RpcChannelImpl::checkOneResponse()
>     @   Hdfs::Internal::RpcChannelImpl::invokeInternal(std::__1::shared_ptr<Hdfs::Internal::RpcRemoteCall>)
>     @   Hdfs::Internal::RpcChannelImpl:",,,,,,,,"SysLoggerMain","syslogger.c",518,
> 2018-06-20 11:21:56.771711 CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd
party error log:
> :invoke(Hdfs::Internal::RpcCall const&)
>     @   Hdfs::Internal::NamenodeImpl::invoke(Hdfs::Internal::RpcCall const&)
>     @   Hdfs::Internal::NamenodeImpl::getFsStats()
>     @   Hdfs::Internal::NamenodeProxy::getFsStats()
>     @   Hdfs::Internal::FileSystemImpl::getFsStats()
>     @   Hdfs::Internal::FileSystemImpl::connect()
>     @   Hdfs::FileSystem::connect(char const*, char const*, char const*)
>     @   Hdfs::FileSystem::connect(char const*)
>     @   hdfsBuilderConnect
>     @   gpfs_hdfs_connect
>     @   HdfsConnect
>     @   HdfsGetConnection
>     @   HdfsGetFileBlockLocati",,,,,,,,"SysLoggerMain","syslogger.c",518,
> {code}
> Considering HDFS has a guc *"ipc.maximum.data.length"* to set it, HAWQ should also add
a guc for it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message