thrift-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jens Geyer (Jira)" <j...@apache.org>
Subject [jira] [Commented] (THRIFT-5322) Go compact_protocol allocating unreasonable buffer size
Date Sun, 13 Dec 2020 10:01:00 GMT

    [ https://issues.apache.org/jira/browse/THRIFT-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248546#comment-17248546
] 

Jens Geyer commented on THRIFT-5322:
------------------------------------

{quote}
But I do not think implementing TConfiguration in go is required to fix this issue
{quote}
You are absolutely right, its not needed to fix the ssue at hand. 

My primary goal was to make you aware of a) what we already have implemented to prevent people
from doing double work and b) that the problem might be slightly bigger than the actual issue
suggests. Because we also have lists/sets/maps where memory effectively is allocated based
on the number of elements which also imply a certain minimum amount of data that has to follow
- same as with binary/strings.

> Go compact_protocol allocating unreasonable buffer size
> -------------------------------------------------------
>
>                 Key: THRIFT-5322
>                 URL: https://issues.apache.org/jira/browse/THRIFT-5322
>             Project: Thrift
>          Issue Type: Bug
>          Components: Go - Library
>    Affects Versions: 0.13.0
>            Reporter: Juraci Paixão Kröhling
>            Assignee: Yuxuan Wang
>            Priority: Major
>         Attachments: main.go
>
>          Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> I don't yet know all the pieces to this puzzle, and it's quite possible that the problem
is on our side, but we use the Thrift Go library in the Jaeger Agent and we are seeing a case
where the memory consumption for a payload of 4k bytes to result in a buffer allocation in
the compact_protocol.go with unreasonable sizes. I found buffers of 1.4GiB while debugging
the issue.
>  
> This is the code that we are seeing this memory usage:
> [https://github.com/apache/thrift/blob/b75e88a33d67ae05ef9b5fa001d2a63a2effe377/lib/go/thrift/compact_protocol.go#L556-L577]
>  
> Here's more information about this, including a reproducer and initial diagnostics:
> [https://github.com/jaegertracing/jaeger/issues/2638#issuecomment-741848201]
>  
> As mentioned above, I'm still getting all the pieces together, but perhaps you've seen
this before or know what might be going on. What I know for sure at the moment is that this
happens on malformed payloads, but I would expect the library to have an upper limit on the
buffer size.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message