On May 7, 2010, at 1:28 PM, Chuck Remes wrote:

> 
> On May 7, 2010, at 11:26 AM, Chuck Remes wrote:
> 

>> http://github.com/chuckremes/ffi-rzmq
>> 
>> I have a tremendous memory leak in my FFI code. I'm pretty sure I know exactly where it is caused but I have no idea how to solve it. This gist shows the code in question.
>> 
>> http://gist.github.com/393627
>> 
>> The first gist block is where I think the memory leak lives. The second gist block is the definition of the leaking Struct and the definitions of the library functions.
>> 
>> Here's the sequence of events for the creation of a message where message is not nil (from the constructor).
>> 
>> 1. Allocate the Msg_t Struct.
>> 
>> 2. Create a MemoryPointer from the message string to create the Msg_t payload.
>> 
>> 3. Hand off the Struct obj to the underlying library and tell the library the size needed to store the data.
>> 
>> 4. Hand off the MemoryPointer pointing to the message data to the underlying library. The library should now *own* the memory allocated in step #2 and manage it from here on.
>> 
>> 
>> After a call is made to transmit OR receive the message obj, it calls Message#close. This makes a call to the underlying library to close the message and release its memory. This should release the memory that was allocated in #2 above. 

> 
> I made a little progress though the leak remains.
> 
> I talked with the library author and he set me straight on one of the calls. In step #3 even though we hand off a buffer we allocated to contain the payload, the library does not handle its deallocation. The handoff now includes a function callback to handle the deallocation. All the callback does is "ptr.free".
> 
> I have confirmed the callback is fired by having it print a statement. It is called for every message yet the leaking memory still grows at the same rate. :(
> 
> I looked at the source code for FFI and see that a call to MemoryPointer#free calls xfree immediately. So I'm still at a loss to explain the leakage. I'll try analyzing the JRuby heap to see if there are orphaned objects and where they are coming from.

Found it. It was my misuse of the wrapped library.

    def initialize message = nil
      @struct = LibZMQ::Msg_t.new

      if message
        data = FFI::MemoryPointer.from_string message.to_s
        result_code = LibZMQ.zmq_msg_init_size @struct, message.size
        error_check ZMQ_MSG_INIT_SIZE_STR, result_code

        # assign a memory deallocation callback function
        result_code = LibZMQ.zmq_msg_init_data @struct, data, message.size, LibZMQ::MessageDeallocator, nil
        error_check ZMQ_MSG_INIT_DATA_STR, result_code
      else
        result_code = LibZMQ.zmq_msg_init @struct
        error_check ZMQ_MSG_INIT_STR, result_code
      end
    end

That first call to zmq_msg_init_size was allocating and intializing my Msg_t struct and assigning +message.size+ bytes to it. Two lines later I call zmq_msg_init_data and pass in my payload which overwrites the buffer pointer in the Msg_t struct and orphans the memory allocated from the first step.

The fix was to eliminate the call to zmq_msg_init_size; it was unnecessary.

I know this probably isn't of interest to most readers of this ML, but I thought it resulted in some useful advice. If you have a memory leak, make sure you aren't misusing the library in such a way that it orphans its own data.

cr