Okay it looks like if I'm caching my row_i in the first loop in some
global variable to avoid GC trying to get the memory it works, but it
looks ugly:

$mem = []
 for i in (0..nrows-1)
    row_i = FFI::MemoryPointer.new :double, ncols
    $mem << row_i
    data.put_pointer i*p_size, row_i
  end

isn't there any  proper way to do it?