[clang-c CXString] What is the lifetime of a CXString?

[illusory0x0]

In the clang-c/Index.h header file, there is no clear description of the CXString's lifetime.

CXString's lifetime might be static, not relevant to CXTranslationUnit and CXIndex, or depending on both CXTranslationUnit and CXIndex.Those behaviors I've observed. But what is the actually lifetime of the CXString?

/**
 * A character string.
 *
 * The \c CXString type is used to return strings from the interface when
 * the ownership of that string might differ from one call to the next.
 * Use \c clang_getCString() to retrieve the string data and, once finished
 * with the string data, call \c clang_disposeString() to free the string.
 */
typedef struct {
  const void *data;
  unsigned private_flags;
} CXString;

Example 1 (depending on CXTranslationUnit and CXIndex)

#include <clang-c/Index.h>
#include <iostream>
#include <vector>

int main() {
  CXIndex index = clang_createIndex(0, 0); // Create index
  CXTranslationUnit unit =
      clang_parseTranslationUnit(index, "file.cpp", nullptr, 0, nullptr, 0,
                                 CXTranslationUnit_DetailedPreprocessingRecord); // Parse "file.cpp"

  if (unit == nullptr) {
    std::cerr << "Unable to parse translation unit. Quitting.\n";
    return 0;
  }
  CXCursor cursor = clang_getTranslationUnitCursor(
      unit); // Obtain a cursor at the root of the translation unit
  std::vector<CXString> vec;

  clang_visitChildren(
      cursor,
      [](CXCursor current_cursor, CXCursor parent, CXClientData client_data) {
        CXString css =
            clang_getCursorSpelling(current_cursor);
        auto vec = (std::vector<CXString> *)client_data;
        vec->push_back(css);
        return CXChildVisit_Recurse;
      },
      &vec);

  clang_disposeTranslationUnit(unit);
  clang_disposeIndex(index);
  for (auto x : vec) {
    std::cout << clang_getCString(x) << '\n';
    clang_disposeString(x);
  }
}

==29048==ERROR: AddressSanitizer: heap-use-after-free on address 0x52100006fa08 at pc 0x7b8d76a7d96f bp 0x7ffecc6e3a90 sp 0x7ffecc6e3238
READ of size 9 at 0x52100006fa08 thread T0
    #0 0x7b8d76a7d96e in strlen ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:391
    #1 0x7b8d7455712d in std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) (/lib/x86_64-linux-gnu/libstdc++.so.6+0x15712d) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
    #2 0x633abfaa6b9f in main (/home/illu/playgound/cxx/libclang-example/build/main+0x2b9f) (BuildId: d56c11e25e7e94615c48a390146a7373f424c47f)
    #3 0x7b8d7402a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #4 0x7b8d7402a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #5 0x633abfaa6484 in _start (/home/illu/playgound/cxx/libclang-example/build/main+0x2484) (BuildId: d56c11e25e7e94615c48a390146a7373f424c47f)

0x52100006fa08 is located 1288 bytes inside of 4096-byte region [0x52100006f500,0x521000070500)
freed by thread T0 here:
    #0 0x7b8d76aff888 in operator delete(void*, std::align_val_t) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:170
    #1 0x7b8d74c4f71e  (/usr/lib/llvm-18/lib/libclang-18.so.18+0x44f71e) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #2 0x7b8d754d2430  (/usr/lib/llvm-18/lib/libclang-18.so.18+0xcd2430) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #3 0x7b8d753320a1  (/usr/lib/llvm-18/lib/libclang-18.so.18+0xb320a1) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #4 0x7b8d74c27a98 in clang_disposeTranslationUnit (/usr/lib/llvm-18/lib/libclang-18.so.18+0x427a98) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #5 0x633abfaa6a36 in main (/home/illu/playgound/cxx/libclang-example/build/main+0x2a36) (BuildId: d56c11e25e7e94615c48a390146a7373f424c47f)
    #6 0x7b8d7402a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #7 0x7b8d7402a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #8 0x633abfaa6484 in _start (/home/illu/playgound/cxx/libclang-example/build/main+0x2484) (BuildId: d56c11e25e7e94615c48a390146a7373f424c47f)

previously allocated by thread T1 here:
    #0 0x7b8d76afeaf8 in operator new(unsigned long, std::align_val_t) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:107
    #1 0x7b8d74c414b3  (/usr/lib/llvm-18/lib/libclang-18.so.18+0x4414b3) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #2 0x7b8d74c43dc0  (/usr/lib/llvm-18/lib/libclang-18.so.18+0x443dc0) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #3 0x7b8d750d281d  (/usr/lib/llvm-18/lib/libclang-18.so.18+0x8d281d) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #4 0x7b8d753cfd92  (/usr/lib/llvm-18/lib/libclang-18.so.18+0xbcfd92) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #5 0x7b8d75334e23  (/usr/lib/llvm-18/lib/libclang-18.so.18+0xb34e23) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #6 0x7b8d75338596  (/usr/lib/llvm-18/lib/libclang-18.so.18+0xb38596) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #7 0x7b8d753393c2  (/usr/lib/llvm-18/lib/libclang-18.so.18+0xb393c2) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #8 0x7b8d74c48a13  (/usr/lib/llvm-18/lib/libclang-18.so.18+0x448a13) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #9 0x7b8d6d4eff76 in llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/usr/lib/llvm-18/lib/../lib/libLLVM.so.18.1+0xceff76) (BuildId: 495b188fed20e21b476ccb1eaea212f4304fad28)
    #10 0x7b8d6d4f016c  (/usr/lib/llvm-18/lib/../lib/libLLVM.so.18.1+0xcf016c) (BuildId: 495b188fed20e21b476ccb1eaea212f4304fad28)
    #11 0x7b8d6d4f02b2  (/usr/lib/llvm-18/lib/../lib/libLLVM.so.18.1+0xcf02b2) (BuildId: 495b188fed20e21b476ccb1eaea212f4304fad28)
    #12 0x7b8d76a5ea41 in asan_thread_start ../../../../src/libsanitizer/asan/asan_interceptors.cpp:234
    #13 0x7b8d7409caa3 in start_thread nptl/pthread_create.c:447

Thread T1 created by T0 here:
    #0 0x7b8d76af51f9 in pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:245
    #1 0x7b8d6d5a7981 in llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::optional<unsigned int>) (/usr/lib/llvm-18/lib/../lib/libLLVM.so.18.1+0xda7981) (BuildId: 495b188fed20e21b476ccb1eaea212f4304fad28)
    #2 0x7b8d6d4f00f4 in llvm::CrashRecoveryContext::RunSafelyOnThread(llvm::function_ref<void ()>, unsigned int) (/usr/lib/llvm-18/lib/../lib/libLLVM.so.18.1+0xcf00f4) (BuildId: 495b188fed20e21b476ccb1eaea212f4304fad28)
    #3 0x7b8d74c3156d in clang_parseTranslationUnit2FullArgv (/usr/lib/llvm-18/lib/libclang-18.so.18+0x43156d) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #4 0x7b8d74c31188 in clang_parseTranslationUnit2 (/usr/lib/llvm-18/lib/libclang-18.so.18+0x431188) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #5 0x7b8d74c31087 in clang_parseTranslationUnit (/usr/lib/llvm-18/lib/libclang-18.so.18+0x431087) (BuildId: ea056e259864b6af08c46875208cc97be270ce0e)
    #6 0x633abfaa689f in main (/home/illu/playgound/cxx/libclang-example/build/main+0x289f) (BuildId: d56c11e25e7e94615c48a390146a7373f424c47f)
    #7 0x7b8d7402a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #8 0x7b8d7402a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #9 0x633abfaa6484 in _start (/home/illu/playgound/cxx/libclang-example/build/main+0x2484) (BuildId: d56c11e25e7e94615c48a390146a7373f424c47f)

SUMMARY: AddressSanitizer: heap-use-after-free ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:391 in strlen
Shadow bytes around the buggy address:
  0x52100006f780: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006f800: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006f880: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006f900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006f980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x52100006fa00: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006fa80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006fb00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006fb80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006fc00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x52100006fc80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==29048==ABORTING
make: *** [Makefile:5: run_main] Error 1

Example 2 (Not Relevant to CXTranslationUnit and CXIndex)

#include <clang-c/Index.h>
#include <iostream>
#include <vector>

int main() {
  CXIndex index = clang_createIndex(0, 0); // Create index
  CXTranslationUnit unit =
      clang_parseTranslationUnit(index, "file.cpp", nullptr, 0, nullptr, 0,
                                 CXTranslationUnit_None); // Parse "file.cpp"

  if (unit == nullptr) {
    std::cerr << "Unable to parse translation unit. Quitting.\n";
    return 0;
  }
  CXCursor cursor = clang_getTranslationUnitCursor(
      unit); // Obtain a cursor at the root of the translation unit
  std::vector<CXString> vec;

  clang_visitChildren(
      cursor,
      [](CXCursor current_cursor, CXCursor parent, CXClientData client_data) {
        CXString css =
            clang_getCursorSpelling(current_cursor);
        auto vec = (std::vector<CXString> *)client_data;
        vec->push_back(css);
        return CXChildVisit_Recurse;
      },
      &vec);

  clang_disposeTranslationUnit(unit);
  clang_disposeIndex(index);
  for (auto x : vec) {
    std::cout << clang_getCString(x) << '\n';
    clang_disposeString(x);
  }
}

cmake -G Ninja -B build -S .; cmake --build build
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/illu/playgound/cxx/libclang-example/build
[2/2] Linking CXX executable main
./build/main
A
value
B
value
struct_value
struct A

[AaronBallman]

The CXString object itself is always passed around by value, so its lifetime is that of any other stack object. The data pointed to by the object, however, has various different lifetimes. That's why you have to call clang_disposeString() to finally destroy the string (if the contents need to be released, that's what does it).

I've not hooked your examples up to a debugger yet, so this is speculation, but my guess is that clang_getCursorSpelling() is getting a CXString object which wraps a pointer (rather than copying the string data to a buffer that needs to be freed) and that pointer is tied to the translation unit (more specifically, the ASTUnit held by the TU object).

[illusory0x0]

CXString point to contents 's lifetime is shorter than TU object,

I think it would be better to change the documentation to say that we need to call clang_disposeString before calling clang_disposeTranslationUnit, the current formulation is too vague.

CXString lifetime

void clang_disposeString(CXString string) {
  switch ((CXStringFlag) string.private_flags) {
    case CXS_Unmanaged:
      break;
    case CXS_Malloc:
      if (string.data)
        free(const_cast<void *>(string.data));
      break;
    case CXS_StringBuf:
      static_cast<cxstring::CXStringBuf *>(
          const_cast<void *>(string.data))->dispose();
      break;
  }
}

void CXStringBuf::dispose() {
  TU->StringPool->Pool.push_back(this);
}

[AaronBallman]

I think it would be better to change the documentation to say that we need to call clang_disposeString before calling clang_disposeTranslationUnit, the current formulation is too vague.

Another approach would be to figure out which strings are tied to the TU and duplicate those strings rather than simply reference their pointers. But yes, something should change here I think, either the documentation or the implementation.

[illusory0x0]

I think it would be better to change the documentation to say that we need to call clang_disposeString before calling clang_disposeTranslationUnit, the current formulation is too vague.

Another approach would be to figure out which strings are tied to the TU and duplicate those strings rather than simply reference their pointers. But yes, something should change here I think, either the documentation or the implementation.

when use clang-c to FFI in other language, just simply reference their pointers would be better, to order to keep lifetime simple will always copy the bytes use itself memory manage strategy in other language . then their can use builtin type.

e.g.

Moonbit Example

// callee owned calling convention
//
// Dont't `decref` object(contains `TranslationUnit`) before call this function
//
// https://github.com/llvm/llvm-project/issues/138881
Bytes illusory0x0_clang_CXString_to_moonbit_Bytes(CXString self) {
  const char *cs = clang_getCString(self);
  int len = strlen(cs);
  Bytes res = moonbit_make_bytes(len, 0);
  memcpy(res, cs, len);
  clang_disposeString(self);
  return res;
}

moonbit example

Rust Example

pub unsafe fn to_string(clang: CXString) -> String {
        let c = CStr::from_ptr(clang_getCString(clang));
        let rust = c.to_str().expect("invalid Rust string").into();
        clang_disposeString(clang);
        rust
}

rust example

Python example

class _CXString(Structure):
    """Helper for transforming CXString results."""

    _fields_ = [("spelling", c_char_p), ("free", c_int)]

    def __del__(self) -> None:
        conf.lib.clang_disposeString(self)

    @staticmethod
    def from_result(res: _CXString) -> str:
        assert isinstance(res, _CXString)
        pystr = c_interop_string.to_python_string(conf.lib.clang_getCString(res))
        if pystr is None:
            return ""
        return pystr

python example

Related Issues

• [clang-c] refactor CXString #139028