[clang-doc] Update serializer for improved template handling

[ilovepi]

This patch updates Serialize.cpp to serialize more data about C++
templates, which are supported by the new mustache HTML template.
Split from #133161.

Co-authored-by: Peter Chou [email protected]

[ilovepi]

• [clang-doc] Track if a type is a template or builtin #138067
• [clang-doc] Update clang-doc tool to enable mustache templates #138066
• [clang-doc] Update serializer for improved template handling #138065 👈 (View in Graphite)
• [clang-doc] Implement setupTemplateValue for HTMLMustacheGenerator #138064
• [clang-doc] Extract Info into JSON values #138063
• [clang-doc] Add helpers for Template config #138062
• [clang-doc] Add HTMLMustacheGenerator methods #138061
• [clang-doc] Add HTMLMustacheGenerator.cpp #138060
• [clang-doc] Add Mustache template assets #138059
• [clang-doc] Track Descriptions and TypeDeclaration for types #138058
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[llvmbot]

@llvm/pr-subscribers-clang-tools-extra

Author: Paul Kirth (ilovepi)

Changes

This patch updates Serialize.cpp to serialize more data about C++
templates, which are supported by the new mustache HTML template.
Split from #133161.

Co-authored-by: Peter Chou <[email protected]>

---

Full diff: https://github.com/llvm/llvm-project/pull/138065.diff

2 Files Affected:

• (modified) clang-tools-extra/clang-doc/Representation.h (+3)
• (modified) clang-tools-extra/clang-doc/Serialize.cpp (+216-9)

diff --git a/clang-tools-extra/clang-doc/Representation.h b/clang-tools-extra/clang-doc/Representation.h
index 71377d10b2f40..89e264f541a76 100644
--- a/clang-tools-extra/clang-doc/Representation.h
+++ b/clang-tools-extra/clang-doc/Representation.h
@@ -362,6 +362,9 @@ struct FunctionInfo : public SymbolInfo {
   // specializations.
   SmallString<16> FullName;
 
+  // Function Prototype
+  SmallString<256> ProtoType;
+
   // When present, this function is a template or specialization.
   std::optional<TemplateInfo> Template;
 };
diff --git a/clang-tools-extra/clang-doc/Serialize.cpp b/clang-tools-extra/clang-doc/Serialize.cpp
index 18db427b5239e..21cf44c1ccd35 100644
--- a/clang-tools-extra/clang-doc/Serialize.cpp
+++ b/clang-tools-extra/clang-doc/Serialize.cpp
@@ -8,10 +8,10 @@
 
 #include "Serialize.h"
 #include "BitcodeWriter.h"
+#include "clang/AST/Attr.h"
 #include "clang/AST/Comment.h"
 #include "clang/Index/USRGeneration.h"
 #include "clang/Lex/Lexer.h"
-#include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/SHA1.h"
 
@@ -35,6 +35,188 @@ static void populateMemberTypeInfo(RecordInfo &I, AccessSpecifier &Access,
                                    const DeclaratorDecl *D,
                                    bool IsStatic = false);
 
+void getTemplateParameters(const TemplateParameterList *TemplateParams,
+                           llvm::raw_ostream &Stream) {
+  Stream << "template <";
+
+  for (unsigned i = 0; i < TemplateParams->size(); ++i) {
+    if (i > 0) {
+      Stream << ", ";
+    }
+
+    const NamedDecl *Param = TemplateParams->getParam(i);
+    if (const auto *TTP = llvm::dyn_cast<TemplateTypeParmDecl>(Param)) {
+      if (TTP->wasDeclaredWithTypename()) {
+        Stream << "typename";
+      } else {
+        Stream << "class";
+      }
+      if (TTP->isParameterPack()) {
+        Stream << "...";
+      }
+      Stream << " " << TTP->getNameAsString();
+    } else if (const auto *NTTP =
+                   llvm::dyn_cast<NonTypeTemplateParmDecl>(Param)) {
+      NTTP->getType().print(Stream, NTTP->getASTContext().getPrintingPolicy());
+      if (NTTP->isParameterPack()) {
+        Stream << "...";
+      }
+      Stream << " " << NTTP->getNameAsString();
+    } else if (const auto *TTPD =
+                   llvm::dyn_cast<TemplateTemplateParmDecl>(Param)) {
+      Stream << "template <";
+      getTemplateParameters(TTPD->getTemplateParameters(), Stream);
+      Stream << "> class " << TTPD->getNameAsString();
+    }
+  }
+
+  Stream << "> ";
+}
+
+// Extract the full function prototype from a FunctionDecl including
+// Full Decl
+llvm::SmallString<256> getFunctionPrototype(const FunctionDecl *FuncDecl) {
+  llvm::SmallString<256> Result;
+  llvm::raw_svector_ostream Stream(Result);
+  const ASTContext &Ctx = FuncDecl->getASTContext();
+  const auto *Method = llvm::dyn_cast<CXXMethodDecl>(FuncDecl);
+  // If it's a templated function, handle the template parameters
+  if (const auto *TmplDecl = FuncDecl->getDescribedTemplate()) {
+    getTemplateParameters(TmplDecl->getTemplateParameters(), Stream);
+  }
+  // If it's a virtual method
+  if (Method) {
+    if (Method->isVirtual()) {
+      Stream << "virtual ";
+    }
+  }
+  // Print return type
+  FuncDecl->getReturnType().print(Stream, Ctx.getPrintingPolicy());
+
+  // Print function name
+  Stream << " " << FuncDecl->getNameAsString() << "(";
+
+  // Print parameter list with types, names, and default values
+  for (unsigned I = 0; I < FuncDecl->getNumParams(); ++I) {
+    if (I > 0) {
+      Stream << ", ";
+    }
+    const ParmVarDecl *ParamDecl = FuncDecl->getParamDecl(I);
+    QualType ParamType = ParamDecl->getType();
+    ParamType.print(Stream, Ctx.getPrintingPolicy());
+
+    // Print parameter name if it has one
+    if (!ParamDecl->getName().empty()) {
+      Stream << " " << ParamDecl->getNameAsString();
+    }
+
+    // Print default argument if it exists
+    if (ParamDecl->hasDefaultArg()) {
+      const Expr *DefaultArg = ParamDecl->getDefaultArg();
+      if (DefaultArg) {
+        Stream << " = ";
+        DefaultArg->printPretty(Stream, nullptr, Ctx.getPrintingPolicy());
+      }
+    }
+  }
+
+  // If it is a variadic function, add '...'
+  if (FuncDecl->isVariadic()) {
+    if (FuncDecl->getNumParams() > 0) {
+      Stream << ", ";
+    }
+    Stream << "...";
+  }
+
+  Stream << ")";
+
+  // If it's a const method, add 'const' qualifier
+  if (Method) {
+    if (Method->size_overridden_methods())
+      Stream << " override";
+    if (Method->hasAttr<clang::FinalAttr>())
+      Stream << " final";
+    if (Method->isConst())
+      Stream << " const";
+    if (Method->isPureVirtual())
+      Stream << " = 0";
+  }
+  return Result; // Convert SmallString to std::string for return
+}
+
+llvm::SmallString<16> getTypeDefDecl(const TypedefDecl *TypeDef) {
+  llvm::SmallString<16> Result;
+  llvm::raw_svector_ostream Stream(Result);
+  const ASTContext &Ctx = TypeDef->getASTContext();
+  Stream << "typedef ";
+  QualType Q = TypeDef->getUnderlyingType();
+  Q.print(Stream, Ctx.getPrintingPolicy());
+  Stream << " " << TypeDef->getNameAsString();
+  return Result;
+}
+
+llvm::SmallString<16> getTypeAlias(const TypeAliasDecl *Alias) {
+  llvm::SmallString<16> Result;
+  llvm::raw_svector_ostream Stream(Result);
+  const ASTContext &Ctx = Alias->getASTContext();
+  if (const auto *TmplDecl = Alias->getDescribedTemplate()) {
+    getTemplateParameters(TmplDecl->getTemplateParameters(), Stream);
+  }
+  Stream << "using " << Alias->getNameAsString() << " = ";
+  QualType Q = Alias->getUnderlyingType();
+  Q.print(Stream, Ctx.getPrintingPolicy());
+
+  return Result;
+}
+
+// extract full syntax for record declaration
+llvm::SmallString<16> getRecordPrototype(const CXXRecordDecl *CXXRD) {
+  llvm::SmallString<16> Result;
+  LangOptions LangOpts;
+  PrintingPolicy Policy(LangOpts);
+  Policy.SuppressTagKeyword = false;
+  Policy.FullyQualifiedName = true;
+  Policy.IncludeNewlines = false;
+  llvm::raw_svector_ostream OS(Result);
+  if (const auto *TD = CXXRD->getDescribedClassTemplate()) {
+    OS << "template <";
+    bool FirstParam = true;
+    for (const auto *Param : *TD->getTemplateParameters()) {
+      if (!FirstParam)
+        OS << ", ";
+      Param->print(OS, Policy);
+      FirstParam = false;
+    }
+    OS << ">\n";
+  }
+  if (CXXRD->isStruct()) {
+    OS << "struct ";
+  } else if (CXXRD->isClass()) {
+    OS << "class ";
+  } else if (CXXRD->isUnion()) {
+    OS << "union ";
+  }
+  OS << CXXRD->getNameAsString();
+
+  // We need to make sure we have a good enough declaration to check. In the
+  // case where the class is a forward declaration, we'll fail assertions  in
+  // DeclCXX.
+  if (CXXRD->isCompleteDefinition() && CXXRD->getNumBases() > 0) {
+    OS << " : ";
+    bool FirstBase = true;
+    for (const auto &Base : CXXRD->bases()) {
+      if (!FirstBase)
+        OS << ", ";
+      if (Base.isVirtual())
+        OS << "virtual ";
+      OS << getAccessSpelling(Base.getAccessSpecifier()) << " ";
+      OS << Base.getType().getAsString(Policy);
+      FirstBase = false;
+    }
+  }
+  return Result;
+}
+
 // A function to extract the appropriate relative path for a given info's
 // documentation. The path returned is a composite of the parent namespaces.
 //
@@ -408,7 +590,6 @@ static void parseEnumerators(EnumInfo &I, const EnumDecl *D) {
     ASTContext &Context = E->getASTContext();
     if (RawComment *Comment =
             E->getASTContext().getRawCommentForDeclNoCache(E)) {
-      CommentInfo CInfo;
       Comment->setAttached();
       if (comments::FullComment *Fc = Comment->parse(Context, nullptr, E)) {
         EnumValueInfo &Member = I.Members.back();
@@ -434,6 +615,7 @@ static void parseBases(RecordInfo &I, const CXXRecordDecl *D) {
   // Don't parse bases if this isn't a definition.
   if (!D->isThisDeclarationADefinition())
     return;
+
   for (const CXXBaseSpecifier &B : D->bases()) {
     if (B.isVirtual())
       continue;
@@ -549,6 +731,7 @@ static void populateFunctionInfo(FunctionInfo &I, const FunctionDecl *D,
   populateSymbolInfo(I, D, FC, Loc, IsInAnonymousNamespace);
   auto &LO = D->getLangOpts();
   I.ReturnType = getTypeInfoForType(D->getReturnType(), LO);
+  I.ProtoType = getFunctionPrototype(D);
   parseParameters(I, D);
 
   populateTemplateParameters(I.Template, D);
@@ -680,15 +863,19 @@ emitInfo(const NamespaceDecl *D, const FullComment *FC, Location Loc,
 std::pair<std::unique_ptr<Info>, std::unique_ptr<Info>>
 emitInfo(const RecordDecl *D, const FullComment *FC, Location Loc,
          bool PublicOnly) {
+
   auto RI = std::make_unique<RecordInfo>();
   bool IsInAnonymousNamespace = false;
+
   populateSymbolInfo(*RI, D, FC, Loc, IsInAnonymousNamespace);
   if (!shouldSerializeInfo(PublicOnly, IsInAnonymousNamespace, D))
     return {};
 
   RI->TagType = D->getTagKind();
   parseFields(*RI, D, PublicOnly);
+
   if (const auto *C = dyn_cast<CXXRecordDecl>(D)) {
+    RI->FullName = getRecordPrototype(C);
     if (const TypedefNameDecl *TD = C->getTypedefNameForAnonDecl()) {
       RI->Name = TD->getNameAsString();
       RI->IsTypeDef = true;
@@ -710,11 +897,12 @@ emitInfo(const RecordDecl *D, const FullComment *FC, Location Loc,
 
     // What this is a specialization of.
     auto SpecOf = CTSD->getSpecializedTemplateOrPartial();
-    if (auto *CTD = dyn_cast<ClassTemplateDecl *>(SpecOf))
-      Specialization.SpecializationOf = getUSRForDecl(CTD);
-    else if (auto *CTPSD =
-                 dyn_cast<ClassTemplatePartialSpecializationDecl *>(SpecOf))
-      Specialization.SpecializationOf = getUSRForDecl(CTPSD);
+    if (auto *SpecPtr = dyn_cast<ClassTemplateDecl *>(SpecOf)) {
+      Specialization.SpecializationOf = getUSRForDecl(SpecPtr);
+    } else if (auto *SpecPtr =
+                   dyn_cast<ClassTemplatePartialSpecializationDecl *>(SpecOf)) {
+      Specialization.SpecializationOf = getUSRForDecl(SpecPtr);
+    }
 
     // Parameters to the specialization. For partial specializations, get the
     // parameters "as written" from the ClassTemplatePartialSpecializationDecl
@@ -786,18 +974,34 @@ emitInfo(const CXXMethodDecl *D, const FullComment *FC, Location Loc,
   return {nullptr, makeAndInsertIntoParent<FunctionInfo &&>(std::move(Func))};
 }
 
+static void extractCommentFromDecl(const Decl *D, TypedefInfo &Info) {
+  assert(D && "Invalid Decl when extracting comment");
+  ASTContext &Context = D->getASTContext();
+  RawComment *Comment = Context.getRawCommentForDeclNoCache(D);
+  if (!Comment)
+    return;
+
+  Comment->setAttached();
+  if (comments::FullComment *Fc = Comment->parse(Context, nullptr, D)) {
+    Info.Description.emplace_back();
+    parseFullComment(Fc, Info.Description.back());
+  }
+}
+
 std::pair<std::unique_ptr<Info>, std::unique_ptr<Info>>
 emitInfo(const TypedefDecl *D, const FullComment *FC, Location Loc,
          bool PublicOnly) {
   TypedefInfo Info;
   bool IsInAnonymousNamespace = false;
   populateInfo(Info, D, FC, IsInAnonymousNamespace);
+
   if (!shouldSerializeInfo(PublicOnly, IsInAnonymousNamespace, D))
     return {};
 
   Info.DefLoc = Loc;
   auto &LO = D->getLangOpts();
   Info.Underlying = getTypeInfoForType(D->getUnderlyingType(), LO);
+
   if (Info.Underlying.Type.Name.empty()) {
     // Typedef for an unnamed type. This is like "typedef struct { } Foo;"
     // The record serializer explicitly checks for this syntax and constructs
@@ -805,6 +1009,7 @@ emitInfo(const TypedefDecl *D, const FullComment *FC, Location Loc,
     return {};
   }
   Info.IsUsing = false;
+  extractCommentFromDecl(D, Info);
 
   // Info is wrapped in its parent scope so is returned in the second position.
   return {nullptr, makeAndInsertIntoParent<TypedefInfo &&>(std::move(Info))};
@@ -816,17 +1021,19 @@ std::pair<std::unique_ptr<Info>, std::unique_ptr<Info>>
 emitInfo(const TypeAliasDecl *D, const FullComment *FC, Location Loc,
          bool PublicOnly) {
   TypedefInfo Info;
-
   bool IsInAnonymousNamespace = false;
   populateInfo(Info, D, FC, IsInAnonymousNamespace);
   if (!shouldSerializeInfo(PublicOnly, IsInAnonymousNamespace, D))
     return {};
 
   Info.DefLoc = Loc;
-  auto &LO = D->getLangOpts();
+  const LangOptions &LO = D->getLangOpts();
   Info.Underlying = getTypeInfoForType(D->getUnderlyingType(), LO);
+  Info.TypeDeclaration = getTypeAlias(D);
   Info.IsUsing = true;
 
+  extractCommentFromDecl(D, Info);
+
   // Info is wrapped in its parent scope so is returned in the second position.
   return {nullptr, makeAndInsertIntoParent<TypedefInfo &&>(std::move(Info))};
 }

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[ilovepi]

@boomanaiden154 any idea why the buildkite job for linux is queued for about 3.5 hours? Also is there a way to see why the github action linux job failed? I don't see anything in the logs, just an exit 1.

[boomanaiden154]

any idea why the buildkite job for linux is queued for about 3.5 hours? Also is there a way to see why the github action linux job failed? I don't see anything in the logs, just an exit 1.

Buildkite is behind currently, by about 5 hours (https://llvm.grafana.net/public-dashboards/21c6e0a7cdd14651a90e118df46be4cc).

The failures look libc related:

2025-05-27T23:11:36.7520300Z FAILED: projects/libc/test/src/math/smoke/CMakeFiles/libc.test.src.math.smoke.asin_test.__unit__.__NO_FMA_OPT /home/gha/actions-runner/_work/llvm-project/llvm-project/build/projects/libc/test/src/math/smoke/CMakeFiles/libc.test.src.math.smoke.asin_test.__unit__.__NO_FMA_OPT 
2025-05-27T23:11:36.7521967Z cd /home/gha/actions-runner/_work/llvm-project/llvm-project/build/projects/libc/test/src/math/smoke && /home/gha/actions-runner/_work/llvm-project/llvm-project/build/projects/libc/test/src/math/smoke/libc.test.src.math.smoke.asin_test.__unit__.__NO_FMA_OPT.__build__
2025-05-27T23:11:36.7522125Z [==========] Running 4 tests from 1 test suite.
2025-05-27T23:11:36.7522337Z [ RUN      ] LlvmLibcAsinTest.SpecialNumbers
2025-05-27T23:11:36.7522523Z [       OK ] LlvmLibcAsinTest.SpecialNumbers (9 us)
2025-05-27T23:11:36.7522672Z [ RUN      ] LlvmLibcAsinTest.FTZMode
2025-05-27T23:11:36.7522837Z [       OK ] LlvmLibcAsinTest.FTZMode (1 us)
2025-05-27T23:11:36.7522984Z [ RUN      ] LlvmLibcAsinTest.DAZMode
2025-05-27T23:11:36.7523527Z /home/gha/actions-runner/_work/llvm-project/llvm-project/libc/test/src/math/smoke/asin_test.cpp:43: FAILURE
2025-05-27T23:11:36.7524650Z Failed to match __llvm_libc_21_0_0_git::asin(min_denormal) against LIBC_NAMESPACE::testing::getMatcher< LIBC_NAMESPACE::testing::TestCond::EQ>(zero).
2025-05-27T23:11:36.7525196Z Expected floating point value: 0x0000000000000000 = (S: 0, E: 0x0000, M: 0x0000000000000000)
2025-05-27T23:11:36.7525539Z Actual floating point value: 0x0000000000000001 = (S: 0, E: 0x0000, M: 0x0000000000000001)
2025-05-27T23:11:36.7525682Z [  FAILED  ] LlvmLibcAsinTest.DAZMode
2025-05-27T23:11:36.7525824Z [ RUN      ] LlvmLibcAsinTest.FTZDAZMode
2025-05-27T23:11:36.7526328Z /home/gha/actions-runner/_work/llvm-project/llvm-project/libc/test/src/math/smoke/asin_test.cpp:49: FAILURE
2025-05-27T23:11:36.7526969Z Failed to match __llvm_libc_21_0_0_git::asin(min_denormal) against LIBC_NAMESPACE::testing::getMatcher< LIBC_NAMESPACE::testing::TestCond::EQ>(zero).
2025-05-27T23:11:36.7527322Z Expected floating point value: 0x0000000000000000 = (S: 0, E: 0x0000, M: 0x0000000000000000)
2025-05-27T23:11:36.7527661Z Actual floating point value: 0x0000000000000001 = (S: 0, E: 0x0000, M: 0x0000000000000001)
2025-05-27T23:11:36.7527804Z [  FAILED  ] LlvmLibcAsinTest.FTZDAZMode
2025-05-27T23:11:36.7527932Z Ran 4 tests.  PASS: 2  FAIL: 2

Probably unrelated to the current PR. We need to work on better logging what ninja steps failed/the error log when the build fails rather than leaving someone to have to dig through the entire log. We only have nice reporting for lit failures and libc doesn't use lit.

[ilovepi]

Ah, thanks for letting me know. And I can't believe I missed the libc bits in the log. Thanks so much for looking! :)

Maybe we should surface that dashboard somewhere? seems useful for contributors... maybe some docs on llvm-zorg? Not sure if there is a better place to document that... could be a good topic for the next infra area team sync.

[ilovepi]

Merge activity

• May 28, 5:18 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• May 28, 5:20 AM UTC: Graphite rebased this pull request as part of a merge.
• May 28, 5:22 AM UTC: @ilovepi merged this pull request with Graphite.

[boomanaiden154]

Maybe we should surface that dashboard somewhere? seems useful for contributors... maybe some docs on llvm-zorg? Not sure if there is a better place to document that... could be a good topic for the next infra area team sync.

It's already linked to from somewhat extensive documentation in llvm-zorg under premerge/. None of it is super visible though (at least to those who don't know about it) and is mostly aimed at people hacking on the premerge system. Writing up a user facing doc on the LLVM side might be a good idea though, on top of just actually surfacing build errors to somewhere more user visible.