Podcast: simplify newsletter references

Removes the need for boilerplate logic when adding newsletter references
The new logic requires just the podcast timestamps to be added in the
related newsletter sections.
For each podcast, the `recap_references_generator` will parse the
`reference` newsletter (declared in frontmatter) looking for podcast
reference marks (timestamps) in order to create the newsletter
references for the podcast page.

Author: kouloumos

_includes/functions/podcast-callout.md

@@ -0,0 +1,32 @@
+{% assign newsletter_sections = page.references | group_by:"header" %}
+{% assign header_names = newsletter_sections | map: "name" %}
+{% unless header_names contains "News" %}
+## News
+*No significant news this week was found on the Bitcoin-Dev or Lightning-Dev mailing lists.*
+{% endunless %}
+
+{% for section in newsletter_sections %}
+## {{ section.name }}
+<ul>
+  {% for reference in section.items %}
+  {% assign podcast_slug = reference.podcast_slug | default: reference.slug %}
+  <li id="{{ podcast_slug | slice: 1, podcast_slug.size }}" class="anchor-list">
+    <p>
+      <a href="{{ podcast_slug }}" class="anchor-list-link">●</a>
+      {{ reference.title }}
+      (<a title="Play this segment of the podcast" onClick="seek('{{reference.timestamp}}')" class="seek">{{reference.timestamp}}</a><noscript>{{reference.timestamp}}</noscript>)
+      <a href="{{page.reference}}{{reference.slug}}">
+        <i class="fa fa-link" title="Link to related content"></i>
+      </a>
+      <!--<a onClick="seek('{{include.timestamp}}')" class="seek"><i class="fa
+  fa-headphones" title="Play this segment of the podcast"></i></a>-->
+      {% if reference.has_transcript_section %}
+      <a href="{{reference.slug}}-transcript">
+        <i class="fa fa-file-text-o" title="Read this segment of the transcription"></i>
+      </a>
+      {% endif %}
+    </p>
+  </li>
+  {% endfor %}
+</ul>
+{% endfor %}

_includes/functions/podcast-note.md

@@ -16,10 +16,7 @@ def auto_anchor(content)
         ## No match, pass item through unchanged
         string
       else
-        ## Remove double-quotes from titles before attempting to slugify
-        title.gsub!('"', '')
-        ## Use Liquid/Jekyll slugify filter to choose our id
-        slug = "\#{{ \"#{title}\" | slugify: 'latin' }}"
+        slug = generate_slug(title)
         id_prefix = "- {:#{slug} .anchor-list} <a href=\"#{slug}\" class=\"anchor-list-link\">●</a>"
         string.sub!(/-/, id_prefix)
       end

_includes/newsletter-references.md

@@ -0,0 +1,10 @@
+def generate_slug(title)
+  ## Remove double-quotes from titles before attempting to slugify
+  title.gsub('"', '')
+  ## Use Liquid/Jekyll slugify filter to choose our id
+  liquid_string = "\#{{ \"#{title}\" | slugify: 'latin' }}"
+  slug = Liquid::Template.parse(liquid_string)
+  # An empty context is used here because we only need to parse the liquid
+  # string and don't require any additional variables or data.
+  slug.render(Liquid::Context.new) 
+end

_plugins/auto-anchor.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+# Regex pattern to match "{% assign timestamp="xx:xx:xx" %}"
+$podcast_reference_mark = /\{%\s*assign\s+timestamp\s*=\s*"([^"]+)"\s*%\}/
+
+# Create the podcast recap references by parsing the referenced newsletter for
+# podcast reference marks (timestamps)
+class RecapReferencesGenerator < Jekyll::Generator
+  def generate(site)
+    podcast_pages = site.documents.select { |doc| doc.data["type"] == "podcast"}
+    podcast_pages.each do |podcast|
+      # podcast episodes have a "reference" field that indicates the related newsletter page 
+      unless podcast.data["reference"].nil?
+        reference_page = site.documents.detect { |page| page.url == podcast.data["reference"] }
+        
+        # override the content of the reference page (newsletter) to now include
+        # the links to the related podcast items
+        reference_page.content,
+        # keep all the references in a podcast page variable to use them later 
+        # during the podcast page creation
+        podcast.data["references"] = get_podcast_references(reference_page.content, podcast.url)
+        
+        # Each podcast transcript splits into segements using the paragraph title
+        # as the title of the segment. These segment splits are added manually but
+        # we can avoid the need to also manually add their anchors by doing that here
+        podcast.data["references"].each do |reference|
+          reference["has_transcript_section"] = podcast.content.sub!(/^(_.*?#{Regexp.escape(reference["title"])}.*?_)/, "{:#{reference["slug"]}-transcript}\n \\1")
+        end
+      end
+    end
+  end
+
+
+  def find_title(string, in_list=true, slugify=true)
+    # this conditional prefix is for the special case of the review club section
+    # which is not a list item (no dash (-) at the start of the line)
+    prefix = in_list ? / *- / : // 
+
+    # Find shortest match for **bold**, or [markdown][links]
+    # note: when we are matching the title in `auto-anchor.rb` we also match *italics*
+    # but on the newsletter sections nested bullets have *italics* titles therefore
+    # by ignoring *italics* we are able to easier link to the outer title
+    title = string.match(/^#{prefix}(?:\*\*(.*?):?\*\*|\[(.*?):?\][(\[])/)&.captures&.compact&.[](0) || ""
+    if title.empty?
+      {}
+    else
+      result = {"title"=> title}
+      slug = slugify ? {"slug"=> generate_slug(title)} : {}
+      result.merge!(slug)
+    end
+  end
+
+  # This method searches the content for paragraphs that indicate that they are
+  # part of a podcast recap. When a paragraph is part of a recap we:
+  # - postfix with a link to the related podcast item 
+  # - get the header, title and title slug of the paragraph to create
+  #   the references for the podcast
+  def get_podcast_references(content, target_page_url)
+    # The logic here assumes that:
+    # - paragraphs have headers
+    # - each block of text (paragraph) is seperated by an empty line 
+      
+    # Split the content into paragraphs
+    paragraphs = content.split(/\n\n+/)
+    # Find all the headers in the content
+    headers = content.scan(/^#+\s+(.*)$/).flatten
+
+    # Create an array of hashes containing:
+    # - the paragraph's title
+    # - the paragraph's title slug
+    # - the associated header
+    # - the timestamp of the podcast in which this paragraph is discussed
+    podcast_references = []
+    current_header = 0
+    current_title = {}
+    in_review_club_section = false
+
+    # Iterate over all paragraphs to find those with a podcast reference mark
+    paragraphs.each do |p|
+      # a title might have multiple paragraphs associated with it
+      # the podcast reference mark might be at the end of an isolated
+      # paragraph snippet that cannot access the title, therefore
+      # we keep this information to be used in the link to the podcast recap
+      title = find_title(p, !in_review_club_section)
+      if !title.empty?
+        # paragraph has title
+        current_title = title
+      end
+
+      # If the current paragraph contains the podcast reference mark,
+      # capture the timestamp, add paragraph to references and replace 
+      # the mark with link to the related podcast item
+      p.gsub!($podcast_reference_mark) do |match|
+        if in_review_club_section
+          # the newsletter's review club section is the only section that does
+          # not have a list item to use as anchor so we use the header
+          current_title["podcast_slug"] = "#pr-review-club" # to avoid duplicate anchor
+          current_title["slug"] = "#bitcoin-core-pr-review-club"
+        end
+        podcast_reference = {"header"=> headers[current_header], "timestamp"=> $1}
+        podcast_reference.merge!(current_title)
+        podcast_references << podcast_reference
+
+        # Replace the whole match with the link
+        headphones_link = "[<i class='fa fa-headphones' title='Listen to our discussion of this on the podcast'></i>]"
+        replacement_link_to_podcast_item = "#{headphones_link}(#{target_page_url}#{current_title["podcast_slug"] || current_title["slug"]})"
+      end
+
+      # update to the next header when parse through it
+      if p.sub(/^#+\s*/, "") == headers[(current_header + 1) % headers.length()]
+        current_header += 1
+        in_review_club_section = headers[current_header] == "Bitcoin Core PR Review Club"
+      end
+
+    end
+
+    # Join the paragraphs back together to return the modified content
+    updated_content = paragraphs.join("\n\n")
+
+    [updated_content, podcast_references]
+  end
+end

_plugins/common_utils.rb

@@ -69,7 +69,7 @@ technical documentation and discussion.
       more than 83 bytes of arbitrary data.  They reasoned that other
       methods for storing large amounts of data are currently in use and
       there would be no additional harm from `OP_RETURN` being used
-      instead.  {% include functions/podcast-callout.md url="pod238 storage" %}
+      instead. {% assign timestamp="1:02" %}
 
 - **Fee dilution in multiparty protocols:** Yuval Kogman
   [posted][kogman dilution] to the Bitcoin-Dev mailing list the
@@ -104,7 +104,7 @@ technical documentation and discussion.
   Kogman describes several mitigations in his post, although all of them
   involve tradeoffs.  In a [second post][kogman dilution2], he notes
   that he's unaware of any currently deployed protocol that's
-  vulnerable. {% include functions/podcast-callout.md url="pod238 dilution" %}
+  vulnerable. {% assign timestamp="27:29" %}
 
 - **Tapscript signature malleability:** in an aside to the above-mentioned
   conversation about fee dilution, developer Russell O'Connor
@@ -133,8 +133,7 @@ technical documentation and discussion.
    opened an [issue][bitcoin inquisition #19] to the Bitcoin
    Inquisition repository being used to test [SIGHASH_ANYPREVOUT][topic
    sighash_anyprevout] (APO) to consider having APO commit to additional
-   data to prevent this issue for users of that extension. {% include
-   functions/podcast-callout.md url="pod238 tapscript" %}
+   data to prevent this issue for users of that extension. {% assign timestamp="33:47" %}
 
 ## Changes to services and client software
 
@@ -143,34 +142,30 @@ wallets and services.*
 
 - **Liana wallet adds multisig:**
   [Liana][news234 liana]'s [0.2 release][liana 0.2] adds multisig support using
-  [descriptors][topic descriptors]. {% include functions/podcast-callout.md
-  url="pod238 sc1" %}
+  [descriptors][topic descriptors]. {% assign timestamp="37:35" %}
 
 - **Sparrow wallet 1.7.2 released:**
   Sparrow's [1.7.2 release][sparrow 1.7.2] adds [taproot][topic taproot]
   support, [BIP329][] import and export features (see [Newsletter #235][news235
-  bip329]), and additional support for hardware signing devices. {% include
-  functions/podcast-callout.md url="pod238 sc2" %}
+  bip329]), and additional support for hardware signing devices. {% assign timestamp="38:46" %}
 
 - **Bitcoinex library adds schnorr support:**
   [Bitcoinex][bitcoinex github] is a Bitcoin utility library for the Elixir
-  functional programming language. {% include functions/podcast-callout.md
-  url="pod238 sc3" %}
+  functional programming language. {% assign timestamp="39:20" %}
 
 - **Libwally 0.8.8 released:**
   [Libwally 0.8.8][] adds [BIP340][] tagged hash support, additional sighash
   support including [BIP118][] ([SIGHASH_ANYPREVOUT][topic SIGHASH_ANYPREVOUT]), and
   additional [miniscript][topic miniscript], descriptor, and [PSBT][topic psbt]
-  functions. {% include functions/podcast-callout.md url="pod238 sc4" %}
+  functions. {% assign timestamp="39:59" %}
 
 ## Optech Recommends
 
-[BitcoinSearch.xyz][] is a recently-launched search engine for Bitcoin
+- [BitcoinSearch.xyz][] is a recently-launched search engine for Bitcoin
 technical documentation and discussions.  It was used to quickly find
 several of the sources linked in this newsletter, a vast improvement
 over other more laborious methods we've previously used.  Contributions
-to its [code][bitcoinsearch repos] are welcome. {% include
-functions/podcast-callout.md url="pod238 bitcoinsearch" %}
+to its [code][bitcoinsearch repos] are welcome. {% assign timestamp="40:40" %}
 
 ## Releases and release candidates
 
@@ -179,20 +174,17 @@ projects.  Please consider upgrading to new releases or helping to test
 release candidates.*
 
 - [Core Lightning 23.02rc2][] is a release candidate for a new
-  maintenance version of this popular LN implementation. {% include
-functions/podcast-callout.md url="pod238 cln" %}
+  maintenance version of this popular LN implementation. {% assign timestamp="42:21" %}
 
 - [BTCPay Server 1.7.11][] is a new release.  Since the last release we
   covered (1.7.1), several new features have been added and many bug
   fixes and improvements have been made.  Especially notable, several
   aspects related to plugins and third-party integrations have been
   changed, a migration path away from legacy MySQL and SQLite has been
-  added, and a cross-site scripting vulnerability has been fixed. {% include
-functions/podcast-callout.md url="pod238 btcpay" %}
+  added, and a cross-site scripting vulnerability has been fixed. {% assign timestamp="47:49" %}
 
 - [BDK 0.27.0][] is an update to this library for building Bitcoin
-  wallets and applications. {% include
-functions/podcast-callout.md url="pod238 bdk" %}
+  wallets and applications. {% assign timestamp="49:32" %}
 
 ## Notable code and documentation changes
 
@@ -214,15 +206,15 @@ Proposals (BIPs)][bips repo], and [Lightning BOLTs][bolts repo].*
   backups].  The merged PR adds support for creating, storing, and retrieving
   the encrypted backups.  As noted in the commit messages, the feature
   hasn't yet been fully specified or adopted by other LN
-  implementations. {% include functions/podcast-callout.md url="pod238 cln5361" %}
+  implementations. {% assign timestamp="49:51" %}
 
 - [Core Lightning #5670][] and [#5956][core lightning #5956] make
   various updates to its implementation of [dual funding][topic dual
   funding] based on both recent changes to the [specification][bolts
   #851] and comments from interoperability testers.  Additionally, an
   `upgradewallet` RPC is added to move all funds in P2SH-wrapped outputs
   to native segwit outputs, which is required for interactive channel
-  opens. {% include functions/podcast-callout.md url="pod238 cln5670" %}
+  opens. {% assign timestamp="54:45" %}
 
 - [Core Lightning #5697][] adds a `signinvoice` RPC that will sign a
   [BOLT11][] invoice.  Previously, CLN would only sign an invoice when
@@ -235,20 +227,20 @@ Proposals (BIPs)][bips repo], and [Lightning BOLTs][bolts repo].*
   your node can claim that payment before it arrives.  That not only
   steals your money but, because you signed the invoice, generates very
   convincing evidence that you were paid (this evidence is so convincing
-  that many LN developers call it *proof of payment*). {% include functions/podcast-callout.md url="pod238 cln5697" %}
+  that many LN developers call it *proof of payment*). {% assign timestamp="56:23" %}
 
 - [Core Lightning #5960][] adds a [security policy][cln security.md]
-  that includes contact addresses and PGP keys. {% include functions/podcast-callout.md url="pod238 cln5960" %}
+  that includes contact addresses and PGP keys. {% assign timestamp="57:38" %}
 
 - [LND #7171][] upgrades the `signrpc` RPC <!--sic--> to support the
   latest [draft BIP][musig draft bip] for [MuSig2][topic musig].  The RPC now creates
   sessions linked to a MuSig2 protocol version number so that all
   operations within a session will use the correct protocol.  A
   security issue with an older version of the MuSig2 protocol was
-  mentioned in [Newsletter #222][news222 musig2]. {% include functions/podcast-callout.md url="pod238 lnd7171" %}
+  mentioned in [Newsletter #222][news222 musig2]. {% assign timestamp="58:54" %}
 
 - [LDK #2002][] adds support for automatically resending [spontaneous
-  payments][topic spontaneous payments] that don't initially succeed. {% include functions/podcast-callout.md url="pod238 ldk2002" %}
+  payments][topic spontaneous payments] that don't initially succeed. {% assign timestamp="1:00:01" %}
 
 - [BTCPay Server #4600][] updates the [coin selection][topic coin selection] for its [payjoin][topic payjoin]
   implementation to try to avoid creating transactions with *unnecessary
@@ -257,7 +249,7 @@ Proposals (BIPs)][bips repo], and [Lightning BOLTs][bolts repo].*
   regular single-spender, single-receiver payment: the largest input
   would have provided sufficient payment for the payment output and no
   additional inputs would have been added.
-  This PR was partly inspired by a [paper analyzing payjoins][]. {% include functions/podcast-callout.md url="pod238 btcpay4600" %}
+  This PR was partly inspired by a [paper analyzing payjoins][]. {% assign timestamp="1:00:59" %}
 
 {% include references.md %}
 {% include linkers/issues.md v=2 issues="5361,5670,5956,851,5697,5960,7171,2002,4541,4600" %}
@@ -288,21 +280,3 @@ Proposals (BIPs)][bips repo], and [Lightning BOLTs][bolts repo].*
 [news235 bip329]: /en/newsletters/2023/01/25/#bips-1383
 [bitcoinex github]: https://github.com/RiverFinancial/bitcoinex
 [libwally 0.8.8]: https://github.com/ElementsProject/libwally-core/releases/tag/release_0.8.8
-[pod238 storage]: /en/podcast/2023/02/16/#continued-discussion-about-block-chain-data-storage
-[pod238 dilution]: /en/podcast/2023/02/16/#fee-dilution-in-multiparty-protocols
-[pod238 tapscript]: /en/podcast/2023/02/16/#tapscript-signature-malleability
-[pod238 sc1]: /en/podcast/2023/02/16/#liana-wallet-adds-multisig
-[pod238 sc2]: /en/podcast/2023/02/16/#sparrow-wallet-1-7-2-released
-[pod238 sc3]: /en/podcast/2023/02/16/#bitcoinex-library-adds-schnorr-support
-[pod238 sc4]: /en/podcast/2023/02/16/#libwally-0-8-8-released
-[pod238 bitcoinsearch]: /en/podcast/2023/02/16/#bitcoinsearch-xyz
-[pod238 cln]: /en/podcast/2023/02/16/#core-lightning-23-02rc2
-[pod238 btcpay]: /en/podcast/2023/02/16/#btcpay-server-1-7-11
-[pod238 bdk]: /en/podcast/2023/02/16/#bdk-0-27-0
-[pod238 cln5361]: /en/podcast/2023/02/16/#core-lightning-5361
-[pod238 cln5670]: /en/podcast/2023/02/16/#core-lightning-5670-and-5956
-[pod238 cln5697]: /en/podcast/2023/02/16/#core-lightning-5697
-[pod238 cln5960]: /en/podcast/2023/02/16/#core-lightning-5960
-[pod238 lnd7171]: /en/podcast/2023/02/16/#lnd-7171
-[pod238 ldk2002]: /en/podcast/2023/02/16/#ldk-2002
-[pod238 btcpay4600]: /en/podcast/2023/02/16/#btcpay-server-4600