Skip to content

Reduce memory usage when rendering markdown #20326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

Reduce memory usage when rendering markdown #20326

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
b9194bf
Reduce memory usage when rendering markdown
lunny Jul 12, 2022
e46f754
Reduce memory usage when rendering markdown
lunny Jul 12, 2022
ea3b13e
Merge branch 'main' into lunny/performance_renderer
lunny Jul 12, 2022
e2f2300
Fix bug
lunny Jul 12, 2022
4c2c466
Apply suggestions from code review
lunny Jul 12, 2022
d121484
Merge branch 'main' into lunny/performance_renderer
6543 Jul 12, 2022
424b7f8
Merge branch 'main' into lunny/performance_renderer
6543 Jul 13, 2022
d2bd8c2
Merge branch 'master' into lunny/performance_renderer
6543 Jul 16, 2022
d6ee57a
add BenchmarkPostProcess
6543 Jul 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions integrations/benchmarks_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (

repo_model "code.gitea.io/gitea/models/repo"
"code.gitea.io/gitea/models/unittest"
"code.gitea.io/gitea/modules/markup"
api "code.gitea.io/gitea/modules/structs"
)

Expand Down Expand Up @@ -70,3 +71,11 @@ func BenchmarkRepoBranchCommit(b *testing.B) {
}
})
}

func BenchmarkPostProcess(b *testing.B) {
input := strings.Repeat("a", 1024)
b.ReportAllocs()
for i := 0; i < b.N; i++ {
markup.PostProcess(&markup.RenderContext{}, strings.NewReader(input), io.Discard)
}
}
31 changes: 20 additions & 11 deletions modules/markup/html.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
package markup

import (
"bytes"
"io"
"net/url"
"path"
Expand Down Expand Up @@ -298,26 +297,36 @@ var (
nulCleaner = strings.NewReplacer("\000", "")
)

func postProcess(ctx *RenderContext, procs []processor, input io.Reader, output io.Writer) error {
defer ctx.Cancel()
// FIXME: don't read all content to memory
rawHTML, err := io.ReadAll(input)
type cleanReader struct {
io.Reader
}

func (c cleanReader) Read(bs []byte) (int, error) {
original := make([]byte, len(bs))
n, err := c.Reader.Read(original)
if err != nil {
return err
return n, err
}
n = copy(bs, tagCleaner.ReplaceAll([]byte(nulCleaner.Replace(string(original[:n]))), []byte("&lt;$1")))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think it is correct.

The content is incomplete in the original buffer, the tagCleaner may not work correctly. And tagCleaner is quite a complex regexp, I am not sure whether it works for incomplete content.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And one more thing, after the replacing, the returned string may be longer than before, it may overflow the bs buffer.

return n, nil
}

var _ io.Reader = cleanReader{}

func postProcess(ctx *RenderContext, procs []processor, input io.Reader, output io.Writer) error {
defer ctx.Cancel()

res := bytes.NewBuffer(make([]byte, 0, len(rawHTML)+50))
// prepend "<html><body>"
_, _ = res.WriteString("<html><body>")
htmlTagPrefix := strings.NewReader("<html><body>")

// Strip out nuls - they're always invalid
_, _ = res.Write(tagCleaner.ReplaceAll([]byte(nulCleaner.Replace(string(rawHTML))), []byte("&lt;$1")))
body := cleanReader{input}

// close the tags
_, _ = res.WriteString("</body></html>")
htmlTagSuffix := strings.NewReader("</body></html>")

// parse the HTML
node, err := html.Parse(res)
node, err := html.Parse(io.MultiReader(htmlTagPrefix, body, htmlTagSuffix))
if err != nil {
return &postProcessError{"invalid HTML", err}
}
Expand Down