zapx: Code indexer crashes gitea on i386/x86 with "panic: unaligned 64-bit atomic operation"

[sebastianertz]

Description

On an i386 system gitea 1.17.4 runs perfectly.
Version 1.18.0 crashes when indexing code.

Gitea Version

1.18.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

panic: unaligned 64-bit atomic operation

goroutine 3636 [running]:
runtime/internal/atomic.panicUnaligned()
	/usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x2d
runtime/internal/atomic.Xadd64(0xffd693c, 0x3b)
	/usr/local/go/src/runtime/internal/atomic/atomic_386.s:125 +0x11
github.com/blevesearch/zapx/v15.(*chunkedContentCoder).incrementBytesWritten(...)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/contentcoder.go:114
github.com/blevesearch/zapx/v15.(*chunkedContentCoder).flushContents(0xffd6900)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/contentcoder.go:143 +0x24a
github.com/blevesearch/zapx/v15.(*chunkedContentCoder).Close(...)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/contentcoder.go:110
github.com/blevesearch/zapx/v15.(*interim).writeDicts(0x1196f4a0)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:803 +0x1443
github.com/blevesearch/zapx/v15.(*interim).convert(0x1196f4a0)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:277 +0x2b5
github.com/blevesearch/zapx/v15.(*ZapPlugin).newWithChunkMode(0xd1d3e88, {0x104bf180, 0x8, 0x8}, 0x402)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:72 +0x18b
github.com/blevesearch/zapx/v15.(*ZapPlugin).New(0xd1d3e88, {0x104bf180, 0x8, 0x8})
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:47 +0x43
github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).Batch(0x10ef4800, 0xfd18cb0)
	/go/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.5/index/scorch/scorch.go:412 +0x5ff
github.com/blevesearch/bleve/v2.(*indexImpl).Batch(0x10f3ea00, 0xfd29de0)
	/go/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.5/index_impl.go:310 +0xa0
code.gitea.io/gitea/modules/indexer/bleve.(*FlushingBatch).Flush(0x1090dda0)
	/source/modules/indexer/bleve/batch.go:53 +0x32
code.gitea.io/gitea/modules/indexer/code.(*BleveIndexer).Index(0xe6b66a0, {0xba5d4c0, 0xe6a1f30}, 0xfb205a0, {0xfa2e400, 0x28}, 0xf8f17d0)
	/source/modules/indexer/code/bleve.go:311 +0x407
code.gitea.io/gitea/modules/indexer/code.(*wrappedIndexer).Index(0xf0ffef0, {0xba5d4c0, 0xe6a1f30}, 0xfb205a0, {0xfa2e400, 0x28}, 0xf8f17d0)
	/source/modules/indexer/code/wrapped.go:85 +0x7d
code.gitea.io/gitea/modules/indexer/code.index({0xba5d4c0, 0xe6a1f30}, {0xba6044c, 0xf0ffef0}, 0x81)
	/source/modules/indexer/code/indexer.go:107 +0x11f
code.gitea.io/gitea/modules/indexer/code.Init.func2({0x10e49c20, 0x1, 0x14})
	/source/modules/indexer/code/indexer.go:157 +0x1e0
code.gitea.io/gitea/modules/queue.NewByteFIFOUniqueQueue.func1({0x10e49c20, 0x1, 0x14})
	/source/modules/queue/queue_bytefifo.go:399 +0x3e
code.gitea.io/gitea/modules/queue.(*WorkerPool).doWork(0x10edc8a0, {0xba5d580, 0xe93d700})
	/source/modules/queue/workerpool.go:564 +0x69f
code.gitea.io/gitea/modules/queue.(*WorkerPool).addWorkers.func1()
	/source/modules/queue/workerpool.go:312 +0x5d
created by code.gitea.io/gitea/modules/queue.(*WorkerPool).addWorkers
	/source/modules/queue/workerpool.go:310 +0x25

Git Version

2.35.1

Operating System

eisfair

How are you running Gitea?

Self hosted, binary from dl.gitea.org

Database

SQLite

[sebastianertz]

See also pull #19531
I wrote a little patch for https://github.com/zeripath/zapx/tree/v15.3.6-alignment-fix-branch

--- a/contentcoder.go
+++ b/contentcoder.go
@@ -37,6 +37,10 @@ var (
 )
 
 type chunkedContentCoder struct {
+       // This field requires to be the first one in the struct.
+       // This is to allow 64 bit atomic operations on 32-bit machines.
+       // See: https://pkg.go.dev/sync/atomic#pkg-note-BUG & Gitea issue 21957 and 22279
+       bytesWritten     uint64 // atomic access to this variable
        final     []byte
        chunkSize uint64
        currChunk uint64
@@ -45,7 +49,6 @@ type chunkedContentCoder struct {
        compressed []byte // temp buf for snappy compression
 
        w                io.Writer
-       bytesWritten     uint64 // atomic access to this variable
        progressiveWrite bool
 
        chunkMeta    []MetaData

[lunny]

but #22218 should have fixed this and it's duplicated as #21957

[sebastianertz]

On my x86 system gitea 1.18.0 crash with the above log.
#22218 should not work on my system. With above patch it works.
See: https://pkg.go.dev/sync/atomic#pkg-note-BUG

[sebastianertz]

Here the patch for https://github.com/zeripath/zapx/tree/v15.3.6-alignment-fix-branch

diff --git a/contentcoder.go b/contentcoder.go
index b6e7336..288898a 100644
--- a/contentcoder.go
+++ b/contentcoder.go
@@ -37,6 +37,9 @@ var (
 )
 
 type chunkedContentCoder struct {
+	// atomic access to this variable
+	bytesWritten uint64
+
 	final     []byte
 	chunkSize uint64
 	currChunk uint64
@@ -45,7 +48,6 @@ type chunkedContentCoder struct {
 	compressed []byte // temp buf for snappy compression
 
 	w                io.Writer
-	bytesWritten     uint64 // atomic access to this variable
 	progressiveWrite bool
 
 	chunkMeta    []MetaData
diff --git a/docvalues.go b/docvalues.go
index bd3574d..384e6f4 100644
--- a/docvalues.go
+++ b/docvalues.go
@@ -43,6 +43,9 @@ type docVisitState struct {
 }
 
 type docValueReader struct {
+	// atomic access to this variable
+	bytesRead      uint64
+
 	field          string
 	curChunkNum    uint64
 	chunkOffsets   []uint64
@@ -50,9 +53,6 @@ type docValueReader struct {
 	curChunkHeader []MetaData
 	curChunkData   []byte // compressed data cache
 	uncompressed   []byte // temp buf for snappy decompression
-
-	// atomic access to this variable
-	bytesRead uint64
 }
 
 func (di *docValueReader) size() int {
diff --git a/intDecoder.go b/intDecoder.go
index 6b8396a..bf5b41b 100644
--- a/intDecoder.go
+++ b/intDecoder.go
@@ -21,15 +21,15 @@ import (
 )
 
 type chunkedIntDecoder struct {
+	// atomic access to this variable
+	bytesRead uint64
+
 	startOffset     uint64
 	dataStartOffset uint64
 	chunkOffsets    []uint64
 	curChunkBytes   []byte
 	data            []byte
 	r               *memUvarintReader
-
-	// atomic access to this variable
-	bytesRead uint64
 }
 
 // newChunkedIntDecoder expects an optional or reset chunkedIntDecoder for better reuse.
diff --git a/intcoder.go b/intcoder.go
index 2957fbd..35e4efb 100644
--- a/intcoder.go
+++ b/intcoder.go
@@ -28,6 +28,9 @@ import (
 const termNotEncoded = 0
 
 type chunkedIntCoder struct {
+	// atomic access to this variable
+	bytesWritten uint64
+
 	final     []byte
 	chunkSize uint64
 	chunkBuf  bytes.Buffer
@@ -35,9 +38,6 @@ type chunkedIntCoder struct {
 	currChunk uint64
 
 	buf []byte
-
-	// atomic access to this variable
-	bytesWritten uint64
 }
 
 // newChunkedIntCoder returns a new chunk int coder which packs data into
diff --git a/new.go b/new.go
index f659a5d..3616699 100644
--- a/new.go
+++ b/new.go
@@ -96,6 +96,9 @@ var interimPool = sync.Pool{New: func() interface{} { return &interim{} }}
 // interim holds temporary working data used while converting from
 // analysis results to a zap-encoded segment
 type interim struct {
+	// atomic access to this variable
+	bytesWritten uint64
+
 	results []index.Document
 
 	chunkMode uint32
@@ -146,9 +149,6 @@ type interim struct {
 
 	lastNumDocs int
 	lastOutSize int
-
-	// atomic access to this variable
-	bytesWritten uint64
 }
 
 func (s *interim) reset() (err error) {
diff --git a/posting.go b/posting.go
index dcae817..0c6c4fa 100644
--- a/posting.go
+++ b/posting.go
@@ -96,6 +96,9 @@ var NormBits1Hit = uint64(1)
 
 // PostingsList is an in-memory representation of a postings list
 type PostingsList struct {
+	// atomic access to this variable
+	bytesRead      uint64
+
 	sb             *SegmentBase
 	postingsOffset uint64
 	freqOffset     uint64
@@ -109,9 +112,6 @@ type PostingsList struct {
 	normBits1Hit uint64
 
 	chunkSize uint64
-
-	// atomic access to this variable
-	bytesRead uint64
 }
 
 // represents an immutable, empty postings list
@@ -324,6 +324,9 @@ func (rv *PostingsList) init1Hit(fstVal uint64) error {
 
 // PostingsIterator provides a way to iterate through the postings list
 type PostingsIterator struct {
+	// atomic access to this variable
+	bytesRead uint64
+
 	postings *PostingsList
 	all      roaring.IntPeekable
 	Actual   roaring.IntPeekable
@@ -344,9 +347,6 @@ type PostingsIterator struct {
 
 	includeFreqNorm bool
 	includeLocs     bool
-
-	// atomic access to this variable
-	bytesRead uint64
 }
 
 var emptyPostingsIterator = &PostingsIterator{}
diff --git a/segment.go b/segment.go
index f62918c..0ebeea7 100644
--- a/segment.go
+++ b/segment.go
@@ -89,6 +89,10 @@ func (*ZapPlugin) Open(path string) (segment.Segment, error) {
 // SegmentBase is a memory only, read-only implementation of the
 // segment.Segment interface, using zap's data representation.
 type SegmentBase struct {
+	// atomic access to this variable
+	bytesRead    uint64
+	bytesWritten uint64
+
 	mem               []byte
 	memCRC            uint32
 	chunkMode         uint32
@@ -103,10 +107,6 @@ type SegmentBase struct {
 	fieldDvNames      []string                   // field names cached in fieldDvReaders
 	size              uint64
 
-	// atomic access to this variable
-	bytesRead    uint64
-	bytesWritten uint64
-
 	m         sync.Mutex
 	fieldFSTs map[uint16]*vellum.FST
 }

[watsom27]

Not sure if this is the same issue, but my gitea instance is currently crashing intermittently with this error which (to my non-go eyes) looks to be similar?

panic: unaligned 64-bit atomic operation

goroutine 7553 [running]:
runtime/internal/atomic.panicUnaligned()
	/usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x2d
runtime/internal/atomic.Xadd64(0x1489153c, 0x93)
	/usr/local/go/src/runtime/internal/atomic/atomic_386.s:125 +0x11
github.com/blevesearch/zapx/v15.(*chunkedContentCoder).incrementBytesWritten(...)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/contentcoder.go:114
github.com/blevesearch/zapx/v15.(*chunkedContentCoder).flushContents(0x14891500)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/contentcoder.go:143 +0x24a
github.com/blevesearch/zapx/v15.(*chunkedContentCoder).Close(...)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/contentcoder.go:110
github.com/blevesearch/zapx/v15.(*interim).writeDicts(0x14738f00)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:803 +0x1443
github.com/blevesearch/zapx/v15.(*interim).convert(0x14738f00)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:277 +0x2b5
github.com/blevesearch/zapx/v15.(*ZapPlugin).newWithChunkMode(0xd1d3e88, {0x149682f0, 0x1, 0x1}, 0x402)
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:72 +0x18b
github.com/blevesearch/zapx/v15.(*ZapPlugin).New(0xd1d3e88, {0x149682f0, 0x1, 0x1})
	/go/pkg/mod/github.com/zeripath/zapx/v15@v15.3.6-alignment-fix/new.go:47 +0x43
github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).Batch(0x10a24000, 0x14963110)
	/go/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.5/index/scorch/scorch.go:412 +0x5ff
github.com/blevesearch/bleve/v2.(*indexImpl).Batch(0x109c17c0, 0x149514c0)
	/go/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.5/index_impl.go:310 +0xa0
code.gitea.io/gitea/modules/indexer/bleve.(*FlushingBatch).Flush(0x1498fdc4)
	/source/modules/indexer/bleve/batch.go:53 +0x32
code.gitea.io/gitea/modules/indexer/issues.(*BleveIndexer).Index(0xf78bf70, {0x14968220, 0x1, 0x1})
	/source/modules/indexer/issues/bleve.go:226 +0x183
code.gitea.io/gitea/modules/indexer/issues.InitIssueIndexer.func1({0x147343c0, 0x1, 0x14})
	/source/modules/indexer/issues/indexer.go:147 +0x5b1
code.gitea.io/gitea/modules/queue.NewByteFIFOQueue.func1({0x147343c0, 0x1, 0x14})
	/source/modules/queue/queue_bytefifo.go:71 +0x3e
code.gitea.io/gitea/modules/queue.(*WorkerPool).doWork(0x1091b0e0, {0xba5d580, 0x1473fe00})
	/source/modules/queue/workerpool.go:564 +0x69f
code.gitea.io/gitea/modules/queue.(*WorkerPool).addWorkers.func1()
	/source/modules/queue/workerpool.go:312 +0x5d
created by code.gitea.io/gitea/modules/queue.(*WorkerPool).addWorkers
	/source/modules/queue/workerpool.go:310 +0x25

Running on a t2.micro EC2 Ubuntu box with a binary from dl.gitea.org and connected to a MySQL RDS database

[sebastianertz]

Fix by #22485