cmd/go: packages with non root vendor elements excluded from module

[gucio321]

Go version

go version go1.24.0 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/uername/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/uername/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3740879903=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/uername/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/uername/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/uername/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

In #70303 I was told that the bug that packages with /vendor in its name not being in a root of the module should not produce errors anymore but I've tried to upgrade https://github.com/AllenDang/cimgui-go as well as github.com/allendang/giu to go 1.24 and doing go mod tidy on giu still produces an error

What did you see happen?

go: finding module for package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/test/named_subexpressions
go: finding module for package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/test/object_cache
go: finding module for package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/test/pathology
go: finding module for package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/test/regress
go: finding module for package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/test/static_mutex
go: finding module for package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/test/unicode
go: finding module for package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/tools/generate
go: github.com/AllenDang/giu imports
	github.com/AllenDang/cimgui-go/imgui imports
	github.com/AllenDang/cimgui-go imports
	github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/build: module github.com/AllenDang/cimgui-go@latest found (v1.3.0), but does not contain package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/build
go: github.com/AllenDang/giu imports
	github.com/AllenDang/cimgui-go/imgui imports
	github.com/AllenDang/cimgui-go imports
	github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/example/grep: module github.com/AllenDang/cimgui-go@latest found (v1.3.0), but does not contain package github.com/AllenDang/cimgui-go/cwrappers/ImGuiColorTextEdit/vendor/regex/example/grep
go: github.com/AllenDang/giu imports

What did you expect to see?

no error should happen since this bug is supposed to be fixed.

[gabyhelp]

Related Issues

• mod/vendor: broken support for git-submodules in vendoring mode #56920 (closed)
• cmd/go: can use invalid sub-dependency version #69437 (closed)
• cmd/go: bad error message when using `vendor` directory while specifying too old Go version #67587 (closed)
• cmd/go: go mod vendor ignore workspace #54452 (closed)
• cmd/go: Inconsistent vendoring detected. Please re-run "go mod vendor" #45103 (closed)
• cmd/go: error out in 'go mod tidy' if the result uses a module for multiple paths #34650 (closed)
• cmd/go: import of vendored package error "cannot find package" from testdata package #50143

Related Code Changes

• cmd/go: import vendored packages only listed in modules.txt
• cmd/go: show an error when a package in a module conflicts with one in std

Related Discussions

• About the change for default "-mod=vendor" in Go 1.14

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

Issue #37397 was about some packages whose import path ends with "/vendor" being treated as if they're vendored, and being left out of module zips.

It looks like what you're running into isn't that issue, since the import paths in the error message do not end with "/vendor", but instead they contain "/vendor/". See https://pkg.go.dev/cmd/go#hdr-Vendor_Directories for documentation about vendor directories, which applies:

Code below a directory named "vendor" is importable only by code in the directory tree rooted at the parent of "vendor", and only using an import path that omits the prefix up to and including the vendor element.

Does the problem go away if you rename the directory vendor in your repo to something else (e.g., "copy") and update the import paths in dummy.go accordingly?

[bep]

@dmitshur I created issue #37397 -- and while you're right about the failing example being an import path that "ends with /vendor", the intention of the issue was certainly to allow "vendor" folders in all but the root of a module, and that's how I understood that the fix in "x/mod: fix handling of vendored packages with '/vendor' in non-top-level paths" tried to accomplish, but I might have read it wrong.

As the issue starter I can confirm that the issue I reported isn't fixed, as in: It didn't fix MY issue.

[gucio321]

Does the problem go away if you rename the directory vendor in your repo to something else (e.g., "copy") and update the import paths in dummy.go accordingly?

It probably would go away after renaming this, however its pretty difficult to rename it as my code is auto-generated and this vendor directory was called by C dependency's maintainer, not by me.
Also I think that as long as vendor sub directories in the root of the module should deffinitly be ommited, the same behaviour should not apply to vendor's supdirectories in non-top-level packages (this doesn't make sense for me tbh).

This documentation quote imo applies only to the "real" vendor directories (these in top-level module directory) not such a "fake" vendors like mine.

[jsteenb2]

I'm coming here from #71978, it looks to be similar but specific to the embed.FS confusion. For my issue renaming the vendor breaks the usecase for the test type I make use of the embed.FS. Was able to reproduce this is a tiny pair of modules listed in the issue.

@dmitshur, I'm curious if the link you shared re: vendor directories should apply to directories that do not have *.go files within? Take my testdata/foo/vendor/bar/msg.txt file for example. My example doesn't involve importing any go code, just creating a file at the correct path that represents real constraints. Its also deeply nested in that filepath. It makes sense the embed.FS would contain everything within the testdata dir developing within the module that contains it. However, its confusing that go.work doesn't respect the same rules as go get might in this situation. That confusion cost me a fair bit of time.

[dmitshur]

@jsteenb2 The intersection of directories "testdata" and "vendor" is an interesting edge case. (It's possible it already came up and was discussed in a prior issue.) It may be reasonable to expect that testdata directory should be able to contain arbitrary data for testing, possibly including directories named vendor. In practice, there will still be some constraints, such as go.mod files inside testdata still having the effect of carving out that directory from the parent module. And it seems directories named vendor currently have the same behavior - today's implementation of isVendoredPackage only looks for "/vendor/" path elements regardless of whether it's inside a "testdata" directory.

Edit: Also note that embed package documentation hints at this limitation, albeit indirectly given its intersection with this issue: "Patterns must not match files outside the package's module, such as ‘.git/*’, symbolic links, 'vendor/', or any directories containing go.mod (these are separate modules)." Emphasis mine.

Given that the /vendor/ directory started out as having effect anywhere back in GOPATH mode, but has been refined to be primarily at the module root in module mode, maybe this could use reconsideration. It would need a proposal. That's something for cmd/go owners to consider, if they haven't already.

CC @matloob, @samthanawalla via owners.

[jsteenb2]

@dmitshur that makes a ton of sense. As I was debugging this I read up on the embed pkg docs and what it states makes perfect sense if the vendor/ is at the root alongside the go.mod file. However, it still included the testdata/**/vendor/** paths locally and via go.work. It only fell apart when the go get of the module was used.

It would be great if we could include vendor in the testdata dir. I fully expect any rouge go.mod files within the testdata dirs to have the existing behavior applied. I've run into similar with a function as as service offering, having to change the go.mod to a go.mod.tmpl to include it in the embed.FS for bootstrapping new functions for our users.

Would love to see testdata/**/vendor/** supported for the embed usecase. This would make it possible to test that part of our system.