We don't need to use a regular expression to find "goN.M" anymore,
as go/version seems to deal with "devel" versions from master just fine.
We can then also stop having two separate fields for the version
of the Go toolchain currently being used.
* reflect_abi_patch.go was added into reflect.go
* shared.go was renamed into cache_shared.go and package caching was moved to cache_pkg.go
* transformer methods in main.go are moved to transformer.go
The behavior is the same, but now we track the dependency in go.mod
via `go get -tool`, which is a better approach.
While here, make a package loading error slightly clearer.
While strictly speaking it would be okay to leave Go 1.24 support
in place for the time being, we are behind on a few tasks at the moment
so it's best to keep the setup at master simpler for the next release.
Go 1.25 already came out two weeks ago, and it seems to have been
a fairly smooth release, so I don't suspect any end users will have
trouble upgrading to it.
Note that two changes were necessary for garble to work on Go 1.25.0.
First, we stop deduplicating runtimeAndLinknamed with runtimeAndDeps.
Otherwise, for GOOS=windows, internal/runtime/cgroup would be missing
as it is a //go:linkname target from runtime on all platforms,
but it is not transitively imported from runtime on GOOS=windows.
Second, the testing/synctest package is now part of std,
and it is a //go:linkname target from the testing package
but not a transitive import from it. Teach appendListedPackages that,
when loading all packages for a `go test` run, it should load
the new testing/synctest package too.
Fixes#968.
Due to unsafe not being a real dependency, type checking during control-flow obfuscation was performed incorrectly.
This is fixed by excluding unsafe from the dependency checks.
Fixes#903
Some programs which could automatically reverse string literals obfuscated with `-literals` exist.
They currently work by emulating the string literal decryption functions we insert.
We prevent this naive emulation from succeeding by making the decryption functions dependent on global state.
This can still be broken with enough effort, we are curious which approach reverse-engineers come up with next, we certainly still have some ideas to make this harder.
Fixes#926
---------
Co-authored-by: Paul Scheduikat <lu4p@pm.me>
The way computePkgCache caches per-package results in GARBLE_CACHE,
under normal circumstances where a user isn't deleting GARBLE_CACHE
we will only load gob files for direct imports, not indirect ones.
Hence, loading a package's gob file twice is not happening normally.
And the way we use go/types, we don't need to set Config.GoVersion.
Some Go version managers like github.com/voidint/g use GOROOT symlinks,
which silently broke the way we patch the linker via go build -overlay.
Reproduced the original crash via the following testscript:
env GARBLE_CACHE=${WORK}/garble-cache
symlink goroot -> /usr/lib/go
env GOROOT=${WORK}/goroot
exec garble run main.go
-- main.go --
package main
import "fmt"
func main() {
fmt.Println("hello world")
}
We don't commit this testscript given how it's an expensive test
and for a relatively rare edge case whose fix is now well documented.
Moreover, as GOTOOLCHAIN is now available, I expect version managers
for Go to fade away with time.
While here, remove a debugging 'exec cat' from a testscript.
Fixes#915.
Or any other package which uses a //go:linkname to the runtime names
"lastmoduledatap" or "moduledataverify1". These are used by sonic
to inject function headers to the runtime, which does not work
as garble patches the runtime as part of obfuscation.
The way it alters the magic number in function headers breaks this.
Add a summary of this as a comment too.
Fixes#898.
When obfuscating a main package whose Go files all import "C",
the first Go file in CompiledGoFiles ends up being _cgo_gotypes.go.
We cannot add our code from reflect_abi_code.go there,
as it leads to the following error about its _trieNode type:
typecheck error: $WORK/b001/_cgo_gotypes.go:185:10: cannot define new methods on non-local type _trieNode
Avoid patching any _cgo_*.go file with our reflect code,
as all of those files are special glue code for cgo.
While here, tweak reflectMainPrePatch to return a string
for consistency with abiNamePatch.
Fixes#916.
Before Go 1.24, `go build` only stamped module versions for modules
resolved via GOPROXY, as the local module only had VCS information.
For that reason, we manually built a pseudo-version from the VCS
timestamp and revision stamped for local builds.
Go 1.24 started stamping the main module with a module version
derived from the local VCS information, much like we already did.
For example, comparing a clean build before this change
against a build with this uncommitted change:
$ go install
$ garble version
mvdan.cc/garble v0.14.3-0.20250413182748-e97968ccae46
[...]
$ git stash pop
$ go install
$ garble version
mvdan.cc/garble v0.14.3-0.20250413182748-e97968ccae46+dirty
[...]
The only user-visible change is that local builds with any
uncommitted changes now get a `+dirty` suffix, but that's probably
a good thing for the majority of users, and provides a useful hint
in case a user forgot about local changes.
The test logic to inject VCS information via an env var
and see that the resulting pseudo-version is what we expect can go too,
as that was testing our own main module version logic.
We now rely on `go build` to do the right thing, so don't test that.
When creating a new debugdir directory, add a sentinel .garble-debugdir
file at its root so that we can later know that we created it
and the user is very unlikely to have left important data there.
When emptying an existing debugdir directory, only do so if it has
that sentinel file, for added safety.
Fixes#932.
We call `go list` to collect information about all the packages
to obfuscate and build, which is crucial to be able to perform
obfuscation of names used across packages.
However, when GOTOOLCHAIN causes a toolchain upgrade,
we must ensure that we use the upgraded Go tool;
otherwise we are mixing information from different toolchain versions.
Fixes#934.
debugdir.txtar also needed tweaking as runtime/map.go is gone
starting in Go 1.24.
Finally, modinfo.txtar needed tweaking since Go 1.24 started stamping
Go binaries with VCS-derived module versions, so we no longer end up
with empty "(devel)" versions.
We only did this for Container in the type switch, but not for Struct.
The added test case panics otherwise.
Just like in the previous case, we still don't need to recurse
into type parameters for fieldToStruct to be filled correctly.
Fixes#899
Allows us to remove EmbeddedAliasFields, recordedObjectString,
and all the logic around using them.
Resolves an issue where a user was running into a panic in
our logic to record embedded aliases.
Note that this means we require Go 1.23.5 or later now,
which also meant some changes to goversion.txtar to keep it green.
Fixes#827.
obfuscatedImportPath already handled ToObfuscate, so the callers
don't have to do that as well. Handle main packages too, whose logic
was sprinkled and repeated throughout the project.
Reflection can show package names alone via reflect.Type.String,
and I'm sure they are available in other ways.
We were obfuscating "package p" exactly like the import path "foo.com/p"
which worked OK for the most part, but not for reflection.
This is also a slight improvement to the quality of the obfuscation,
given that package names and import paths will no longer be aligned
when obfuscated.
There's no need to reach for sharedCache.ListedPackages
when the caller is computePkgCache, which already has the lpkg value.
While here, compare *go/types.Package pointers directly
rather than via their import path strings.
Go code can retrieve and use field and method names via the `reflect` package.
For that reason, historically we did not obfuscate names of fields and methods
underneath types that we detected as used for reflection, via e.g. `reflect.TypeOf`.
However, that caused a number of issues. Since we obfuscate and build one package
at a time, we could only detect when types were used for reflection in their own package
or in upstream packages. Use of reflection in downstream packages would be detected
too late, causing one package to obfuscate the names and the other not to, leading to a build failure.
A different approach is implemented here. All names are obfuscated now, but we collect
those types used for reflection, and at the end of a build in `package main`,
we inject a function into the runtime's `internal/abi` package to reverse the obfuscation
for those names which can be used for reflection.
This does mean that the obfuscation for these names is very weak, as the binary
contains a one-to-one mapping to their original names, but they cannot be obfuscated
without breaking too many Go packages out in the wild. There is also some amount
of overhead in `internal/abi` due to this, but we aim to make the overhead insignificant.
Fixes#884, #799, #817, #881, #858, #843, #842Closes#406
This allows me to collect a full CPU profile, showing us that we clearly
spend too much CPU time in garbage collection.
When collecting a full memory profile, we can see where the allocations
come from now:
Showing nodes accounting for 5636770, 31.07% of 18141579 total
Dropped 630 nodes (cum <= 90707)
Showing top 10 nodes out of 278
flat flat% sum% cum cum%
1692229 9.33% 9.33% 1910679 10.53% encoding/gob.decStringSlice
1005596 5.54% 14.87% 1005596 5.54% golang.org/x/tools/go/ssa.buildReferrers
458753 2.53% 17.40% 458753 2.53% go/scanner.(*Scanner).scanIdentifier
458752 2.53% 19.93% 458752 2.53% reflect.(*structType).Field
425984 2.35% 22.28% 448136 2.47% go/parser.(*parser).parseIdent
390049 2.15% 24.43% 390049 2.15% golang.org/x/tools/go/ssa.(*BasicBlock).emit
327683 1.81% 26.23% 371373 2.05% golang.org/x/tools/go/ssa.NewConst
311296 1.72% 27.95% 1024551 5.65% mvdan.cc/garble.(*transformer).transformGoFile.func1
287891 1.59% 29.54% 287891 1.59% encoding/gob.decString
278537 1.54% 31.07% 657043 3.62% golang.org/x/tools/go/ssa.(*builder).compLit
This method can work for any invocation of garble,
but for now we only directly wire it up for `go test -bench`.
It can still be used for regular invocations of `garble build`.
x/exp/rand was being used for no apparent reason; use math/rand.
x/exp/maps and x/exp/slices can be replaced with maps and slices
respectively now that we require Go 1.23 or later.
Note that the APIs are slightly different due to iterators.
This lets us start taking advantage of featurs from Go 1.23,
particularly tracking aliases in go/types and iterators.
Note that we need to add code to properly handle or skip over the new
*types.Alias type which go/types produces for Go type aliases.
Also note that we actually turn this mode off entirely for now,
due to the bug reported at https://go.dev/issue/70394.
We don't yet remove our own alias tracking code yet due to the above.
We hope to be able to remove it very soon.
Otherwise we miscalculate int sizes, type sizes, alignments, and so on.
Caught by the GOARCH=386 go test on CI, since the os package imports
internal/syscall/unix, which uses arch-dependent padding.
The different padding between our incorrect use of go/types
and the correct typechecking done by the compiler caused different
obfuscation of fields, as the struct types stringified differently,
and they are used as a hash salt for field name obfuscation.
Right now, we only have linker patches for Go 1.22.x.
We only ever maintain those for one or two major Go versions at a time.
If a user tries to use the Go toolchain from 1.21, we already fail
with "Go version too old" messages early on, but we don't for 1.23,
causing a relatively confusing error later on when we link a binary:
cannot get modified linker: cannot retrieve linker patches: open patches/go1.23: file does not exist
Instead, fail early and with a good error message.
Panicking in small helpers or in funcs that don't return error
has proved useful to keep code easier to maintain,
particularly for cases that should typically never happen.
However, in these cases we can error just as easily.
In particular, I was getting a panic whenever I forgot
that I was running garble with Go master (1.23), which is over the top.
We no longer need to worry about the scope of range variables,
we can iterate over integers directly, and we can use cmp.Or too.
I haven't paid close attention to using these everywhere.
This is mainly testing out the new features where I saw some benefit.
That is, the assembly line
#include "foo.h" // bar
would make garble run into an error, as we would try to parse
the #include directive before we stripped comments,
effectively trying to unquote the string
"foo.h" // bar
rather than just the included filename
"foo.h"
Add test cases for asm but also cgo, while at it.
Fixes#812.
TypeName.Pkg is documented as:
Pkg returns the package to which the object belongs.
The result is nil for labels and objects in the Universe scope.
When a struct type embeds a builtin alias type, such as byte,
this would lead to a panic since we assumed we could use the Pkg method.
Fixes#798.
This was a long standing TODO, and a user finally ran into it.
The fix isn't terribly straightforward, but I'm pretty happy with it.
Add a test case where the same struct field is identical
with no tag, with the "tagged1" json tag, and again with "tagged2".
While here, we add a test case for a regular named field too.
Fixes#801.
Go 1.21.0 was released in August 2023, so our upcoming release
will no longer support the Go 1.20 release series.
The first Go 1.22 release candidate is also due in December 2023,
less than a month from now, so dropping 1.20 will simplify 1.22 work.
Otherwise we have to update that `@semver` string
alongside the regular x/tools updates in go.mod.
There's no reason to separate the two versions either.
