llgo/README.md

llgo - A Go compiler based on LLVM

LLGo is a Go compiler based on LLVM in order to better integrate Go with the C ecosystem including Python and JavaScript. It's a subproject of the XGo project.

LLGo aims to expand the boundaries of Go/XGo, providing limitless possibilities such as:

• Game development
• AI and data science
• WebAssembly
• Embedded development
• ...

How can these be achieved?

LLGo := Go * C ecosystem

LLGo is compatible with C ecosystem through the language's Application Binary Interface (ABI), while LLGo is compatible with Go through its syntax (source code). C ecosystem includes all languages that are ABI compatible with C (eg. C/C++, Python, JavaScript, Objective-C, Swift, etc).

C/C++ standard libary support

You can import a C/C++ standard library in LLGo!

• c
• c/syscall
• c/sys
• c/os
• c/math
• c/math/cmplx
• c/math/rand
• c/pthread
• c/pthread/sync
• c/sync/atomic
• c/time
• c/net
• cpp/std

Here is a simple example:

package main

import "github.com/goplus/lib/c"

func main() {
	c.Printf(c.Str("Hello world\n"))
}

This is a simple example of calling the C printf function to print Hello world. Here, c.Str is not a function for converting a Go string to a C string, but a built-in instruction supported by llgo for generating a C string constant.

The _demo directory contains some C standard libary related demos (it start with _ to prevent the go command from compiling it):

• hello: call C printf to print Hello world
• concat: call C fprintf with stderr
• qsort: call C function with a callback (eg. qsort)

To run these demos (If you haven't installed llgo yet, please refer to How to install):

cd <demo-directory>  # eg. cd _demo/c/hello
llgo run .

How to support C/C++ and Python

LLGo use go:linkname to link an extern symbol througth its ABI:

import _ "unsafe" // for go:linkname

//go:linkname Sqrt C.sqrt
func Sqrt(x float64) float64

You can directly integrate it into your own code:

package main

import _ "unsafe" // for go:linkname

//go:linkname Sqrt C.sqrt
func Sqrt(x float64) float64

func main() {
	println("sqrt(2) =", Sqrt(2))
}

Or put it into a package (see c/math):

package main

import "github.com/goplus/lib/c/math"

func main() {
	println("sqrt(2) =", math.Sqrt(2))
}

Python support

You can import a Python library in LLGo!

And you can import any Python library into llgo through a program called llpyg (see Development tools). The following libraries have been included in llgo:

• py (abi)
• py/std (builtins)
• py/sys
• py/os
• py/math
• py/json
• py/inspect
• py/statistics
• py/numpy
• py/pandas
• py/torch
• py/matplotlib

Note: For third-party libraries (such as pandas and pytorch), you still need to install the library files.

Here is an example:

package main

import (
	"github.com/goplus/lib/py"
	"github.com/goplus/lib/py/math"
	"github.com/goplus/lib/py/std"
)

func main() {
	x := math.Sqrt(py.Float(2))       // x = sqrt(2)
	std.Print(py.Str("sqrt(2) ="), x) // print("sqrt(2) =", x)
}

It is equivalent to the following Python code:

import math

x = math.sqrt(2)
print("sqrt =", x)

Here, We call py.Float(2) to create a Python number 2, and pass it to Python’s math.sqrt to get x. Then we call std.Print to print the result.

Let's look at a slightly more complex example. For example, we use numpy to calculate:

package main

import (
	"github.com/goplus/lib/py"
	"github.com/goplus/lib/py/numpy"
	"github.com/goplus/lib/py/std"
)

func main() {
	a := py.List(
		py.List(1.0, 2.0, 3.0),
		py.List(4.0, 5.0, 6.0),
		py.List(7.0, 8.0, 9.0),
	)
	b := py.List(
		py.List(9.0, 8.0, 7.0),
		py.List(6.0, 5.0, 4.0),
		py.List(3.0, 2.0, 1.0),
	)
	x := numpy.Add(a, b)
	std.Print(py.Str("a+b ="), x)
}

Here we define two 3x3 matrices a and b, add them to get x, and then print the result.

The _demo/py/ directory contains some python related demos:

• callpy: call Python standard library function math.sqrt
• pi: print python constants math.pi
• statistics: define a python list and call statistics.mean to get the mean
• matrix: a basic numpy demo

To run these demos (If you haven't installed llgo yet, please refer to How to install):

cd <demo-directory>  # eg. cd _demo/py/callpy
llgo run .

Other frequently used libraries

LLGo can easily import any libraries from the C ecosystem. Currently, this import process is still manual, but in the future, it will be automated similar to Python library imports.

The currently supported libraries include:

• c/bdwgc
• c/cjson
• c/clang
• c/ffi
• c/libuv
• c/llama2
• c/lua
• c/neco
• c/openssl
• c/raylib
• c/sqlite
• c/zlib
• cpp/inih
• cpp/llvm

Here are some examples related to them:

• llama2-c: inference Llama 2 (It's the first llgo AI example)
• mkjson: create a json object and print it
• sqlitedemo: a basic sqlite demo
• tetris: a tetris game based on raylib

Go syntax support

All Go syntax (including cgo) is already supported. Here are some examples:

• concat: define a variadic function
• genints: various forms of closure usage (including C function, recv.method and anonymous function)
• errors: demo to implement error interface
• defer: defer demo
• goroutine: goroutine demo

Defer

LLGo defer does not support usage in loops. This is not a bug but a feature, because we think that using defer in a loop is a very unrecommended practice.

Garbage Collection (GC)

By default, LLGo implements gc based on bdwgc (also known as libgc).

However, you can disable gc by specifying the nogc tag. For example:

llgo run -tags nogc .

Go packages support

Here are the Go packages that can be imported correctly:

• unsafe
• unicode
• unicode/utf8
• unicode/utf16
• math
• math/big (partially)
• math/bits
• math/cmplx
• math/rand
• net/url
• errors
• context
• io
• io/fs
• io/ioutil
• log
• flag
• sort
• bytes
• bufio
• strings
• strconv
• path
• path/filepath
• sync/atomic
• sync (partially)
• syscall (partially)
• runtime (partially)
• os (partially)
• os/exec (partially)
• fmt (partially)
• reflect (partially)
• time (partially)
• encoding
• encoding/binary
• encoding/hex
• encoding/base32
• encoding/base64
• encoding/csv
• net/textproto
• hash
• hash/adler32
• hash/crc32 (partially)
• hash/crc64
• crypto
• crypto/md5
• crypto/sha1
• crypto/sha256
• crypto/sha512 (partially)
• crypto/hmac (partially)
• crypto/rand (partially)
• crypto/subtle (partially)
• regexp
• regexp/syntax
• go/token
• go/scanner
• go/parser

Dependencies

• Go 1.21+
• LLVM 18
• Clang 18
• LLD 18
• pkg-config 0.29+
• bdwgc/libgc 8.0+
• OpenSSL 3.0+
• zlib 1.2+
• Python 3.12+ (optional, for github.com/goplus/lib/py)

How to install

Follow these steps to generate the llgo command (its usage is the same as the go command):

on macOS

brew update
brew install llvm@19 lld@19 bdw-gc openssl cjson libffi libuv pkg-config
brew install python@3.12 # optional
brew link --overwrite llvm@19 lld@19 libffi
# curl https://raw.githubusercontent.com/goplus/llgo/refs/heads/main/install.sh | bash
./install.sh

on Linux

Debian/Ubuntu

echo "deb http://apt.llvm.org/$(lsb_release -cs)/ llvm-toolchain-$(lsb_release -cs)-19 main" | sudo tee /etc/apt/sources.list.d/llvm.list
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install -y llvm-19-dev clang-19 libclang-19-dev lld-19 libunwind-19-dev libc++-19-dev pkg-config libgc-dev libssl-dev zlib1g-dev libcjson-dev libsqlite3-dev libuv1-dev
sudo apt-get install -y python3.12-dev # optional
#curl https://raw.githubusercontent.com/goplus/llgo/refs/heads/main/install.sh | bash
./install.sh

Alpine Linux

apk add go llvm19-dev clang19-dev lld19 pkgconf gc-dev libunwind-dev openssl-dev zlib-dev
apk add python3-dev # optional
apk add g++ # build only
export LLVM_CONFIG=/usr/lib/llvm19/bin/llvm-config
export CGO_CPPFLAGS="$($LLVM_CONFIG --cppflags)"
export CGO_CXXFLAGS=-std=c++17
export CGO_LDFLAGS="$($LLVM_CONFIG --ldflags) $($LLVM_CONFIG --libs all)"
curl https://raw.githubusercontent.com/goplus/llgo/refs/heads/main/install.sh | bash

docker alpine 386 llgo environment

export GCC_ROOT_DIR=$(gcc -print-search-dirs | grep 'install:' | awk -F': ' '{print $2}')
export LDFLAGS="-L$GCC_ROOT_DIR -B$GCC_ROOT_DIR -Wl,-dynamic-linker,/lib/ld-musl-i386.so.1"
llgo run .

on Windows

TODO

Install from source

git clone https://github.com/goplus/llgo.git
cd llgo
./install.sh

Development tools

• pydump: It's the first program compiled by llgo (NOT go) in a production environment. It outputs symbol information (functions, variables, and constants) from a Python library in JSON format, preparing for the generation of corresponding packages in llgo.
• pysigfetch: It generates symbol information by extracting information from Python's documentation site. This tool is not part of the llgo project, but we depend on it.
• llpyg: It is used to automatically convert Python libraries into Go packages that llgo can import. It depends on pydump and pysigfetch to accomplish the task.
• llgen: It is used to compile Go packages into LLVM IR files (*.ll).
• ssadump: It is a Go SSA builder and interpreter.

How do I generate these tools?

git clone https://github.com/goplus/llgo.git
cd llgo
go install -v ./cmd/...
go install -v ./chore/...  # compile all tools except pydump
export LLGO_ROOT=$PWD
cd _xtool
llgo install ./...   # compile pydump
go install github.com/goplus/hdq/chore/pysigfetch@v0.8.1  # compile pysigfetch

Key modules

Below are the key modules for understanding the implementation principles of llgo:

• ssa: It generates LLVM IR files (LLVM SSA) using the semantics (interfaces) of Go SSA. Although LLVM SSA and Go SSA are both IR languages, they work at completely different levels. LLVM SSA is closer to machine code, which abstracts different instruction sets. While Go SSA is closer to a high-level language. We can think of it as the instruction set of the Go computer. llgo/ssa is not just limited to the llgo compiler. If we view it as the high-level expressive power of LLVM, you'll find it very useful. Prior to llgo/ssa, you had to operate LLVM using machine code semantics. But now, with the advanced SSA form (in the semantics of Go SSA), you can conveniently utilize LLVM.
• cl: It is the core of the llgo compiler. It converts a Go package into LLVM IR files. It depends on llgo/ssa.
• internal/build: It strings together the entire compilation process of llgo. It depends on llgo/ssa and llgo/cl.