Before we get started, I need to get you on the same page. Throughout this entire article, I will be using the go version 1.18.3. So, I’d recommend you either use the version that I use, version 1.11, or any above to grasp everything properly.
What is GOROOT?
GOROOT is the variable that defines where Go SDK is located. This is where Go’s code, compiler, and rest of the required tooling lives. This folder does not hold our source code. The $GOROOT is something similar to /usr/local/go or /opt/homebrew/Cellar/go/1.X.X/bin.
In older versions of Go, we set $GOROOT manually. But in newer versions, we don’t need to set up the $GOROOT variable unless you use different Go versions. If you install, go in a different path, then export the variable $GOROOT in your shell’s default profile (i.e., .zshrc, .profile).
What is GOPATH?
To explain this, we need to travel back in time. Let’s see how things were before Go 1.11.
How old GOPATH works
When Go was first introduced in 2009, Go authors required you to organize the Go code specifically when working with the go tool. A convention to say. Here’s some simplified information I borrowed from the docs.
- Go programmers typically keep all their Go codes in a single workspace.
- A workspace contains many version control repositories (i.e., managed by Git, Bitbucket, etc.).
- Each repository contains one or more packages.
- Each package consists of Go source files in a single directory.
- The path to a package’s directory determines its import path.
Go authors had this notion called a single workspace directory. It is very different from other programming language environments (i.e., C++) in which the project has a separate workspace and can be multiple workspaces closely tied to version-controlled repositories.
But in Golang, a workspace refers to a directory hierarchy with three directories at its root.
| Directory | Purpose |
|---|---|
| src | Location of your Go source code i.e., .go, .c, .g, .s. The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages. |
| pkg | Location of compiled package code (i.e., .a). For example, when you run go install, you can use it in your code. |
| bin | Location of compiled executable programs built by Go. The go tool builds and installs binaries to this directory. |
To give a rough idea of how a workspace looks in practice, here’s an example:
pkg/
darwin_arm64/
golang.org/x/
image/
bmp.a # compiled object
example/
stringutil.a # compiled object
bin/
hello # command executable
outyet # command executable
src/
golang.org/x/example/
.git/ # Git repository metadata
hello/
hello.go # command source
outyet/
main.go # command source
main_test.go # test source
stringutil/
reverse.go # package source
reverse_test.go # test source
golang.org/x/image/
.git/ # Git repository metadata
bmp/
reader.go # package source
writer.go # package source
... (many more repositories and packages omitted) ...Well, now let’s take this convention into practice and understand how it was before back then.
HolUp ✋ Wait a Minute
From this point onwards, we will look at how GOPATH used to be in earlier versions before 1.11. And for that, we need to turn off module-aware mode from environment variables. You can check whether you have it enabled or disabled by using the following command.
$ go env GO111MODULEThis will show the value of the GO111MODULE variable.
If you don’t see the value as off, I want you to overwrite the default value by executing go env -w GO111MODULE=off to explicitly turn off the module-aware mode. Now we can make sure that we use the semantics before 1.11.
# disable
$ go env -w GO111MODULE=off
# enable
$ go env -w GO111MODULE=onNow that Go module-aware mode is disabled, the packages we develop and install should be in $GOPATH so that the Go build system knows where the imported packages are.
Now go tool expects you to keep your project and source files in GOPATH/src. And go-tool uses pkg/ for compiled packages and the bin/ for executables. This gives you all the necessary files for your development, and go-tool can resolve packages you have imported into your project.
Why learn if it’s obsolete?
Well, I’m glad you asked! Frankly, you don’t need to. Especially if you were a Gopher from the very beginning. But if you’re just starting out, you might run into slight hiccups here and there from time to time because we still have content that refers to older Go $GOPATH behavior. So, it’s always better to know the history.
Okay, now let’s see an example.
We can import third-party packages (i.e., libraries written Go, C, or even C++) or our own custom packages to our programs. For example, consider the below application.
Let’s say we want to create a calculator application. And along the way, publish our calculator operations as a reusable module so that other developers can reuse them. And make the interface and the logic separately in a different package as a driver application. And driver uses a third-party package called chalk to change the output colors.
Sounds easy, right? Let’s see how we can do that.bash
$ mkdir -p $GOPATH/src/operations # create directory
$ cd $GOPATH/src/operations # navigate to the directory
$ touch operations.go # create package fileOpen the operations.go file in your editor and paste in the following source.
package operations
func Add(a int, b int) int {
return a + b
}
func Sub(a int, b int) int {
return a - b
}Okay, we have already created our reusable operations above. But how can we practically compile and use it in other projects?
Well, obviously we need another package to do so. But first, we need to execute go install inside the $GOPATH/src/operations package to create a compiled binary to use in other applications.
$ cd $GOPATH/src/operations && go installIf you navigate to $GOPATH/pkg, you will see that operations.a compiled binary file will be generated in $GOPATH/pkg/{GOOS}_${GOARCH} directory.
├── bin
├── pkg
│ └── darwin_arm64
│ └── operations.a
└── src
└── operations
└── operations.go
5 directories, 2 filesNow that we have a binary file, we can actually go ahead and create a new application package called calcapp*.bash
$ mkdir -p $GOPATH/src/calcapp # create directory
$ cd $GOPATH/src/calcapp # navigate to application directory
$ touch main.go # create driver main fileRemember that we want the third-party library called chalk to format our output. So let’s go ahead and install that too.
$ go get github.com/ttacon/chalkYou can execute this command from anywhere in your machine as long as you have $GOPATH in your environment variables.
There is quite a bit happening in the background. First, Without Go 11 modules enabled, the package we get from go get should be in $GOPATH so that the Go build system knows where the imported packages are.
├── bin
├── pkg
│ └── darwin_arm64
│ ├── github.com
│ │ └── ttacon
│ │ └── chalk.a
│ └── operations.a
└── src
├── calcapp
│ └── main.go
├── github.com
│ └── ttacon
│ └── chalk
│ ├── LICENSE
│ ├── README.md
│ ├── chalk.go
│ ├── chalk_test.go
│ ├── doc.go
│ ├── examples
│ │ └── main.go
│ └── img
│ └── chalk_example.png
└── operations
└── operations.go
13 directories, 11 filesFirst, Go fetches the package chalk and then puts its source under $GOPATH/src in a domain/org/package manner. And it installs the package and places its compiled binary in $GOPATH/pkg/${GOOS}_${GOARCH} in the same way as we talked about.
Now we can start writing our driver application.
main.go
package main
import (
"fmt"
"github.com/ttacon/chalk"
"operations"
)
func main() {
log(fmt.Sprintf("Add: %d", operations.Add(2, 2)))
log(fmt.Sprintf("Subtract: %d", operations.Sub(4, 2)))
}
func log(message string) {
fmt.Println(
chalk.Green,
message,
chalk.ResetColor,
)
}Now we can run the driver application by executing go run main.go and see the output.
$ go run main.go
Add: 4
Subtract: 2Now, if you execute go install and check /bin, you can see that the executable is put. Navigate to $GOPATH/src/calcapp and execute the following.
$ go install && tree $GOPATH
/Users/yasin/go
├── bin
│ └── calcapp
├── pkg
│ └── darwin_arm64
│ ├── github.com
│ │ └── ttacon
│ │ └── chalk.a
│ └── operations.a
└── src
├── calcapp
│ └── main.go
├── github.com
│ └── ttacon
│ └── chalk
│ ├── LICENSE
│ ├── README.md
│ ├── chalk.go
│ ├── chalk_test.go
│ ├── doc.go
│ ├── examples
│ │ └── main.go
│ └── img
│ └── chalk_example.png
└── operations
└── operations.go
13 directories, 12 files
$ ./calcapp # Works because I have added $GOPATH/bin to $PATH
Add: 4
Subtract: 2With this, we explored all subdirectories src/, pkg/, and bin/ in the root workspace directory. Well, that’s not it. We have a couple of more things left to learn about the old $GOPATH.
Can We Place Our Project Outside $GOPATH?
When I started learning Go back in 2017, I didn’t put my go files in $GOPATH. My dumbazz didn’t refer the docs properly. But it turns out, it strangely compiled the code, and it worked for one main package. Even when GO111MODULE is disabled or in older versions of Go, we can place our projects outside the go path. This is what I mean: –
Here’s a Working Example
I’m implementing a project named myproject, with a package main and including two files main.go and some_functions.go as follows:
$ mkdir -p $HOME/Desktop/myproject
$ cd $HOME/Desktop/myproject
$ touch main.go some_functions.gosome_functions.go
package main
import "fmt"
func Test1() {
fmt.Printf("hello ")
}
func Test2() {
fmt.Printf("world!\n")
}main.go
package main
func main() {
Test1()
Test2()
}And run it like this: –
$ cd $HOME/Desktop/myproject && go run .
hello world!And voilà! Surprisingly, the program runs, even when the project is outside $GOPATH. Actually, all projects must be in $GOPATH is due to sub-packages. In the above example, we only had the main package. So, it makes sense why it worked. And one main package is not the case, so it is not confined by this limitation. But when you add another package to the project, things get a bit fussy.
Here’s a Problematic Example
$ mkdir -p $HOME/Desktop/myproject2
$ cd $HOME/Desktop/myproject2 && touch main.go
$ mkdir functions && cd functions && touch some_functions.gosome_functions.go
package functions
import "fmt"
func Test1() {
fmt.Printf("hello ")
}
func Test2() {
fmt.Printf("world!\n")
}main.go
package main
import "myproject2/functions" // ❌ Compilation error.
// import "functions" // ❌ Big nope.
func main() {
functions.Test1()
functions.Test2()
}Go is unable to resolve the package by it’s relative path or package name.
Without the Go module feature enabled, We cannot specify our functions/ package in the main package, so we cannot find out which should be in the import statement “function” location and cannot build this project. The only way to make it work without Go modules was to move the project into $GOPATH/src like so: –
$ tree $GOPATH
/Users/yasin/go
├── bin
│ ...( executables ) ...
├── pkg
│ ...( compiled objects ) ...
└── src
├── myproject2
│ ├── functions
│ │ └── some_functions.go
│ └── main.go
|
...( more packages ) ...
15 directories, 14 filesmain.go
package main
import "myproject2/functions" // 🎉
func main() {
functions.Test1()
functions.Test2()
}When you move the project inside GOPATH/src go is able to resolve the package by it’s relative path.
This was a strictly opinionated approach back in the day, and every Gopher had to follow this convention and maintain their source like this back in the day. Conceptually, this allowed Gophers to link any go code at any instant of time without ambiguity. Well seems pretty reasonable, isn’t it? But, NO!.
Well, what’s the problem then?
The above $GOPATH approach worked well for a cohesive, more extensive monorepos that doesn’t rely on third-party packages*.
Problem
Imagine you have semantic versioning in third-party packages and including your own. So with time, it will get hairy real quick. Without any versioning for the packages, it led to all this pain of managing different tags in the source.
Because of this, Go authors decided to introduce the GO111MODULE environment variable to handle it. Before Go 1.11, authors didn’t consider shipping the go tool with a package manager. Instead, we used go get to fetch all the sources by using their repository import path and placing them in $GOPATH/src. Since there was no package manager or any versioning, the master branch would represent a stable version of the package.
When Google released Go 1.11, they said: This release adds preliminary support for a new concept called “modules,” an alternative to GOPATH with integrated support for versioning and package distribution. Go 1.11 was released on August 24, 2018.
Gotcha
You might see references to Go Modules as vgo. Basically, it means the same thing. And it implies Versioned Go. Instead of using the $GOPATH for storing a single git checkout of every package, Go Modules stores tagged versions with go.mod keeping track of each package’s version 🥳.
However, Go runtime still does use $GOPATH as a download directory for Go packages. To make Google’s saying correct, the Go module does not entirely replace $GOPATH. The go tool uses it for package distribution and versioning. The main goal of go modules is to distribute modules in a much more streamlined way. And now we are no longer confined to GOPATH. So, placing sources under src/ folder is ineffective and is not the best practice when you have module-aware mode enabled.
The interaction between the GOPATH behavior and the Go Modules behavior has become one of Go’s most significant turning points. Finally, now we can learn how it works in practice.
How GOPATH + GO111MODULE Works
Now it’s time to see how GOPATH works with Go Modules. I’ll give you a similar example we tried above and modify it to cover the things I mention below.
- How to import locally created modules into a project.
- How to use remote modules installed via
go get. - How to use module sub-packages.
Switch to Go Modules
From this point onwards, we are using the modern approach. We are going to turn on GO111MODULE from environment variables.
$ go env -w GO111MODULE=onAlso, to increase the clarity of the tutorial, I will be cleaning up my $GOPATH and previous examples from my workspace.
$ sudo rm -rf ~/Desktop/myproject $GOPATH/* && cd $GOPATH && mkdir -p src pkg binUse this to command wisely. Don’t run this if you have any other projects in your $GOPATH.
Go ahead and create a workspace anywhere*. First, let’s start off with the operations package.
$ DIR="$HOME/Desktop/go_workspace" # My root workspace
$ mkdir -p $DIR/operations && cd $DIR/operations # Create package directory.
$ go mod init operations # Initialize go module.
$ touch $DIR/operations/operations.go # Create the source file.Paste the following source in operations.go.
operations.go
package operations
func Add(a int, b int) int {
return a + b
}
func Sub(a int, b int) int {
return a - b
}Then, let’s write the driver application calcapp.
$ mkdir -p $DIR/calcapp $DIR/calcapp/formatters # Create package directory.
$ cd $DIR/calcapp && go mod init calcapp # Initialize go module.
$ touch $DIR/calcapp/formatters/formatters.go $DIR/calcapp/main.go # Create source files.Note that we create a sub package named calcapp/formatters.
Also, for the driver application, we need a third-party package called chalk.
$ go install github.com/ttacon/chalk # Installs the chalk package.
$ go mod tidy # Sync sumsRemember, with go 1.18, go get no longer builds or installs packages in module-aware mode. So we install it!
Paste the following source in formatters.go.
formatters.go
package formatters
import (
"fmt"
"github.com/ttacon/chalk"
)
func Red(message string) {
fmt.Println(
chalk.Red,
message,
chalk.ResetColor,
)
}
func Green(message string) {
fmt.Println(
chalk.Green,
message,
chalk.ResetColor,
)
}Now notice that we have a custom local package called operations. We need to import that package into our calcapp to make it work. So what do we do? To point to the local version of a dependency in Go rather than the one over the web, we use the replace keyword within the go.mod file.
The replace line goes above your require statements, like so: –
calcapp/go.mod
module calcapp
go 1.18
// Can be github.com/yourname/operations
replace favtuts.com/operations => ../operations
// 💡 The actual semantic version hash will be different in yours.
require github.com/ttacon/chalk v0.0.0-20160626202418-22c06c80ed31
// This is a random version number I added. You can actually put any semantic version here
require favtuts.com/operations v0.0.0And now, when you compile calcapp module using go install, it will use your local code rather than resolve a non-existing web dependency.
Replace Directive
According to the docs, when using replace, you need to make sure that the code you’re pointing to also has a go.mod file. If not, initialize a module using go mod init.
We can safely paste the following source in main.go.
main.go
package main
import (
"calcapp/formatters"
"flag"
"fmt"
"favtuts.com/operations"
)
func main() {
isSubtraction := flag.Bool("sub", false, "subtraction operation")
aValue := flag.Int("a", 0, "a value")
bValue := flag.Int("b", 0, "b value")
flag.Parse()
if *isSubtraction {
formatters.Red(
fmt.Sprintf(
"Subtraction: %d",
operations.Sub(*aValue, *bValue),
),
)
} else {
formatters.Green(
fmt.Sprintf(
"Addition: %d",
operations.Add(*aValue, *bValue),
),
)
}
}Did you notice? We can directly import our package by a path like calcapp/formatters and even reference local modules effortlessly! How cool is that? Go mod is more intelligent than this. It can even recursively resolve multiple nested sub-packages.
$ cd $DIR/calcapp
$ go install
$ calcapp -a 10 -b 10 # => Addition: 20
$ calcapp -sub -a 10 -b 5 # => Subtraction: 5Install the program and run it 🥳!
Conclusion
Now we know with Go module-aware mode enabled, Go projects are no longer confined to $GOPATH. Meaning Go never restricts the structure or the location of Go projects. Go module alleviates versioning and module resolution constraints elegantly. I hope now you have a better understanding of $GOPATH and $GOROOT.
Download Source Code
$ git https://github.com/favtuts/golang-tutorial-beginners.git
$ cd go-modules-demo
Thanks for reading 🥰. Until next time!