Building your own HTTP Server Using Go

I've recently been using CodeCrafters which is a platform to practice writing complex software. They offer a catalog of challenges that break down real world technologies into bite sized pieces that are approachable and as a whole can help you become a better developer.

Here is the current list of challenges:

What I like most about this platform is that it's very real world oriented.

These challenges mimick patterns and technologies used heavily in industry rather than typical Leetcode style challenges that you may see during a technical interview but not in a typical work environment

I feel that these challenges can help you become a better developer as it allows you to dive deep into protocols / methodologies while strengthening your use of a coding language and its standarad library. I will say that Leetcode style challenges do strengthen your understanding of data structures and algorithms, but CodeCrafters does a fantastic job with providing more real-world challenges that you see in industry.

Some other benefits to this platform include:

1. A robust test harness to get back feedback instantly
2. Git integration to run these tests and also the ability to push a scaffolded repository of your work
3. Extensions on top of challenges that offer additional tasks which are constantly being revised
4. Code Examples / Screencasts / Concepts for each Instruction Task to keep you going and help with any blockers that may pop up
5. Language Tracks to help you get better in a particular language

I don't have any affiliation to CodeCrafters, but I do highly recommend them based on my experience with this challenge and their overall platform!

You can find the full implementation here: https://github.com/kavinaravind/codecrafters-http-server-go

---

Challenge Walkthrough

I took a stab at Building your own HTTP server from scratch in Go and really enjoyed the way the challenge broke this down into approachable steps, ultimately leading to a functioning http server.

Here is the challenge description

"Welcome to the Build your own HTTP server challenge!

HTTP is the protocol that powers the web. In this challenge, you'll build a HTTP server that's capable of handling simple GET/POST requests, serving files and handling multiple concurrent connections.

Along the way, we'll learn about TCP connections, HTTP headers, HTTP verbs, handling multiple connections and more."

Here are the following tasks:

1. Create a TCP server that listens on port 4221
2. Respond to an HTTP request with a 200 response
3. Extract the URL path from an HTTP request, and respond with either a 200 or 404, depending on the path
4. Implement the /echo/{str} endpoint, which accepts a string and returns it in the response body
5. Implement the /user-agent endpoint, which reads the User-Agent request header and returns it in the response body
6. Add support for concurrent connections
7. Implement the /files/{filename} endpoint, which returns a requested file to the client
8. Add support for the POST method of the /files/{filename} endpoint, which accepts text from the client and creates a new file with that text
9. Add support for compression to your HTTP server
10. Add support for Accept-Encoding headers that contain multiple compression schemes
11. Add support for gzip compression to your HTTP server

The challenge offers way more details than above and I encourage you to look through them beforehand. The walkthrough below is more of a summary of how to tackle each of these tasks.

Task 1 / 6

Using the go standard library, we can use the net package to Listen on the specified port. We can then Accept connections and handle them accordingly.

In the code below, we instantiate the listener and within a loop, we wait for and return the next connection to the listener via Accept. We then handle this connection in a separate go routine.

func main() {
	l, err := net.Listen("tcp", "0.0.0.0:4221")
	if err != nil {
		fmt.Println("Failed to bind to port 4221")
		os.Exit(1)
	}
	defer l.Close()

	for {
		conn, err := l.Accept()
		if err != nil {
			fmt.Printf("Error accepting connection: %s\n", err.Error())
			continue
		}
		go handleConnection(conn)
	}
}

Task 2

We can now handle the connection. The code below is used for subsequent tasks, but for this specific task, the following code is necessary:

1. We are using the bufio package from the standard library to instantiate a reader and writer to read from the connection and write to the connection
2. StatusOK = "HTTP/1.1 200 OK\r\n\r\n" is the correct status reponses format
3. writer.WriteString(StatusOK) will write this response to the connection

const (
	StatusOK                  = "HTTP/1.1 200 OK\r\n\r\n"
	StatusCreated             = "HTTP/1.1 201 Created\r\n\r\n"
	StatusNotFound            = "HTTP/1.1 404 Not Found\r\n\r\n"
	StatusBadRequest          = "HTTP/1.1 400 Bad Request\r\n\r\n"
	StatusInternalServerError = "HTTP/1.1 500 Internal Server Error\r\n\r\n"
)

// handleConnection handles the incoming connection
func handleConnection(conn net.Conn) {
	defer conn.Close()

	reader := bufio.NewReader(conn)
	writer := bufio.NewWriter(conn)

	lines, request, path, err := readRequest(reader)
	if err != nil {
		fmt.Printf("Error reading request: %s\n", err.Error())
		writer.WriteString(StatusBadRequest)
		writer.Flush()
		return
	}

	switch {
	case path == "":
		writer.WriteString(StatusOK)
	case path == "user-agent":
		handleUserAgentRequest(writer, lines)
	case strings.HasPrefix(path, "echo/"):
		handleEchoRequest(writer, lines, path)
	case strings.HasPrefix(path, "files/"):
		handleFileRequest(reader, writer, request[0], lines, path)
	default:
		writer.WriteString(StatusNotFound)
	}

	writer.Flush()
}

Task 3

To extact the URL path, we can read from bufio.Reader. The code below demonstrates how to read line by line until the end denoted via io.EOF. We leverage the strings package from the standard library to exctract relavent details such as the entire request, url path, and body. We can then use the following from the code above:

1. StatusNotFound = "HTTP/1.1 404 Not Found\r\n\r\n"
2. writer.WriteString(StatusNotFound)

// readRequest reads the HTTP request from the client
func readRequest(reader *bufio.Reader) ([]string, []string, string, error) {
	var lines []string
	for {
		line, err := reader.ReadString('\n')
		if err != nil {
			if err != io.EOF {
				return nil, nil, "", err
			}
			break
		}

		line = strings.TrimSuffix(line, "\r\n")
		lines = append(lines, line)

		// If the line is empty, we have reached the end of the HTTP request header
		if line == "" {
			break
		}
	}

	if len(lines) == 0 {
		return nil, nil, "", errors.New("empty request")
	}

	request := strings.Split(lines[0], " ")
	if len(request) == 0 {
		return nil, nil, "", errors.New("invalid request")
	}

	path := strings.Trim(request[1], "/")

	return lines, request, path, nil
}

Task 4 / 10 / 11

In this task, we implement the /echo/{str} endpoint.

1. The function below pulls the encoding type from the request along with the content encoding
2. We also leverage the gzip package from the standard library to write the compressed data

// handleEchoRequest will handle requests for echo
func handleEchoRequest(writer *bufio.Writer, lines []string, path string) {
	acceptEncoding := ""
	for _, line := range lines {
		if strings.HasPrefix(line, "Accept-Encoding: ") {
			acceptEncoding = strings.TrimPrefix(line, "Accept-Encoding: ")
			break
		}
	}

	contentEncodingHeader := ""
	for _, encoding := range strings.Split(acceptEncoding, " ") {
		encoding = strings.TrimSuffix(encoding, ",")
		if encoding == "gzip" {
			contentEncodingHeader = "Content-Encoding: gzip\r\n"
		}
	}

	word := strings.TrimPrefix(path, "echo/")

	var body bytes.Buffer
	if contentEncodingHeader != "" {
		zw := gzip.NewWriter(&body)
		zw.Write([]byte(word))
		zw.Close()
	} else {
		body.Write([]byte(word))
	}

	res := fmt.Sprintf("HTTP/1.1 200 OK\r\n%sContent-Type: text/plain\r\nContent-Length: %d\r\n\r\n", contentEncodingHeader, body.Len())

	writer.WriteString(res)
	writer.Write(body.Bytes())
}

Task 5

In this task, we implement the /user-agent endpoint.

1. We can extract the user agent from the request and print this to the writer.

// handleUserAgentRequest will handle requests for user-agent
func handleUserAgentRequest(writer *bufio.Writer, lines []string) {
	userAgent := ""
	for _, line := range lines {
		if strings.HasPrefix(line, "User-Agent: ") {
			userAgent = strings.TrimPrefix(line, "User-Agent: ")
			break
		}
	}

	res := fmt.Sprintf("HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: %d\r\n\r\n%s", len(userAgent), userAgent)
	writer.WriteString(res)
}

Task 7 / 8 / 9

This function is a bit more beefy as it contains the logic to handle GET and POST requests.

To break this down:

1. We first retrieve the directory path via the passed in flag to put together the full filePath.
2. For GET, we open the file, read the file, and write to buffer. (Make sure you flush the buffer at the very end!)
3. For POST, we read the body and write to the file depending on the content length

// handleFileRequest will handle requests for files
func handleFileRequest(reader *bufio.Reader, writer *bufio.Writer, method string, lines []string, path string) {
	if len(os.Args) != 3 || os.Args[1] != "--directory" {
		fmt.Println("Flag --directory <directory> is required")
		os.Exit(1)
	}

	directory := os.Args[2]
	_, err := os.Stat(directory)
	if os.IsNotExist(err) {
		fmt.Println("Directory does not exist")
		os.Exit(1)
	}

	filePath := fmt.Sprintf("%s%s", directory, strings.TrimPrefix(path, "files/"))

	switch method {
	case "GET":
		file, err := os.Open(filePath)
		if err != nil {
			writer.WriteString(StatusNotFound)
			return
		}
		defer file.Close()

		fileInfo, err := file.Stat()
		if err != nil {
			writer.WriteString(StatusInternalServerError)
			return
		}

		res := fmt.Sprintf("HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\nContent-Length: %d\r\n\r\n", fileInfo.Size())
		writer.WriteString(res)

		buffer := make([]byte, 4096)
		for {
			n, err := file.Read(buffer)
			if err != nil {
				break
			}
			writer.Write(buffer[:n])
		}
	case "POST":
		file, err := os.Create(filePath)
		if err != nil {
			writer.WriteString(StatusInternalServerError)
			return
		}
		defer file.Close()

		contentLengthHeader := ""
		for _, line := range lines {
			if strings.HasPrefix(line, "Content-Length: ") {
				contentLengthHeader = strings.TrimPrefix(line, "Content-Length: ")
				contentLengthHeader = strings.TrimSpace(contentLengthHeader)
				break
			}
		}

		if contentLengthHeader == "" {
			writer.WriteString(StatusBadRequest)
			return
		}

		contentLength, err := strconv.Atoi(contentLengthHeader)
		if err != nil {
			writer.WriteString(StatusBadRequest)
			return
		}

		if contentLength > 0 {
			buffer := make([]byte, 4096)
			remaining := contentLength
			for remaining > 0 {
				n, err := reader.Read(buffer)
				if err != nil && err != io.EOF {
					writer.WriteString(StatusInternalServerError)
					return
				}
				if n == 0 {
					break
				}

				if _, err := file.Write(buffer[:n]); err != nil {
					writer.WriteString(StatusInternalServerError)
					return
				}

				remaining -= n
			}
		}

		writer.WriteString(StatusCreated)
	}
}

---

Overall, this was a thought provoking exercise! I enjoyed dabbling into the weeds of Go's standard library which is very robust with functionality. The breakdown of the challenge into approachable steps also removed the overwhelming nature that I sometimes feel when starting a complex project.

I highly recommend you checking out this platform and giving it a try!

Thanks for reading! Happy coding!