I tried to build my own game-playing AI from scratch

Building your own AI can feel impossible when you’re just starting out. But with a few core ideas, some patience, and a lot of trial and error, you can actually create a model that learns to play a video game on its own.

This guide walks through the journey of going from zero machine learning knowledge to training a neural network to play a classic platformer level. Along the way, you’ll see how evolution-inspired algorithms, neural networks, and game memory all come together to create a working game-playing AI.

From random guessing to evolution-inspired learning

Imagine you want a program to guess a hidden phrase that’s 18 characters long. You could brute-force it by randomly guessing strings until you hit the exact phrase, but with 27 possible characters for each of the 18 positions, that’s 27¹⁸ combinations—far too many to try one by one.

A smarter approach is to mimic natural selection. Instead of pure randomness, you evolve a population of candidate strings over time:

1. Create a population of random strings of length 18.

2. Score each string by how many characters are correct and in the right position.

3. Select the fittest strings (the ones with the highest scores) and give them a higher chance to reproduce.

4. Breed new strings by combining parts of two parent strings into a child string.

5. Add mutation by randomly changing a small percentage of characters to keep the population exploring new possibilities.

Repeat this for hundreds of generations and the population slowly converges toward the correct phrase. You don’t tell the program the answer—you just reward better guesses and let selection do the rest. This process is known as a genetic algorithm or evolutionary algorithm.

Applying evolution to moving objects

The same idea can be applied to more visual problems, like guiding rockets toward a target on the screen. Each rocket has a set of parameters (its “genes”) that control how it moves. You let a whole population of rockets try to reach a target, score them by how close they get, then breed the best performers to create the next generation.

Over many generations, the rockets “learn” to navigate toward the target more efficiently, even though they never explicitly know the rules of the world. They just get rewarded for good behavior and punished for bad behavior.

This evolution system becomes the foundation for training more complex agents later—like a character in a platformer game.

Neural networks in plain language

Evolution tells you how to search for better solutions, but you still need a flexible model that can map inputs (like what’s happening in the game) to outputs (like which buttons to press). That’s where neural networks come in.

Biologically, your brain is made up of billions of neurons that send signals to each other. A single neuron doesn’t know much, but together they can learn complex behaviors like walking, talking, or playing games.

A machine learning neural network borrows this idea, but replaces biology with math. At its core, a single artificial neuron does something very simple:

1. It takes several input values.

2. Each input is multiplied by a weight (a number the network can adjust).

3. All these weighted inputs are added together, plus a bias term.

4. The result is passed through an activation function (a simple formula that adds non-linearity) to produce an output.

Even this tiny neuron can learn a simple pattern, like guessing whether a point on a graph lies above or below a hidden line. Every time it guesses wrong, you nudge the weights and bias in the direction that would have made the guess less wrong. After enough examples, it effectively learns where the line is.

Scaling up: from one neuron to full networks

To handle more complex tasks, you stack many neurons into layers:

• An input layer that receives raw data (for example, the state of a game screen).

• One or more hidden layers that transform and combine information.

• An output layer that produces a set of scores or probabilities (for example, which game button to press).

Each neuron in one layer connects to each neuron in the next layer via weights. Instead of updating each connection one by one, you can store the weights in matrices and use matrix multiplication to compute an entire layer at once. This is where linear algebra comes in, but conceptually it’s just a faster way to do a lot of multiplications and additions.

With this structure, a neural network can take in a snapshot of a game state and output a decision—like “press right and jump.”

Connecting AI to a real game

To make a game-playing AI, you need a way for your code to both see the game and control it. For classic games, that usually means working with an emulator.

In this project, a Python library was used that wraps a real NES emulator and exposes it as an environment. The code can:

• Read the current game state.

• Send button inputs (like A, B, left, right, jump).

• Step the game forward frame by frame.

At first, the environment was tested by sending random inputs every frame, just to confirm that the wiring worked. Then multiple game instances were run in parallel, each with its own stream of random inputs. This parallelization is important later when training many agents at once.

Teaching the AI to “see” the level

To learn to play rather than just memorize button sequences, the AI needs a structured view of the world around the character. Basic info like position and coin count is not enough—it needs to know where blocks, pipes, enemies, and gaps are.

That required digging into the game’s RAM maps and reverse engineering how the game stores its level data in memory. Using public RAM documentation, a similar open-source project, and a lot of trial and error, the raw memory was translated into a clean grid representation.

The final result was a 2D grid of numbers:

• 1 for solid tiles like ground, pipes, or blocks.

• -1 for enemies.

• 2 for the player character.

Flattened out, this grid produced 208 input values for the neural network. This step was one of the most tedious parts of the entire project, but it’s also where a lot of the real-world learning happens: understanding how games store state and how to turn that into something an AI can use.

Designing the Mario-playing neural network

With the environment grid ready, the next step was to design the neural network architecture that would control the character.

The basic setup looked like this:

• Input layer: 208 neurons, one for each cell in the environment grid.

• Hidden layer: starting with 18 neurons (this number was later tweaked multiple times).

• Output layer: 12 neurons, one for each possible controller input recognized by the game library.

On each frame:

1. The current environment grid is fed into the network.

2. The network processes it through the hidden layer.

3. The output layer produces 12 scores, one per possible action.

4. The action with the highest score is chosen and sent to the game.

Each game instance gets its own neural network with its own set of random weights at the start. Even with no training, this created some fun emergent behaviors: some characters ran back and forth, some jumped constantly, some held right, and many did nothing at all. It was still random, but it felt more like distinct “personalities” than pure noise.

Using evolution to train the network

Instead of using gradient-based training methods like backpropagation, this project leaned on the earlier evolutionary idea: treat each neural network as an individual in a population and evolve the weights over time.

To do that, you need a fitness function—a way to score how well each network performs. The fitness function here considered:

• Distance traveled: how far the character gets in the level.

• Completion bonus: a big reward for actually beating the level.

• Speed: extra points for finishing quickly.

Once each network’s run was scored, the best performers were added to a breeding pool. Their weights were then mixed with others to create child networks for the next generation, with a bit of random mutation added to keep the search space wide.

This is essentially the same as the earlier string-guessing example—but now the “genes” are the neural network’s weights, and the “environment” is a platformer level.

Debugging, bottlenecks, and failed runs

The first 24-hour training run was a disappointment: the population didn’t learn anything meaningful. That’s a common experience when you’re building your first AI system from scratch.

To fix it, several aspects had to be tuned and reworked:

• Fitness function tweaks: adjusting how distance, time, and completion were weighted.

• Environment conditions: deciding when to reset runs, how long to let agents try before timing out, and how to handle deaths.

• Network architecture: increasing the size of the hidden layer and experimenting with different configurations.

Another major bottleneck was processing power. Evolutionary training works best when you can run many agents in parallel. Real-world evolution has billions of organisms running at once; on a home machine, you’re limited by how many emulators and networks you can simulate simultaneously.

After many iterations and about a week of trial and error, the training finally produced a population that could play the level in a surprisingly competent way.

Putting the AI through a level gauntlet

Once the AI could reliably beat a specific level, the next test was generalization: could the same trained network handle different levels it hadn’t seen during training?

The agent was dropped into various levels from different parts of the game. In some cases, it showed very tight movement—dodging enemies, navigating staircases, and timing jumps well. In other cases, it struggled with unfamiliar structures or new mechanics, like springs it had never encountered before.

Because the simulation is deterministic and the network weights are fixed during evaluation, running the same test multiple times produced identical behavior. That makes it easier to debug and analyze, but it also means that if the AI gets stuck in a bad behavior pattern in a new situation, it will repeat that mistake every time.

What this kind of project really teaches you

The most valuable outcome of building a game-playing AI from scratch isn’t just watching a character clear a level—it’s the experience you gain along the way:

• Neural network fundamentals: understanding inputs, weights, biases, activations, and architectures in a hands-on way.

• Evolutionary methods: seeing how selection, crossover, and mutation can search huge solution spaces without explicit instructions.

• Reverse engineering: learning how to interpret RAM maps, extract game state, and transform it into a usable representation.

• Debugging complex systems: tuning fitness functions, adjusting environments, and iterating on architectures when things don’t work.

These skills transfer to a huge range of AI projects, from robotics and simulations to more practical automation tasks. If you enjoy this kind of hands-on experimentation, you might also like exploring projects that use AI to automate work or generate income, such as those covered in this breakdown of AI tools that can actually make you money or more ambitious experiments like trying to earn with AI agents in a fixed time window.

Final thoughts

Building a game-playing AI from scratch is a demanding but incredibly fun way to learn modern AI concepts. You start with simple math and basic evolution rules, and end up with a system that can navigate a world, avoid enemies, and complete goals you never explicitly programmed in.

If you’re just getting into AI, tackling a project like this forces you to touch almost every part of the stack: algorithms, neural networks, tooling, performance, and debugging. And even if your first model isn’t perfect, the experience you gain is easily a 10 out of 10 in terms of learning value.