Learning RL from Scratch: A Practical Notes Series and Open-Source Implementation

Why Learning RL from Scratch Is Worth Documenting

Reinforcement learning (RL) is often regarded as one of the hardest fields in machine learning to get started with—abstract concepts, a high mathematical barrier, and difficult debugging. A developer shared their "learning RL from scratch" journey on Reddit and decided to organize the learning process into a series of short notes published on their personal website. The first installment, "1-Page RL Basics," is already live, covering the most important terminology for beginners and interspersing mathematical details the author found interesting.

This "learn while writing" approach both tests one's own understanding and provides a reusable roadmap for those who come after. The author mentions that subsequent notes will gradually cover content from fundamentals to the latest advances, accompanied by from-scratch implementations and experiments.

What's in the Notes: Foundational Terminology and Mathematical Intuition

According to the author, the defining feature of this series is that it is "very short"—each note focuses on a single topic and works well as review material or an introductory read for newcomers. The first note condenses the core terminology of RL onto a single page and adds mathematical derivations the author found interesting along the way.

For beginners, RL's terminology system (states, actions, rewards, policies, value functions, etc.) is often easy to confuse. A high-density one-page summary helps quickly build a conceptual map. The author didn't disclose exactly which terms are covered, but given the "foundational and important" positioning, it likely includes classic content such as Markov decision processes, the Bellman equation, and exploration versus exploitation.

A Markov decision process (MDP) is the standard mathematical framework for RL problems, defined by the five-tuple $(S, A, P, R, \gamma)$: the state set $S$, action set $A$, state transition probability $P(s'|s,a)$, reward function $R(s,a)$, and discount factor $\gamma \in [0,1]$. Its core is the Markov property—the next state depends only on the current state and action, independent of history. The Bellman equation is the recursive solution form of an MDP, decomposing the value function into the sum of immediate reward and discounted future value, and it is the theoretical foundation for value iteration, policy iteration, and algorithms like Q-learning. Understanding these two concepts is essential for reading the derivations of nearly all subsequent RL algorithms.

Exploration versus exploitation is the core challenge that distinguishes RL from supervised learning: the agent must trade off between "trying new actions to gain information" and "exploiting known optimal actions to obtain reward." Common strategies include $\epsilon$-greedy (random exploration with probability $\epsilon$), Boltzmann exploration (sampling according to the value distribution), and optimism-based exploration under uncertainty (such as UCB). This trade-off directly determines whether an algorithm can converge to the global optimum and is also the main subject of study in the multi-armed bandit problem.

From Notes to Implementation: Future Plans

The author's plans go beyond theoretical notes. He says that in the "coming days" he will publish from-scratch implementations and experiments, covering content from fundamentals to more recent methods. This means readers will get not only conceptual explanations but also runnable code and experimental observations.

This combination of "notes + implementation" is especially important in RL learning. The stochasticity and hyperparameter sensitivity of RL algorithms often make purely theoretical understanding insufficient; hands-on reproduction is key to building intuition. By publicly sharing the implementation process, the author also provides an entry point for the community to discuss and improve.

The sources of stochasticity in RL algorithms are diverse: randomness in environment transitions, randomness in policy sampling, and randomness in neural network initialization. This causes significant performance differences for the same algorithm under different random seeds, making it hard during debugging to tell whether a failure stems from an implementation error or from variance. In practice, methods such as fixing random seeds, averaging over multiple seeds, and smoothing learning curves are commonly used to reduce noise. In addition, RL is extremely sensitive to hyperparameters (learning rate, discount factor, batch size, target network update frequency, etc.), and purely theoretical understanding is often insufficient to predict the direction of tuning; hands-on reproduction and systematic experiment logging are necessary steps for building intuition.

Community Feedback and Learning Approach

The author actively invites feedback in the post and hopes to hear readers' thoughts. This openness reflects a characteristic of the RL learning community: knowledge density is high and updates are fast, so going it alone easily leads to blind spots, while public sharing can bring corrections and additions.

The path to learning RL from scratch varies from person to person, but "write it down, explain it clearly, and get the code running" is a widely recognized effective method. If this developer's notes series continues to be updated, it has the potential to become a lightweight but practical learning resource.

Takeaways for Beginners

If you are also learning RL, you might consider adopting this approach: after learning each concept, write it up in the shortest possible space and try to verify it with code. There's no need to pursue an all-encompassing tutorial—short, iterable notes are easier to sustain and easier to get feedback on. We look forward to the author's upcoming from-scratch implementations, and to more learners sharing their own roadmaps.