Disk Position RL | Sujjay Karthikeyan

The Problem.

Before building a full ROS2/Gazebo simulation, the core coverage problem needed to be validated in 2D: can a reinforcement learning agent learn to position 3 agents (disks) to collectively observe all 25 target points on a 5×5 grid? This is the disk-position repository — the prototype that proved the concept before the SmartNet Lab work began in earnest.

Algorithm

Tabular Q-Learning

Abstraction

2D Grid World

Agents

3 Disks

Targets

25 Points (5×5)

Stack

Python / NumPy

The Environment.

2D Coverage Abstraction — 5×5 Target Grid

Agent position (disk, r=1.2 coverage)

Target point — within coverage

Target point — uncovered

Reward is proportional to unique targets covered. Agents learn to spread without overlapping — maximizing joint coverage.

Implementation.

State Space Encoding Each state encodes all 3 agent positions on the 5×5 grid. Actions are discrete moves in 4 cardinal directions per agent, with boundary clipping. The joint state space is 25³ = 15,625 states — small enough for tabular Q-learning.

Coverage Reward Function Reward = number of unique target points within coverage radius r=1.2 of any agent. Agents are penalized for overlap — two disks covering the same target earn less than two disks covering separate ones.

Epsilon-Greedy Exploration Standard epsilon-greedy with linear decay. The agent transitions from exploration to exploitation as the Q-table fills in. Convergence typically within ~10,000 episodes for the 5×5 world.

Tabular → DQN Transition Once tabular Q-learning validated the reward structure and action space, the same problem was transferred to a PyTorch DQN — the version that runs in Gazebo. The 2D prototype confirmed the problem was learnable before investing in 3D simulation infrastructure.

Lineage.

disk-position — This Repo

2D tabular Q-learning prototype. 3 agents, 25 targets, 5×5 grid. Validates reward design and convergence before scaling.

PyTorch DQN

Same coverage problem, same reward function — but a 4-layer FC network (4 → 64 → 64 → 64 → 25) replaces the Q-table. Experience replay + target network added. Scales to larger state spaces.

DiscWorld / DroneSimulation

Gazebo SDF world generator that places 25 cylindrical targets in a 100×100m field — the 3D translation of the 2D grid. DQN coverage policy transferred into simulation.

ROS2-Drone-Simulation

Full multi-UAV coordination in Gazebo Harmonic under ROS2 Jazzy. MRS UAV System integration. The complete research system the prototype was always pointing toward.

The Prototype That Mattered.

This is the repository that made the SmartNet Lab research possible. Before committing to ROS2, Gazebo, and the MRS UAV System, the disk-position prototype proved that the core RL formulation was correct. Every design decision in the full 3D system — reward shaping, action space, exploration schedule — was validated here first, in 50 lines of NumPy.

View on GitHub →