Learning coordinated behavior and cooperation among agents in combat simulations—where multiple autonomous agents operate as a team—poses a significant challenge. This complexity arises from factors such as partial observability, inter-agent communication, the non-stationary nature of multi-agent environments, a vast state space, and the difficulty of attributing rewards to individual agents. In this work, we developed a custom simulation environment in which two teams of three agents compete on maps containing obstacles. We train the agents using the MAPPO algorithm, following the paradigm of centralized training and decentralized execution. Through experiments, we evaluate the impact of various techniques—including memory mechanisms, exploration incentives, the composition of agent observations, extended training durations, environment representations, neural network architectures, and hierarchical agent control. The resulting policies are evaluated through direct head-to-head matches and contests against heuristic-based opponents across multiple maps. In addition to win rates, we tracked Elo ratings (a measure of relative competitive performance) as well as various combat and behavioral metrics. Finally, we extended the problem and the models to a 2.5-dimensional space and evaluated their performance there as well. The results of the project are several successful combat models, a simulation environment, and a 3D visualization of the combat sequences, which provide further insight into the rationale behind certain decisions made by the trained agents. The results show that the most successful policies combine a short-term memory of the history of observations, a memory map, and fine-tuning with a lower learning rate. The best policy in the multi-model tournament achieves an average score of 66.08 %.
|
