Sionna Bench: Towards Benchmarks in Wireless Communications

Overview & Synopsis

Academic research in wireless communications has no tradition of benchmarks so far. Remarkable progress in benchmark-oriented fields like machine learning suggests that the adoption of benchmarks could benefit our field. In the absence of benchmarks, however, a scientific debt has accumulated: since the academic community has not canonized specific problem setups, methods that address essentially the same problem were often developed for slightly different setups and lack a common framework for comparison. Any attempt to establish a new benchmark is faced with the challenge of resolving this debt. Recent advances in agentic artificial intelligence (AI) create an opportunity to do so by leveraging AI-assisted coding. This makes it possible to implement the state-of-the art at scale and to adapt it to a common framework for comparison. We present Sionna Bench to initiate benchmarks in academic wireless communications research. As a pilot study, we use the problem of MIMO-OFDM channel estimation. At the time of writing, Sionna Bench's channel estimation benchmark features twenty channel estimation methods that were implemented from ten different papers. The ability to compare methods against the state-of-the-art will be crucial for navigating the research landscape in an era where AI-generated methods coexist with human-developed algorithms.