Neural Network Architectures: Taxonomy of Design Principles, Evolution, and Emerging Frontiers

Abstract

Neural network architectures have undergone a remarkable transformation over the past seven decades, evolving from the single-layer perceptron to billion-parameter foundation models defining the state of the art across vision, language, and multimodal reasoning. The proliferation of architectures, however, has outpaced theoretical understanding of underlying performance mechanisms, resulting in design decisions driven largely by empirical benchmarking rather than principled reasoning. This review presents a unified taxonomy of neural network architectures organized around four core design principles: locality, memory, attention, and hierarchy, supplemented by two secondary efficiency principles: parameter sharing and sparsity. The four core principles provide a stable explanatory framework for characterizing representational capabilities, whereas parameter sharing and sparsity function as cross-cutting efficiency mechanisms that regulate computational cost. Nine major architectural families are systematically examined: multilayer perceptrons, convolutional neural networks, recurrent networks and gated variants, autoencoders, generative adversarial networks, transformers, graph neural networks, diffusion models, and state space models. Analysis focuses on the mechanisms through which each architectural family instantiates the core principles, the resulting inductive biases, and associated computational and representational trade-offs. A systematic literature review synthesizes comparative evidence concerning parameter complexity, data efficiency, scalability, and representative benchmark performance. Cross-cutting analysis further examines trade-offs involving inductive biases, scaling laws, emergent behaviors, hybrid and modular architectures, computational efficiency, interpretability, and robustness. Emerging research frontiers, including state space models, neural architecture search, biologically inspired computing, neuro-symbolic integration, and generalist foundation models, are subsequently reviewed alongside open problems in theoretical understanding, architecture design, evaluation methodology, and sustainability. The analysis identifies four core design principles as persistent architectural invariants across successive eras of neural network development, while parameter sharing and sparsity operate primarily as efficiency-oriented modifiers. A principle-centered organization of neural network architectures provides a conceptual framework for understanding architectural evolution, a decision guide for architecture selection, and a research agenda supporting principle-driven rather than benchmark-driven design.