{"id":105,"date":"2026-09-24T15:11:05","date_gmt":"2026-09-24T15:11:05","guid":{"rendered":"https:\/\/evoxia.online\/?p=105"},"modified":"2026-09-24T15:11:05","modified_gmt":"2026-09-24T15:11:05","slug":"neural-architecture-search-letting-algorithms-design-neural-networks","status":"publish","type":"post","link":"https:\/\/evoxia.online\/?p=105","title":{"rendered":"Neural Architecture Search: Letting Algorithms Design Neural Networks"},"content":{"rendered":"\n

Architecture Design by Hand<\/strong><\/p>\n\n\n\n

Every famous neural network was designed by people. AlexNet, VGG, ResNet and the Transformer came from researchers who tried ideas, trained models, read the results and tried again. How many layers, what kind of layers, how wide each one should be, where to put skip connections, which activation function to use: each decision was the result of intuition and a lot of trial and error.<\/p>\n\n\n\n

Neural architecture search (NAS) tries to automate that process. It treats the architecture itself as something to optimize: define a space of possible networks, search it with an algorithm, and evaluate candidates by training them and measuring their performance. Any NAS method can be described by three choices. The search space defines which architectures are possible. The search strategy decides which ones to try. The performance estimation strategy decides how to judge a candidate without spending too much compute on it. Thomas Elsken, Jan Hendrik Metzen and Frank Hutter used this framing in their 2019 survey, and it is still the standard way to compare methods.<\/p>\n\n\n\n

The Reinforcement Learning Approach<\/strong><\/p>\n\n\n\n

The paper that started the modern field was \u00abNeural Architecture Search with Reinforcement Learning\u00bb by Barret Zoph and Quoc Le at Google, presented at ICLR 2017. A recurrent neural network acted as a controller that generated a description of an architecture one decision at a time: filter size, stride, number of filters for each layer. That architecture was trained, its validation accuracy was used as a reward, and the controller was updated with a policy gradient method to produce better architectures over time. The networks it found were competitive with the best hand-designed ones on CIFAR-10, and the recurrent cell it found for language modeling beat the LSTM on Penn Treebank.<\/p>\n\n\n\n

The cost was enormous, because every sample meant training a network from scratch. The authors used hundreds of GPUs in parallel. The follow-up, NASNet (Zoph, Vasudevan, Shlens and Le, 2018), made the search more practical by searching for a small building block, a \u00abcell,\u00bb instead of a whole network. The best cell was then stacked many times to build the full model, and a cell found on the small CIFAR-10 dataset transferred to ImageNet with state-of-the-art results. Even so, the search took on the order of 2,000 GPU days.<\/p>\n\n\n\n

Evolutionary Approaches<\/strong><\/p>\n\n\n\n

Evolutionary algorithms were used on neural networks long before deep learning. NEAT, by Kenneth Stanley and Risto Miikkulainen in 2002, evolved both the weights and the topology of small networks. For modern deep networks, evolution usually searches only the architecture and leaves the weights to gradient descent.<\/p>\n\n\n\n

Esteban Real and colleagues at Google published AmoebaNet in 2019, using what they called regularized evolution, also known as aging evolution. It is a tournament-selection algorithm with one change: when a new architecture is added to the population, the oldest one is removed, not the worst. That forces the population to keep producing good architectures instead of relying on a single lucky one that happened to get a high score from a noisy evaluation. In a direct comparison with the reinforcement learning approach on the same search space, evolution reached similar or better accuracy and got there faster. The cost was still around 3,150 GPU days.<\/p>\n\n\n\n

Evolution has a natural advantage when there are several goals. NSGA-Net (Lu and colleagues, 2019) used NSGA-II to search for architectures that trade off accuracy against computational cost, returning a whole set of options instead of one network.<\/p>\n\n\n\n

Weight Sharing and One-Shot Models<\/strong><\/p>\n\n\n\n

Thousands of GPU days put NAS out of reach for almost everyone outside a few large companies. The idea that changed this was weight sharing. ENAS (Pham and colleagues, 2018) observed that all architectures in a cell-based search space are subgraphs of one large graph containing every possible operation. If that \u00absupernet\u00bb is trained once, each candidate architecture can inherit its weights from the supernet instead of being trained from scratch. The paper reported a search cost about 1,000 times lower than the original NAS.<\/p>\n\n\n\n

DARTS (Differentiable Architecture Search), from Hanxiao Liu, Karen Simonyan and Yiming Yang at ICLR 2019, went a step further. Instead of choosing one operation for each connection in the cell, DARTS places all candidate operations there at once and combines their outputs with a softmax over learnable weights. The architecture becomes continuous, so it can be optimized with gradient descent alongside the network weights. After the search, the operation with the largest weight on each edge is kept. DARTS found cells comparable to NASNet and AmoebaNet in 1.5 to 4 GPU days.<\/p>\n\n\n\n

The speed came with problems. Later studies found that DARTS often collapses toward architectures full of skip connections, which are easy to train in the supernet but perform poorly when trained alone. Supernet performance also did not always predict how well an architecture would do when trained on its own. Many variants since then have focused on fixing this instability.<\/p>\n\n\n\n

Hardware-Aware Search<\/strong><\/p>\n\n\n\n

The best architecture depends on where it will run. A network for a phone has to meet a latency budget, and the number of floating-point operations is a poor proxy for real speed because it ignores memory access and hardware-specific kernels. MnasNet (Tan and colleagues, 2019) included latency measured on actual phones in the search objective. EfficientNet (Tan and Le, 2019) used a similar search to find a small baseline network, then scaled its depth, width and input resolution together with a fixed ratio, producing a family of models that was more accurate than previous networks at every size.<\/p>\n\n\n\n

Once-for-All (Cai and colleagues, 2020) trained a single large network in a way that makes many of its subnetworks usable directly, so a specialized model for a new device can be extracted without retraining. This moves most of the cost into one training run and makes deployment on many devices cheap.<\/p>\n\n\n\n

Is Search Better Than Random?<\/strong><\/p>\n\n\n\n

As results piled up, a skeptical question appeared: how much of the performance comes from the search algorithm and how much from a well-designed search space? Liam Li and Ameet Talwalkar showed in 2019 that random search with weight sharing was competitive with DARTS and ENAS on standard benchmarks. Yang and colleagues reported in 2020 that on several popular search spaces, randomly sampled architectures already performed very well, and that details of the training pipeline mattered more than the search method.<\/p>\n\n\n\n

To make comparisons fair and cheap, researchers built tabular benchmarks. NAS-Bench-101 (Ying and colleagues, 2019) trained all of about 423,000 unique architectures in a fixed cell space and published their results, so a search algorithm can be tested by looking up accuracies instead of training anything. NAS-Bench-201 and others followed. These benchmarks made it possible to compare algorithms reproducibly, and they also made it clearer that many published gains were small.<\/p>\n\n\n\n

The Cost Question<\/strong><\/p>\n\n\n\n

NAS also became an example in debates about the environmental cost of AI. In 2019, Emma Strubell, Ananya Ganesh and Andrew McCallum estimated the carbon emissions of training NLP models, and the scenario with the largest footprint was a Transformer found through NAS, estimated at roughly 626,000 pounds of CO2. The figure was widely quoted. In 2021, David Patterson and colleagues at Google argued that it was far too high for the actual search that had been run, because it assumed the full search was performed at the largest model size on older hardware. The disagreement is a useful reminder that estimates of AI’s energy use depend heavily on assumptions about hardware, data centers and exactly what was run.<\/p>\n\n\n\n

Where NAS Stands Now<\/strong><\/p>\n\n\n\n

NAS produced real, widely deployed models, especially for mobile vision, and weight sharing and hardware-aware objectives are now standard tools. At the same time, the biggest architectural shift of recent years, the move to Transformers and their scaling, came from human design and experiments with scale, not from search. For large models, the practical questions are usually about scaling width, depth and data, and search is applied to smaller choices like the shape of attention blocks, activation functions or the configuration of mixture-of-experts layers. In industry, NAS is most common when a model has to fit a specific device under strict latency, memory or energy constraints.<\/p>\n\n\n\n

Reflection<\/strong><\/p>\n\n\n\n

NAS started as an effort to have algorithms design networks without human input, and it ended up showing how much the human choices still matter. The search space is designed by people, and it often decides most of the result before the search even begins. The strongest work in the field is honest about that. It treats NAS as a way to search carefully through a space that experts defined, not as a replacement for the experts.<\/p>\n\n\n\n

By: Max Johnson B.<\/p>\n","protected":false},"excerpt":{"rendered":"

Architecture Design by Hand Every famous neural network was designed by people. AlexNet, VGG, ResNet and the Transformer came from researchers who tried ideas, trained models, read the results and tried again. How many layers, what kind of layers, how wide each one should be, where to put skip connections, which activation function to use: […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-105","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/105","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=105"}],"version-history":[{"count":0,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/105\/revisions"}],"wp:attachment":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}