AI Winters and Comebacks

After understanding what AI is at a high level, it helps to see how it actually developed in the real world. The story is not a straight line of constant improvement. From the 1970s onward, AI moved through clear cycles: moments of excitement and big promises, followed by periods where progress slowed, investors pulled back, and researchers had to rethink their methods. These cooler periods came to be known as “AI winters.”

An AI winter describes a time when confidence, funding, and public interest in AI shrink for a while. Ambitious goals meet hard limits in hardware, data, or algorithms, and the gap between marketing and reality becomes too large to ignore. Research does not stop, and talent does not disappear. Instead, the field goes into a more disciplined phase: expectations are reset, weaker ideas fall away, and new approaches are tested quietly.

These cycles matter for today’s leaders, because they show what actually moves AI forward: evidence, scale, and practical results, rather than hype alone. In the next part of this section, we will look at the major AI winters and the comebacks that followed them, starting with the first major slowdown in the 1970s and 1980s and moving toward the deep learning breakthroughs that shaped the systems in use today.

Why winters occurred

• Computing power was limited. Early machines struggled with large search spaces, complex grammars, and dense rule sets. Memory was small, storage was costly, and floating-point performance was modest.
• Data was scarce or difficult to use. Large curated text collections, labeled images, and transcribed speech were rare. Shared benchmarks and open repositories were not yet common.
• Methods did not scale to real-world variation. Rule bases became brittle, single layer perceptrons had clear limits, and probabilistic tools were not yet standard practice.
• Funding followed results. Government reviews in the United States and the United Kingdom questioned near-term feasibility, which led to reduced support and narrower programs.

What persisted during winters

• Core formalisms for logic, search, grammar, and control survived and matured in textbooks and software.
• Tooling improved, including programming languages, parsers, and inference engines.
• New mathematical threads took root, such as probability for uncertain reasoning and optimization for learning.

Why momentum returned

• Data grew with the spread of the internet, mobile devices, and digitized archives. Email, web pages, images, audio, and transactions created vast training corpora.
• Compute accelerated through GPUs and distributed systems. Training times dropped from weeks to days, which made iterative research practical.
• Algorithms advanced. Backpropagation gained practical traction with better initialization and regularization. Support vector machines, probabilistic graphical models, and later deep neural networks delivered state-of-the-art results across tasks.
• Benchmarks and competitions focused effort. Shared datasets for vision, language, and speech allowed fair comparisons and rapid iteration.

What changed by the 2010s

• Vision systems recognized objects and faces at near human levels on standard benchmarks.
• Machine translation moved from phrase tables to neural models and improved markedly in fluency.
• Game playing agents surpassed strong human opponents in chess, Jeopardy, and Go.
• Speech recognition became robust enough for mainstream assistants on phones and in homes.

This section will trace those cycles in detail. You will see why the first winter formed in the 1970s, how a second contraction followed the expert systems boom in the late 1980s, and how the field returned to growth through data, compute, and new learning methods. The goal is not to dramatize setbacks. The goal is to understand how constraints shaped better science and how those lessons inform present practice. By looking closely at these quieter periods, we can see how pressure, limits, and recalibration strengthened the foundations of the field and continue to influence today’s best practices.

1.1.3.1 The Second Chance: Expert Systems (1980s)

Expert systems encoded specialist knowledge as thousands of “if…then” rules and used an inference engine to apply those rules to facts about a case. Most systems used forward chaining, where the system starts from known facts and pushes consequences forward, or backward chaining, where it starts from a goal and works backward to see which facts support it, and many included an explanation facility that listed the rules that fired.

How they worked in practice

Medicine

Finance

Manufacturing and Configuration (XCON)

Early Predictive Maintenance

Why investors returned, and how big the wave was

By 1985, corporations worldwide spent over one billion US dollars per year on AI initiatives, much of it for in-house expert system groups and commercial shells from companies such as Teknowledge and IntelliCorp. Specialized Lisp machines from Symbolics, LMI, and others were purchased to run these systems, and unit prices for high end configurations typically reached tens of thousands of US dollars per workstation. National programs also invested heavily. Japan’s Fifth Generation Computer Systems program spent about 57 billion yen across 1982 to 1994, roughly 320 million US dollars at the time.

The business case was straightforward. Internal case studies reported multi million dollar annual savings for configuration, scheduling, and support tasks, and the ability to codify scarce expertise was viewed as strategic.

Where the limits appeared

Knowledge acquisition bottleneck

The main problem was getting expert knowledge into the system in the first place.

To build an expert system, teams had to sit with top specialists for hours, ask hundreds of questions, turn their answers into rules, and then test those rules repeatedly. Tools such as TEIRESIAS and MOLE tried to make this easier, but large systems still needed long interview cycles, careful validation, and a lot of debugging. In practice, capturing and updating expert knowledge was slow, tiring, and expensive.

Brittleness and scale

As these systems grew from a few hundred rules to thousands, small changes started to cause big surprises. Rules interacted with each other in ways that designers did not always expect. A minor change in the business environment, a new product, or a new regulation often meant that many rules had to be rewritten. Many real world domains also needed a better way to handle uncertainty and “grey areas” than simple confidence scores. The result was that large expert systems became fragile and hard to maintain.

Hardware economics

Many early expert systems ran on special computers built for AI work, for example Lisp machines. These machines were powerful for their time but they were also expensive. At the same time, general purpose UNIX workstations and personal computers were getting faster and cheaper every year. Companies started to ask why they should keep paying a premium for specialist hardware when cheaper machines could handle most tasks. As a result, the market for those AI specific machines declined rapidly.

Expectations versus delivery

In small pilot projects, expert systems often looked impressive. They solved narrow problems well and produced good case studies. However, when organisations tried to roll them out across departments or entire businesses, new issues appeared. Integration with existing systems was difficult, updating rules took time, and the cost of ongoing maintenance climbed. In many cases, the long term cost of keeping the system accurate and up to date became higher than the value it produced, so investment slowed or stopped.

The Second AI Winter and its causes

By the late 1980s and early 1990s, a combination of rising maintenance costs, hardware platform shifts, and overstated projections led to cancellations and a sharp reduction in purchases. The downturn is widely referred to as the Second AI Winter. It reflected economic and technical realities. Organizations needed systems that could adapt as data and policy changed, and hand built rule bases were not sufficient.

What endured from this wave

• The knowledge base plus inference engine architecture and the practice of explanations influenced today’s decision support and model governance.
• Experience with brittleness and maintenance costs motivated the shift toward learning from data and toward probabilistic reasoning.
• Demonstrated savings, such as the tens of millions of US dollars per year reported by XCON at DEC, proved that applied AI could generate measurable business value when tightly scoped and well engineered.

In short, expert systems showed how to encode expertise and deliver value in narrow domains. They also revealed the limits of rule only approaches at scale, which set the stage for the later move to statistical learning, larger datasets, and modern machine learning.

1.1.3.2 The Internet and the Return of Data (1990s – 2000s)

The commercial internet connected people, institutions, and machines at global scale. Connectivity produced something the earlier decades never had in abundance: continuous, diverse, and machine-readable data.

How much data?

• Web content grew from a few thousand publicly reachable pages in the early 1990s to billions of indexable pages by the late 2000s. Public websites expanded from roughly 17 million in 2000 to about 65 million in 2005.
• Search engines crawled hundreds of billions of unique URLs by the end of the decade.
• Global email traffic reached about 10 billion messages per day by the end of 2000, with forecasts of 35 billion per day by 2005, providing massive supervised and unsupervised text for spam filtering, topic grouping, and language modeling.
• Consumer photography shifted from film to digital. Annual photo creation moved into the tens of billions of images per year by the late 2000s.
• Video moved online. User platforms hosted millions of uploads and many thousands of hours of new video per day by the end of the decade.
• Industry analysts estimated that annual digital data creation crossed the hundreds of exabytes and entered the zettabyte scale soon after 2010.

These figures are directional, but the lesson is clear. The internet created a persistent, labeled, and timestamped record of text, images, audio, clicks, and transactions at a scale that earlier AI researchers could not access.

Why this enabled a different kind of AI

Earlier programs tried to capture expertise by writing rules. The new environment rewarded systems that learn from examples. With enough labeled emails, a spam filter can infer patterns of unwanted messages. With enough captioned images, a vision model can learn to recognize objects. With enough parallel text, a translation system can learn correspondences between languages.

Three ingredients converged.

1. Data.
   The web, email, forums, digital media, and enterprise systems produced training corpora of unprecedented size and diversity. IDC estimated the “digital universe” at 281 exabytes in 2007. That volume equals 281 billion gigabytes.
2. Compute.
   Processors became cheaper and faster. Graphics processors became available for general computation, which accelerated linear algebra at the heart of modern learning. Distributed clusters allowed researchers to train models across many machines.
3. Algorithms.
   As data and computing power grew, a branch of AI called statistical learning came of age. Instead of writing thousands of rules by hand, engineers started using mathematical models that could learn patterns directly from data.

Some of the key methods were:

• Logistic regression

  A simple model that takes different factors as inputs, such as age, income, or transaction size, and estimates the probability of an outcome, for example “will this customer churn” or “is this transaction risky.”

• Support vector machines (SVMs)

  A method that looks at lots of examples and learns a clear boundary between groups, for example separating “spam” emails from “legitimate” emails by finding the best dividing line in the data.

• Probabilistic graphical models

  Models that describe how different variables are connected, for example how symptoms, diseases, and test results relate to one another, and then use those relationships to estimate which outcome is most likely.

• Deep neural networks

  Layered models inspired loosely by the brain that can spot complex patterns in images, speech, text, and other data, for example recognising faces in photos or understanding spoken commands.

When these methods were combined with large, labelled datasets, they started to outperform older rule based systems in many tasks, such as classification, prediction, and pattern recognition.

What changed in practice

• Filtering and ranking improved.
  Web search, spam detection, and recommendation systems began to learn directly from user behavior.
• Perception systems improved.
  Speech recognition error rates fell, and image classifiers reached levels that supported real products.
• Language systems improved.
  Data-driven translation moved beyond hand-built grammars and phrase tables toward neural models by the early 2010s.

The field advanced through measurable progress powered by data, computing capacity, and structured learning methods, rather than vague aspirations. What followed was a sustained period of renewed growth, with benchmarks that improved year over year and applications that reached mainstream use.

1.1.3.3 New Tools: Better Hardware & Smarter Math

By the early 2000s, three ingredients came together and changed the pace of progress: faster hardware, larger datasets, and better training methods.

GPUs: who led the change and how it worked

What a GPU is, in simple terms

A GPU is a chip that can do many small math problems at the same time. Training modern AI models is a giant pile of the same math repeated again and again. That makes GPUs a good fit.

Who pushed GPUs into AI

• NVIDIA’s CUDA team (from 2007 onward) made it possible for researchers to program GPUs for general math, not only for graphics. CUDA turned the GPU into a tool you could use from common programming languages.
• Research groups made the breakthrough credible.
  • Andrew Ng’s lab at Stanford (around 2009) showed that GPUs could speed up neural network training by big factors in early demonstrators.
  • Jürgen Schmidhuber’s group at IDSIA with Dan Cireșan (2010 to 2011) trained vision models on GPUs that set strong results on handwriting and traffic sign benchmarks.
  • Geoffrey Hinton’s team with Alex Krizhevsky and Ilya Sutskever (2012) trained AlexNet on two consumer GPUs and cut the error rate on the large ImageNet image challenge by a wide margin. That single result convinced many labs and companies to adopt GPU training.

How it works at a high level

You can think of training a modern AI model as teaching it through many rounds of trial and error:

1. Start with a large batch of data

   For example, you might use thousands or millions of pictures of cats, dogs, cars, and buildings, each correctly labelled.

2. Run simple calculations on each example

   Inside the model there are many numbers called weights. You can think of weights as “dials” that tell the model how strongly to pay attention to different features in the data.

   For each picture, the model:

   • Makes a guess (for example, “this is a cat”).

   • Compares its guess to the correct label.

   • Calculates how wrong it was and by how much.

   • Slightly turns many of those internal “dials” so that next time it will be less wrong.

     Those “turns” are the math: lots of small additions and multiplications that slowly adjust the weights.

3. Use GPUs to do this many times at once

   A GPU (graphics processing unit) is a type of chip designed to do the same simple calculation thousands of times in parallel.

   Instead of adjusting the model one picture at a time, a GPU lets you adjust it for hundreds or thousands of pictures at the same time. This parallel training:

   • Cuts training time from months to days or hours.
   • Makes it realistic to train very large models on very large datasets.

4. Repeat the cycle many times

   The model sees the same data again and again, each time:

   • Guessing,

   • Measuring the error,

   • Adjusting its internal weights.

     After many rounds (called epochs), the model becomes very good at recognizing patterns such as objects in images, words in sentences, or trends in financial data.

For now, the key idea is this: GPUs speed up the training loop by doing many small weight adjustments in parallel. That speed is what made large scale deep learning practical and opened the door to the modern generation of AI systems.

Big Data tools: storing and moving the fuel

As AI models grew more capable, they needed far more “fuel” in the form of data. The rise of the web, smartphones, and connected devices meant that companies were suddenly generating logs, clicks, messages, transactions, images, and sensor readings at a scale that older systems could not handle. Traditional databases were designed for millions of records, not trillions, and single servers quickly became a bottleneck for both storage and analysis.

To keep up, a new generation of Big Data tools emerged. These tools focused on two goals: spreading work across many ordinary machines instead of relying on one expensive server, and making it practical to store and process years of historical data rather than throwing it away. In practice, that meant new ways to split large jobs into smaller tasks, new engines for fast analysis, and cheaper, more scalable hardware strategies.

Distributed storage and processing
This refers to systems that break big jobs into smaller pieces and spread them across many ordinary computers instead of one powerful machine. Tools inspired by Google’s MapReduce and open source platforms like Hadoop made this possible.

Instead of asking one computer to process a huge file, you ask one hundred computers to process a small slice, and then the system combines the results. This made it possible to work with datasets that were far too large for a single machine.

Faster analytics engines
Frameworks like Spark made analysing large datasets much faster by keeping information in memory rather than constantly reading it from a disk.

It is like keeping ingredients on the counter while cooking instead of running back to the fridge every few seconds. This allowed teams to test ideas, adjust models, and run experiments more quickly.

Cheaper hardware
The cost of storage and memory dropped sharply. That meant companies could afford to keep many years of logs, images, videos, and audio files instead of deleting them to save space. More stored data meant more training material for AI models.

Result
Because of these three developments, both research labs and companies were able to train AI systems on far larger sets of examples than ever before. This increase in data volume directly strengthened model accuracy, reliability, and real world capability.

Algorithms that benefited

With more data and faster training, familiar methods started to perform at a new level.

• Speech recognition moved from small vocabularies to robust voice assistants on phones.
• Handwriting recognition improved on postal codes, bank checks, and forms.
• Image recognition jumped forward on large benchmarks and then in everyday products such as photo search and safety systems in cars.
• Language modeling and translation improved steadily as larger datasets and models became available.

We will study the details of these model families in the coming modules. For now, the main lesson is that scale matters. When data and computers reach a certain level, accuracy and reliability often improve quickly.

Earlier generations had the right ideas but not the resources to test them at full size. The 2000s brought the power to run very large training jobs and the data to teach models patterns found in the real world. That combination turned prototypes into systems that people could use every day.

1.1.3.4. AI Learns to Learn — The Deep Learning Era

Deep Learning arrives: layered models and landmark demonstrations

Around 2010 researchers showed that stacking many simple learning layers could extract patterns of growing complexity from raw data. This approach became known as deep learning. You will study the details of neural networks in the next module. For now, focus on what was achieved and why it mattered.

What “many layers” achieved in practice

A single layer can pick up simple features in data. Several layers can combine those features into shapes, sounds, and word fragments. When trained on large datasets with fast hardware, deep models learned useful representations directly from pixels, audio waveforms, and text. The result was a series of public demonstrations that were easy to understand and hard to ignore.

2011: IBM Watson wins Jeopardy!

IBM’s Watson system defeated top champions Ken Jennings and Brad Rutter on the television quiz show Jeopardy! in February 2011.

Watson played Jeopardy using its own internal knowledge base rather than live access to the internet. It ran on a large cluster of computers loaded with encyclopedias, news articles, and reference books. When it received a clue in natural language, the system generated several possible answers, searched its database for evidence for and against each one, and scored those options using statistical models. Based on those scores, it decided whether to press the buzzer and which answer to give. In practical terms, Watson followed a tight loop: interpret the clue, search its stored knowledge, compare candidate answers, select the most likely one, and answer only when its confidence passed a threshold.

Viewers saw a system handle puns, wordplay, and broad knowledge under time pressure. The demonstration illustrated how language processing, information retrieval, and probabilistic decision making could be integrated at scale. It also exposed the limits of such systems outside curated settings and motivated later work on grounding, robustness, and dialog.

2012: Deep learning transforms image recognition

In 2012, researchers entered a deep learning system called AlexNet into a major image recognition competition called ImageNet. The challenge was to identify objects in photographs from a very large dataset. Previous top systems were wrong about 26 percent of the time when picking from their five best guesses. AlexNet brought that error down to about 15 percent in a single year. For this field, that drop was huge. It showed a clear step change in how well machines could recognise objects in images.

How it worked

AlexNet did not rely on hand crafted rules about shapes or colours. The system learned directly from millions of images that had been labelled by humans.

• Early layers of the network learned to spot simple features such as edges and textures.
• Middle layers learned combinations of these features such as corners or simple patterns.
• Deeper layers learned to recognise whole parts and objects such as faces, wheels, or animals.

It used a function called ReLU to speed up learning and a technique called dropout to prevent overfitting, which means becoming too specialised on the training images and failing on new ones. Training needed a large amount of computation, so the team used two graphics processing units (GPUs) to run many calculations in parallel and finish training in a reasonable time.

Why it mattered

The size of the improvement convinced many research groups and companies that deep learning could beat older, hand engineered methods on vision tasks. After AlexNet, more teams invested in:

• Larger labelled datasets
• GPU based training
• Deeper and more complex neural networks

This success did not stay limited to images. The same ideas and hardware approaches soon spread to speech recognition and later to language models, forming the foundation of the AI systems people use in business today.

2015: DeepMind shows end-to-end learning to play Atari from pixels

Researchers at DeepMind showed that one learning system could learn to play many classic Atari 2600 video games just by looking at the screen and seeing the score go up or down. Over time, this system reached, and in some games exceeded, the level of human expert players. The results were published in Nature in 2015 after earlier technical reports in 2013.

How it worked

The Atari agent used a mix of deep learning and reinforcement learning:

• The system watched the game as a sequence of images (screen pixels).
• A deep neural network turned these images into an internal understanding of the game state.
• For each possible move (action), the network estimated how “valuable” that move might be for getting a higher score.
• After each game step, the system compared what actually happened to what it expected and adjusted its internal numbers to do better next time.
• It stored past game frames and replayed them during training so learning would be more stable and less noisy.

The important point is that the system did not receive any game specific tips or rules. It only received the images and a reward signal, for example a higher score, and learned through trial and error.

This work showed that a single general learning algorithm could:

• Learn directly from raw experience, not from handcrafted features,
• Improve through practice,
• And reach strong performance across many different tasks within the same framework.

It demonstrated that combining deep neural networks with trial and error learning could produce flexible behaviour, not just pattern recognition. This insight encouraged further research into systems that learn by interacting with their environment, which now influences robotics, recommendation systems, and many other areas of AI.

Deep learning is built on a simple but powerful idea: by stacking many small trainable layers, a model can gradually learn useful internal representations from raw data. Public demonstrations such as IBM Watson’s performance on Jeopardy!, AlexNet’s breakthrough on the ImageNet image recognition challenge, and DeepMind’s Atari-playing agent made this visible to the world by showing how large amounts of data and compute can unlock big jumps in accuracy and capability. The same training principles that powered these milestones soon spread into speech recognition, language translation, and many other applications, turning deep learning into a general foundation for modern AI systems.

1.1.3.5 From Winter to Widespread Use: What We Learned and Where It Leads

By the mid 2010s, artificial intelligence moved from research labs into everyday products and services.

• Phones authenticated users with face recognition.
• Driver assistance systems handled lane keeping and adaptive cruise control.
• Voice assistants transcribed speech and carried out simple tasks.

Artificial intelligence became an economic engine that supported entire product lines, supply chains, and service platforms.

What the AI winters taught the field

• Evidence over enthusiasm

  In both AI winters, funding and interest dropped when bold promises did not match real results in the lab or in production. The lesson was; that serious progress in AI must be proven with hard numbers on shared tests, not with visionary speeches.

  Modern AI teams therefore rely on clear metrics such as accuracy, error rate, or revenue uplift on specific benchmarks. For example, image recognition models are evaluated on well known datasets, speech systems are measured by word error rate, and recommendation systems are judged by click through or conversion rate.

  A claim only counts as progress when an independent team can run the same test and see the same improvement. This culture of measurable evidence is one of the main differences between current AI and earlier hype driven cycles.

• Learning over hand crafted rules

  The winters also showed that systems built from thousands of manually written rules break easily when the real world changes. Rule based expert systems performed well in narrow, stable situations, but they were expensive to update and often failed in new cases that the rules did not anticipate.

  In contrast, models that learn from data improve when they see more examples. When customer behavior shifts, a learned model can be retrained on fresh data. When new fraud patterns appear, a modern fraud model can adapt because its behavior comes from statistics, not from a fixed checklist of conditions.

  The practical insight is that long term robustness comes from learning pipelines (data collection, cleaning, training, evaluation), not from one time knowledge capture sessions. Organisations that invest in the ability to learn continuously from data are more resilient than those that rely only on static business rules.

• Scale matters

  The field also learned that many AI techniques only reach reliable performance when three ingredients are available at scale:

  1. Large, relevant datasets for training and testing.
  2. Sufficient compute power to train models on those datasets in a reasonable time.
  3. Iterative cycles of improvement where models are trained, evaluated, and refined repeatedly.

• Early prototypes often looked promising on small, carefully chosen examples, but they failed when deployed in large, messy real environments. As data storage became cheaper, GPUs became widely available, and cloud platforms made large scale training accessible, results on real world problems improved dramatically and in measurable ways.

  The practical lesson for organisations is that meaningful AI performance does not come from a single clever algorithm on a laptop. It comes from scaleable infrastructure that supports big datasets, regular retraining, and ongoing evaluation in production.

These lessons came from decades of constraint and consolidation. The field matured by building better datasets, better evaluations, and better training pipelines. That discipline enabled the breakthroughs that define contemporary practice.

Where capability stood by 2010 to 2016

• Vision models classified objects and faces with high accuracy on standard datasets.
• Speech recognition supported consumer grade dictation and assistants.
• Predictive models improved recommendations, fraud detection, and logistics planning.

Most systems were narrow. One model handled images, a different model handled speech, and another handled text. Cross domain reasoning was limited because representations were trained separately.

The next step unifies these strands. Advances in deep learning produced large language models and related foundation models that learn broad representations of text, code, images, and audio. These models handle long context, follow instructions, and transfer skills across tasks. They provide the substrate for modern platforms such as Cyrenza, where specialized agents coordinate over shared representations to deliver business outcomes.