A Historic Moment#

On June 12, 2017, a paper titled “Attention Is All You Need” quietly appeared on the arXiv preprint server. The title seemed casual, even playful—it was clearly a tribute to The Beatles’ classic song “All You Need Is Love.” However, this seemingly unremarkable paper would trigger a revolution that would sweep through the entire artificial intelligence field in the following years.

The paper’s eight authors came from Google: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. Interestingly, these eight authors were listed as “equal contributors,” with the order in the paper being random, reflecting the truly collaborative nature of this work.

More interestingly, the origin of the name “Transformer” was quite dramatic. According to Jakob Uszkoreit’s recollection, he chose this name simply because he “liked the sound of the word.” Early design documents were even named “Transformers: Iterative Self-Attention and Processing for Various Tasks” and included illustrations of six characters from the Transformers animated series. The research team was also called “Team Transformer.”

Core Innovation: Abandoning Everything, Keeping Only Attention#

The core contribution of this paper can be summarized in one sentence: It completely abandoned traditional recurrent and convolutional structures and proposed a new architecture based purely on attention mechanisms—the Transformer.

Before this, attention mechanisms usually existed only as auxiliary components to RNNs or CNNs. The revolutionary aspect of the Transformer was that it proved attention mechanisms alone were powerful enough, requiring no help from recurrence or convolution.

Self-Attention: Letting Every Word “See” the Global Context#

To understand the core of the Transformer—the self-attention mechanism—we can use a vivid analogy:

Imagine you’re reading a sentence: “The cat sat on the mat, it looked very comfortable.” When you read the word “it,” your brain automatically connects it with the earlier “cat.” This process is completed instantly; you don’t need to review word by word, but can “scan” the entire sentence at a glance to find the most relevant words.

The self-attention mechanism simulates exactly this process. For each word in a sentence, the model calculates its “relevance score” with all other words in the sentence (including itself). This process involves three key concepts:

Query: Can be understood as “What information am I looking for?” Key: Can be understood as “What information can I provide?” Value: Can be understood as “What information do I actually contain?”

By calculating the similarity between Query and Key, the model can determine the importance of each word to the current word, then perform a weighted sum of Values based on this importance to obtain the final representation.

The Revolution of Parallelization#

The greatest advantage brought by the self-attention mechanism is parallelization. Unlike RNNs that need to process sequences step by step, Transformers can process all positions in a sequence simultaneously. This means:

• Training speed improved by several orders of magnitude: Training that previously took weeks might now only take days
• Better utilization of GPU resources: Modern GPUs excel at parallel computation, and the Transformer architecture perfectly matches this characteristic
• Easier scaling to larger models: Parallelization makes training ultra-large-scale models possible

The experimental results in the paper were shocking: On the WMT 2014 English-German translation task, the Transformer achieved a BLEU score of 28.4, improving over the previous best result by more than 2 BLEU points. More importantly, this result was achieved with only 3.5 days of training on 8 GPUs, while previous best models required much higher training costs.

Philosophical Thinking on Paradigm Shift#

The emergence of the Transformer was not just the birth of a new architecture, but more importantly, it catalyzed a completely new machine learning paradigm: “pre-training-fine-tuning.” The core idea of this paradigm can be understood through a simple analogy:

Traditional machine learning was like training specialized technicians for each specific task—you needed a technician specifically for car repair, one for computer repair, and one for watch repair. Each technician had to learn from scratch, even though their work had many commonalities.

The “pre-training-fine-tuning” paradigm is like first training a knowledgeable generalist, letting them master various basic knowledge and skills, then providing short-term specialized training for specific tasks. This generalist, with a solid foundation, can quickly adapt to various different tasks.

Pre-training: Becoming a “World Knowledge Compressor”#

The goal of the pre-training phase is not to learn any specific task, but to let the model learn the grammar, facts, and logic of language itself. This process usually involves two main training objectives:

Masked Language Model: Randomly mask some words in sentences and let the model predict the masked words based on context. This is like having students do fill-in-the-blank exercises, understanding the inherent patterns of language through extensive practice.

Next Token Prediction: Given a sequence of preceding words, predict the next most likely word. This task seems simple, but to do it well, the model must understand grammar, semantics, and even world knowledge.

The scale of pre-training data is unprecedented. Researchers used text data from the entire internet—from Wikipedia entries to news articles, from novels to technical documents, from social media posts to academic papers. This massive text data contains almost all human knowledge and experience, and by learning from this data, models gradually become “world knowledge compressors.”

Fine-tuning: The Elegant Transformation from Generalist to Specialist#

With the foundation of pre-training, the fine-tuning phase becomes relatively simple. Researchers only need to use a small amount of labeled data for specific tasks to provide short-term specialized training to this “knowledgeable generalist,” and it can perform excellently on that task.

The effectiveness of this approach is astonishing. A model pre-trained on large amounts of text only needs a few thousand labeled samples to achieve or even exceed the performance of models specifically designed for tasks like sentiment analysis, question answering, and text summarization.

The GPT Series: From Proof of Concept to Phenomenal Breakthrough#

OpenAI keenly seized the opportunity of this paradigm shift and launched the GPT (Generative Pre-trained Transformer) series:

GPT-1 (2018): Proof of concept phase, with 117 million parameters. Although not large in scale, it proved the feasibility of the “pre-training + fine-tuning” paradigm.

GPT-2 (2019): Parameters jumped to 1.5 billion, demonstrating surprising text generation capabilities. OpenAI even initially refused to release the complete model due to concerns about misuse.

GPT-3 (2020): A behemoth with 175 billion parameters, demonstrating unprecedented “emergent abilities.” It could not only generate coherent articles but also perform mathematical reasoning, write code, create poetry, and even show certain common-sense reasoning abilities.

Emergent Abilities: When Quantitative Change Leads to Qualitative Change#

The most shocking discovery of GPT-3 was the existence of “emergent abilities.” When model scale breaks through a certain critical point, it suddenly demonstrates abilities that were never explicitly learned during training. This is like water suddenly freezing at 0 degrees—when quantitative accumulation reaches a certain level, qualitative leaps occur.

These emergent abilities include:

• Few-shot learning: Understanding new tasks with just a few examples
• Zero-shot learning: Completing never-before-seen tasks based solely on task descriptions
• Reasoning abilities: Performing multi-step logical reasoning
• Creative abilities: Creating original stories, poetry, and code

These discoveries made researchers realize they might be approaching some form of more general artificial intelligence.

Google’s Counterattack: From Inventor to Chaser#

Ironically, Google, the inventor of the Transformer, fell behind in the revolution it started. While OpenAI’s GPT series was making great strides in generative AI, Google seemed to still be immersed in the comfort zone of search and advertising.

But Google didn’t sit idle. They launched a series of powerful counterattacks:

BERT (2018): Bidirectional Encoder Representations from Transformers, a bidirectional Transformer encoder. Unlike GPT’s unidirectional generation, BERT could utilize contextual information simultaneously, performing excellently on understanding tasks. BERT’s release triggered another revolution in the NLP field, with almost all understanding tasks being refreshed by BERT and its variants.

T5 (2019): Text-to-Text Transfer Transformer, unifying all NLP tasks as “text-to-text” conversion problems. This unified framework demonstrated the powerful versatility of the Transformer architecture.

LaMDA, PaLM, Gemini: Google’s continued exploration in conversational and multimodal AI, attempting to regain technological leadership.

OpenAI’s Commercial Transformation: From Open to Closed Source#

OpenAI’s development trajectory is quite dramatic. This organization, initially named for “openness,” gradually moved toward a closed-source path:

Open Period (2015-2019): When OpenAI was founded, its mission was to “ensure artificial general intelligence benefits all of humanity.” They publicly released research results, including the complete GPT-1 model.

Turning Point (2019-2020): As GPT-2’s powerful capabilities became apparent, OpenAI began worrying about technology misuse and chose not to fully release the model for the first time. GPT-3’s release marked the establishment of OpenAI’s commercialization strategy.

API Economy (2020-present): OpenAI no longer directly releases models but provides services through APIs. This model both protects technological advantages and creates considerable commercial value.

The Awakening of the Open Source Community: The Power of Democratization#

Just as OpenAI moved toward closed source, the open source community began to awaken. The catalyst for this awakening was Meta’s (Facebook) release of the LLaMA series in 2023.

LLaMA’s Accidental Leak: Although Meta initially only provided LLaMA to research institutions, the model was quickly leaked to the internet. This “accident” triggered an explosion of innovation in the open source community.

The Flourishing of Derivative Models: Based on LLaMA, the open source community quickly developed numerous derivative models:

• Alpaca: Stanford University’s instruction-following model fine-tuned from LLaMA
• Vicuna: A conversational model developed by UC Berkeley and other institutions
• WizardLM: Microsoft Asia Research Institute’s complex instruction-following model

These open source models gradually approached or even exceeded closed source models in certain tasks, proving the viability of the open source path.

Global Competition Landscape: Technology Democratization vs. Commercial Monopoly#

This LLM competition quickly evolved into global technological competition:

China’s Rise:

• Baidu Wenxin: Large language model based on self-developed architecture
• Zhipu ChatGLM: Open source model with Tsinghua University technical background
• Alibaba Tongyi Qianwen: Alibaba’s multimodal large model

European Efforts:

• Mistral AI: French open source large model company, trying to carve a third path in US-China competition

Other Players:

• Anthropic: Founded by former OpenAI employees, focusing on AI safety with the Claude series
• Cohere: Canadian company focusing on enterprise applications

The essence of this competition is the game between technology democratization and commercial monopoly. The open source camp believes AI technology should belong to all humanity, while the closed source camp believes only through commercialization can technology safety and sustainable development be ensured.

From GPT-3 to ChatGPT: The Crucial Final Step#

When OpenAI released ChatGPT on November 30, 2022, many people didn’t realize this would be a world-changing moment. From a technical perspective, ChatGPT wasn’t a completely new breakthrough—it was based on GPT-3.5 and had no fundamental architectural changes compared to GPT-3.

But it was this “final step” that truly brought powerful AI technology to the masses. The key to this step was a technology called RLHF (Reinforcement Learning from Human Feedback).

RLHF: Teaching AI to “Read the Room”#

The core idea of RLHF technology is to teach AI models to understand and satisfy human preferences. This process can be divided into three steps:

Step 1: Supervised Fine-tuning (SFT) Human annotators write high-quality responses to various prompts, and the model learns basic conversational skills by studying these examples.

Step 2: Reward Model Training For the same prompt, the model generates multiple different responses, and human annotators rank these responses to indicate which is better. Based on this ranking data, a “reward model” is trained to predict human preferences.

Step 3: Reinforcement Learning Optimization Using the reward model as a “teacher,” reinforcement learning algorithms (usually PPO, Proximal Policy Optimization) are used to optimize the language model to generate responses that better align with human preferences.

The effect of this process is significant. Models trained with RLHF can not only generate more helpful, honest, and harmless responses but also understand complex instructions and engage in multi-turn conversations.

The “iPhone Moment”: Simple Interface Behind Complex Technology#

The release on November 30, 2022, can be called the “iPhone moment” of the AI field. Just as the iPhone packaged complex smartphone technology in a simple, easy-to-use interface, ChatGPT packaged powerful large language model technology in a simple chat interface.

Behind this seemingly simple interface was the crystallization of years of technological accumulation:

• Transformer architecture provided powerful language understanding and generation capabilities
• Large-scale pre-training gave the model rich world knowledge
• RLHF technology taught the model the art of conversing with humans
• Carefully designed user interface allowed ordinary users to easily use these advanced technologies

Global Phenomenon: From Tech Circles to All of Society#

ChatGPT’s influence far exceeded tech circles. Within just two months of release, it gained 100 million active users, becoming the fastest-growing application in history.

Shock in Education: Students began using ChatGPT for homework, forcing teachers to rethink teaching methods and assessment standards.

Workplace Transformation: From programmers to lawyers, from journalists to marketers, professionals in all industries began exploring how to use ChatGPT to improve work efficiency.

Investment Boom: ChatGPT’s success triggered a new wave of AI investment, with countless startups emerging to try to get a piece of this emerging market.

Intensified Social Discussion: From AI’s potential risks to employment impacts, from education’s future to the nature of creativity, ChatGPT sparked deep societal thinking about AI.

A Milestone in Technology Democratization#

ChatGPT’s most important significance lies in achieving true democratization of AI technology. Before this, using advanced AI technology required deep technical background and expensive computational resources. ChatGPT allowed anyone to use the most advanced AI technology through simple natural language conversation.

This democratization brought far-reaching impacts:

• Lowered the threshold for AI applications: No programming knowledge needed, anyone could use AI to solve problems
• Sparked innovation potential: Experts in various industries began exploring AI applications in their fields
• Promoted AI education: More and more people began learning about AI technology
• Advanced AI ethics discussions: Technology popularization made more people concerned about AI ethics and social impacts

The Official Opening of the Large Language Model Era#

ChatGPT’s success marked our official entry into the “Large Language Model Era.” In this era, AI is no longer just a tool but begins to play the role of assistant, partner, and even creator.

Rapid Expansion of Capability Boundaries: Every few months, new models are released demonstrating stronger capabilities. From text generation to code writing, from mathematical reasoning to creative writing, AI’s capability boundaries are expanding at unprecedented speed.

Explosive Growth of Application Scenarios: Applications based on large language models are emerging like mushrooms after rain, covering almost all fields including education, healthcare, law, finance, and entertainment.

Paradigm Shift in Human-Computer Interaction: Natural language is becoming the primary mode of human-computer interaction. We no longer need to learn complex commands or operation interfaces but can directly converse with AI in everyday language.

Facing Enormous Challenges#

However, this new era also brings unprecedented challenges:

Hallucination Problem: The Blurred Boundary Between Fact and Fiction Large language models sometimes generate information that seems reasonable but is actually incorrect, called the “hallucination” problem. When AI is used in scenarios requiring high accuracy (such as medical diagnosis, legal consultation), this problem can have serious consequences.

Bias and Toxicity: The Original Sin of Training Data Large language models’ training data comes from the internet, which contains large amounts of bias, discrimination, and harmful content. Models may learn and amplify these problems, showing bias in gender, race, religion, and other aspects when generating content.

Energy Consumption: Environmental Sustainability Considerations Training and running large language models require enormous computational resources and energy consumption. It’s estimated that training a GPT-3-scale model consumes electricity equivalent to several hundred households’ annual usage. As model scales continue to grow, this problem becomes increasingly serious.

Social Impact: The Dual Challenge of Employment and Ethics The popularization of large language models may have profound impacts on the job market. Some jobs traditionally requiring human intelligence (such as writing, translation, customer service) may be replaced by AI. Meanwhile, ethical issues such as the authenticity and copyright ownership of AI-generated content urgently need resolution.

The Urgency of Regulation and Governance#

Facing these challenges, governments and international organizations worldwide are accelerating AI governance:

EU AI Act: The EU is developing the world’s first comprehensive AI regulatory law, trying to find a balance between promoting innovation and protecting citizens’ rights.

US Executive Orders: The Biden administration issued executive orders on AI safety and trustworthiness, requiring AI companies to conduct safety assessments before releasing large models.

China’s Management Measures: China is also developing relevant AI management measures, particularly targeting algorithmic recommendation and deep synthesis technologies.

Strengthened International Cooperation: International organizations like G7 and G20 are beginning to discuss international cooperation mechanisms for AI governance.

New Directions in Technological Development#

To address these challenges, researchers are exploring multiple technical directions:

Interpretability Research: Trying to understand the internal working mechanisms of large language models to make AI decision processes more transparent.

Alignment Research: Ensuring AI system behavior remains consistent with human values, avoiding harmful or inappropriate AI behavior.

Efficiency Optimization: Through model compression, knowledge distillation, and other technologies, maintaining performance while reducing computational costs and energy consumption.

Multimodal Fusion: Combining text, images, audio, and other modalities to develop more general and powerful AI systems.

Looking Back at History and Forward to the Future#

From the publication of the “Attention Is All You Need” paper in 2017 to ChatGPT’s global explosion in 2022, in just five years, the Transformer architecture and large language models completely changed the trajectory of artificial intelligence development. This process was filled with surprises from technological breakthroughs, intense commercial competition, and profound social transformation.

Looking back at this history, we can see several key turning points:

• 2017: The proposal of the Transformer architecture laid the foundation for subsequent development
• 2018-2020: The evolution of the GPT series proved the power of large-scale pre-training
• 2022: ChatGPT’s release achieved mass popularization of technology
• 2023-present: The white-hot global AI competition, intense rivalry between open source and closed source

The Unfinished Journey#

However, this is just the beginning. While current large language models are powerful, they still have a long way to go before achieving true Artificial General Intelligence (AGI). They lack true understanding capabilities, cannot directly interact with the physical world, and lack continuous learning and self-improvement abilities.

The next major breakthrough might come from:

• Multimodal Fusion: Enabling AI to simultaneously process text, images, audio, video, and other types of information
• Embodied Intelligence: Giving AI physical bodies to act and learn in the real world
• Continuous Learning: Enabling AI to continuously learn from new experiences without retraining
• Causal Reasoning: Enabling AI to understand causal relationships between things, not just statistical correlations

Humanity’s Role in the AI Era#

Faced with AI’s rapid capability improvement, humans need to rethink their role in this world. We should not view AI as a threat but as a tool to enhance human capabilities. The key lies in ensuring AI development always serves human welfare and maintaining human agency and creativity while enjoying AI’s convenience.

Next Episode Preview: The New World of Multimodality#

When the boundaries of text are broken, when AI begins to learn to “see” and “hear,” when the boundaries between virtual and reality become blurred, we will welcome a completely new multimodal AI era. In the next episode, we will explore how AI breaks through single-modality limitations toward true multimodal fusion, and how this will drive us toward more general artificial intelligence.

The revolution ignited by attention mechanisms is far from over; the most exciting chapters may still lie ahead.

In this era of rapid AI development, each day may bring new breakthroughs. Let us maintain curiosity and an open mind, witnessing together the unfolding of this great era.