SCDS Course Materials

DATA 202 Module 10: Advanced Ethics - AI Safety, Governance, and the Future

Introduction

Building on the ethical foundations from DATA 201, this module confronts the advanced challenges emerging as AI systems become more capable and pervasive. From existential risk debates to international governance, from deepfakes to autonomous weapons, we grapple with questions that will shape the 21st century.

Part 1: AI Safety and Alignment

The Alignment Problem

As AI systems become more powerful, ensuring they pursue intended goals becomes critical. The alignment problem asks: how do we ensure AI systems do what we actually want?

Specification Problems:

Examples:

Inner vs. Outer Alignment

Outer Alignment: Is the objective function correct?

Inner Alignment: Does the system pursue the specified objective?

Alignment Techniques

Reinforcement Learning from Human Feedback (RLHF):

  1. Train initial model
  2. Have humans rank outputs
  3. Train reward model on rankings
  4. Fine-tune with reinforcement learning

Constitutional AI: Encode principles and have model self-critique

Debate and Amplification: Have AI systems argue positions for human judgment

Part 2: Existential Risk

The Long-Term Perspective

Some researchers argue that advanced AI poses existential risk—potential to cause human extinction or permanent civilization collapse.

Arguments for Concern:

Arguments Against:

Notable Voices:

The Pause Debate

In March 2023, prominent researchers signed an open letter calling for a six-month pause on training AI systems more powerful than GPT-4. The debate highlighted tensions between:

Part 3: Synthetic Media and Deepfakes

The State of Synthetic Media

Deepfakes: AI-generated or manipulated media

Capabilities in 2024+:

Harms and Applications

Harmful Uses:

Legitimate Uses:

Detection and Defense

Technical Approaches:

Policy Approaches:

Part 4: AI Governance

The Regulatory Landscape

EU AI Act (2024):

US Approach:

China:

International:

The Governance Gap

Challenges for effective governance:

Part 5: Labor and Economic Disruption

The Automation Wave

AI threatens different jobs than previous automation:

Studies estimate:

Responses and Adaptation

Education and Retraining:

Policy Responses:

New Opportunities:

Part 6: Environmental Impact

The Energy Cost of AI

Training Costs:

Inference at Scale:

Mitigations

Technical:

Policy:

DEEP DIVE: The AI Pause Letter and the Future of AI Governance

A Plea to Slow Down

On March 22, 2023, the Future of Life Institute published an open letter titled “Pause Giant AI Experiments.” Signed by over 30,000 people including Elon Musk, Steve Wozniak, and Yoshua Bengio, it called for a six-month pause on training systems more powerful than GPT-4.

The letter argued:

The Debate

Supporters argued:

Critics argued:

The industry response: No major lab paused. Training continued. OpenAI released GPT-4 during the petition period.

What Happened Next

The letter galvanized debate but didn’t pause development. What followed:

Lessons

The episode revealed:

  1. Coordination is hard: Competitive pressures prevent unilateral slowing
  2. The Overton window shifted: Safety became mainstream discussion
  3. Governance lags capability: Policy follows technology
  4. No consensus on risk: Experts disagree fundamentally on priorities

DISCUSSION EXERCISE: Governance Scenarios

Scenario 1: Deepfake Election Interference

A realistic deepfake of a political candidate surfaces days before an election. The candidate claims it’s fake, but verification takes time. What policies could prevent or mitigate this? Who is responsible?

Scenario 2: Autonomous Weapons

An AI system controls a military drone that makes lethal decisions without human approval. A strike kills civilians. Who is responsible—the programmer, commander, manufacturer, algorithm?

Scenario 3: Job Displacement

AI automation eliminates 50% of jobs in a particular industry within 5 years. Workers cannot easily retrain. What policies should governments enact? What responsibilities do companies have?

Scenario 4: AI-Generated Science

Researchers use AI to generate papers that pass peer review but contain fabricated data. How should the scientific community respond? What policies would help?

Books

Organizations

Papers

Module 10 confronts advanced ethical challenges in AI—from alignment and existential risk to deepfakes and governance. As AI systems become more capable, the questions become more urgent and the stakes higher.