teaching

Course materials for the Interactive Audio Lab

View the Project on GitHub interactiveaudiolab/teaching

COMPUTATIONAL CREATIVITY, Northwestern University EECS 397/497 fall 2019

Loctation: Technological Institute LG52

Day/Time: Thursdays, 1:00pm - 3:50pm

Instructor: Bryan Pardo

Course Description

Computational creativity is a multidisciplinary field that lies at the intersection of artificial intelligence, cognitive psychology, philosophy, and the arts. The field is concerned with the theoretical and practical issues in the study of creativity. The goal of computational creativity is to achieve one of the following:

In this course, students will read and discuss theoretical writings on the nature and proper definition of creativity. They will also read about and experiment with existing computational creativity systems (e.g. David Cope’s Experiments in Musical Intelligence and Google’s Project Magenta). In parallel, they will perform practical work implementing systems aimed at achieving one of the three goals listed above.

Course Calendar

Week Date Topic Due
1 Sep 26 Basics of deep nets. Can computers create art?  
2 Oct 3 What is Creativity? Who owns creative works? 5 reviews
3 Oct 10 Algorithmic music composition 4 reviews
4 Oct 17 Algorithmic music composition 4 reviews
5 Oct 24 No class: prepare your proposal 4 reviews
6 Oct 31 Algorithmic image generation initial proposal
7 Nov 7 Algorithmic image generation Project plan
8 Nov 14 Cross-modal generation 4 reviews
9 Nov 21 Text and story generation 4 reviews
10 Nov 28 No class: Thanksgiving  
11 Dec 5 Support rather than supplant? project website
12 Dec 10 (9-11am) Final project presentation final presentation

Course assignments

Reading: 50 points

You will submit 25 reviews of readings/videos/music from the course website. Each will be a single-page reaction to something you read/watched/heard from the links provided below. Each review will be worth 2 points. Reviews are due on the schedule shown in the course calendar.

Class Paper Presentations: 10 points

Once during the course of the term, you will be the lead person discussing the reading in class. This will mean you haven’t just read the paper, but you’ve read related work, really understand it and can give a brief presentation of the paper (including slides) and then lead a discussion about it. (10 points)

Class participation: 10 points

Each week (even weeks when you’re not presenting) you are expected to show up, have read the papers and be able to discuss ideas. Every week you show up and substantially contribute to the discussion, you get 1 point. If you don’t show up or you don’t say anything that week, you don’t get the point.

Project in computational creativity: 30 points

You will make, modify, and or analyze some work or project in computational creativity. This may mean modifying MusicVAE or making a simple story generator of your own. It may mean downloading an existing thing and experimenting with it or it may mean building a new thing. It may mean making a program that analyses creativity or a creativity aid…or something I haven’t been able to come up with. The point breakdown for the project is as follows

Week 1 Basics of Deep Nets

Basics of Deep Nets

Chapter 4 from Machine Learning

Convolutional Networks for Images, Speech, and Time-Series

Generative Adversarial Nets

Slides of the NeurIPS GAN tutorial

Understanding LSTMs

Can Computers Create Art?

On the Future of Computers and Creativity

Can Computers Create Art?

The “Can Computers Create Art?” Talk on video

A Computer “Artist”

Deepfake Salvador Dalí takes selfies with museum visitors

Week 2: What is Creativity? How do you measure it? Who owns creativce works?

In Class Presentations

Max Morrison: What is KL Divergence?

Andong Li Zhao: Approaches to Measuring Creativity: A Systematic Literature Review

Max Morrison: Quantifying the Creativity Support of Digital Tools through the Creativity Support Index

Brandon Harris: Monkey Selfie Copyright Dispute

Other readings…

What is Creativity?

The Standard Definition of Creativity

How do we measure creativity?

How Creativity is Measured

Who are the stakeholders in a creative work?

Artificial Intelligence and Music: Open Questions of Copyright Law and Engineering Praxis

Week 3: Algorithmic music composition

A talk you should go to

Come to Andrew McPherson’s talk and react to that: 3rd floor lecture room, Mudd building. Noon, Oct 9.

A Historical Perspective

Andreas Bugler CHAPTER 4 BOOK: Formalized Music Thought and Mathematics in Composition - Xenakis ** You can review up to 3 chapters of this book. Each chapter is worth 1 point.**

Connor Bain David Cope and Experiements in Musical Intelligence

David Cope’s YouTube Channel

“Deep” takes on music making

A Universal Music Translation Network

The output of Facebooks Universal Music Translation Network

S Dadabots. This is a project where deep nets are generating entire albums of content. Note there are several papers an albums on this site. If a student presenter wants to pick one, I’m open to that.

Week 4: “Deep” algorithmic music composition

Variational Auto Encoders (VAES)

Autoencoders

Jacob Kelter A tutorial on Variational Autoencoders

Stretch reading goal: A more in depth tutorial on Variational Autoencoders (academic)

Learning Latent Representations of Music to Generate Interactive Musical Palettes This is the MusicVAE paper.

The MusicVAE blog description

GAN music generation

The ISMIR 2019 tutorial on generating music with GANs

Other deep methods

Alexander Fang DeepBach: a Steerable Model for Bach Chorales Generation

Other Readings

Deepfakes and Cheapfakes

Music Generation by Deep Learning - Challenges and Directions

Deep Learning Techniques for Music Generation – A Survey

Creative Computation Systems

Magenta

DeepBach Example Output & Code

Week 5: NO CLASS. Read papers. Write your proposal

Weeek 6: Algorithmic Image Generation

Before there was deep learning there was…

Jack Wiig Coloring without seeeing: A problem in machine creativity

Kevin Chan Painterly Rendering for Video and Interaction

Video Creation

Cooper Barth The video-to-video paper

The video-to-video video

A video tutorial on video generation with GANs

Creative Computation Systems

5 Video creation aids

Week 7 Algorithmic Image Generation

Gan

A Neural Algorithm of Artistic Style

Marko Sterbentz Image Style Transfer Using Convolutional Neural Networks

Thomas Young GANGogh: Creating Art with GANs

Non Technical Readings

Google’s psychedelic ‘paint brush’ raises the oldest question in art

Creative Computation Systems you can try/build

The deep dream image generator

A Practical guide to build your first Deep Dream Experience

Art Breeder

Week 8: Cross-modal Creation

Making visuals or music, condition from text

S Generative Adversarial Text-to-image Synthesis

S StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks

A video explanation of StackGAN

Jayden Soni Conditional LSTM-GAN for Melody Generation from Lyrics

S Algorithmic Songwriting with ALYSIA

The Alysia app

Gabriel Caniglia Image-to-image translation with conditional adversarial networks

Non Technical Readings

Creative Computation Systems

Sourcecode for Conditional LSTM-GAN for Melody Generation from Lyrics

Demo of Pix2Pix network from the Image-to-Image Translation paper

Week 9: Text and story generation

Technical Readings

S Attention models: AKA Transformer networks: Attention is all you need (academic)

Sarah Ahmad Narrative Planning: Balancing Plot and Character

Lisa Cox Event representations for automated story generation with deep neural nets

Alisa LiuRationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations

The Shape of Games to Come: Critical Digital Storytelling in the Era of Communicative Capitalism This is a dissertation. So it’s long.

Non Technical Readings

Popular press on Open AI’s writing system

Open AI has released GPT-2

The writer in the machine: Automatic story generation

AI Wrote a Road Trip Novel. Is it a good read?

You can buy ‘1 the Road’here

Creative Computation Systems

Essaybot

Open AI’s blog about the GPT-2 Language Model

The actual Open AI GPT-2 Language Model

Sunspring: A 9 minute movie whose script was written by the “Jetson” LSTM

Week 10: NO CLASS (THANSKGIVING) Work on your project

Week 11: Support rather than supplant? Who owns the work?

Technical Readings

Siddhartha Pamidighantam Fashion++: Minimal Edits for Outfit Improvement

Katherine O’Toole Learning to build Natural Audio Production Interfaces

S (iGAN) Generative Visual Manipulation on the Natural Image Manifold

Non Technical Readings

Creative Computation Systems

Prisma app

iGan video

iGan source code

Getting started with ML coding

Anaconda is the most popular Python distribution for ML work.

Jupyter is the most popular notebok and visualization framework for python ML development.

Scikit learn is the most popular general ML framework.

The Pytorch homepage gets you started on the easiest of the popular deep learning frameworks. It has source code, blogs, tutorials.

The tensorflow homepage gets you started on the harder popular deep learning framework. It has source code, blogs, tutorials.