Magic Paper: drawing out ideas

Dates

September 2003 — September 2005

Principal Investigator

Professor Randall Davis (Department of Electrical Engineering and Computer Science)

Goal

We want designers to be able to sketch, gesture, and talk with the computer about their work in the same way they would with another designer. Our vision is a design environment with intelligence embedded in the environment, allowing designers to work in familiar ways in familiar media (e.g., paper, whiteboards), yet give those media new and powerful capabilities (e.g., the ability to understand a sketch, ask intelligent questions about a design, etc.).

Overview

Drawings, sketches, and diagrams of many sorts are ubiquitous in engineering education and practice, providing a powerful way to envision, explain, and reason. Yet historically diagrams have been static, passive pictures, comprehensible only to human observers. We aim to change that:   we want to create a kind of "magic paper " that understands what is being drawn.

We want to make it possible for a variety of surfaces to behave as that "paper," ranging from active whiteboards, to tablet computers, to desktops in classrooms and benches in laboratories. This is a vision of computing that spans the range from tablet computer notebooks full of magic paper to a new view of desktop computing:   your (physical) desktop should compute.

More generally yet, we want people to be able to sketch, gesture, and talk about their ideas in the way they do when interacting with each other, and have the "paper" understand the messy freehand sketches, casual gestures, and fragmentary utterances that are part and parcel of such interaction. Once this happens, a variety of powerful next steps are possible:   a sketch of a design for a mechanical device, for example, might be simulated to display its behavior, analyzed (e.g., for structural soundness), or critiqued (e.g., for manufacturability).

We want to produce systems for a variety of domains, including mechanical engineering, software architecture (e.g., UML diagrams), and digital electronic schematics, as well as sets of tools that will make it simple for others to create sketch understanding systems in additional domains.

To date we have produced a demonstration system and a substantial foundation for the architecture of the next generation of this system. The proposed work will carry us through to the completion of that next generation architecture.

Project Output

Press Coverage

"MIT's tablet tech gets a look-see from Microsoft," Mass High Tech, July 21, 2003. View

"Smart software makes sense of rough sketches," New Scientist, September 12, 2003. View

"Breathing life into messy sketches," CNN, October 13, 2003. View

"Microsoft Looks in the Crystal Ball: Research chief demonstrates Magic Paper, virtual classrooms, and other projects," PC World, October 29, 2003. View

Conference and Workshop Papers

Hammond, Tracy and Davis, Randall (2003) LADDER: A Language to Describe Drawing, Display, and Editing in Sketch Recognition. Proceedings of IJCAI (International Joint Conference on Artificial Intelligence) 2003.

Alvarado, Christine and Davis, Randall (2002). A Framework for Multi-Domain Sketch Recognition. 2002 AAAI Spring Symposium on Sketch Understanding.

Davis, Randall (2002) Position Statement and Overview: Sketch Recognition at MIT. 2002 AAAI Spring Symposium on Sketch Recognition.

Hammond, Tracy (2002). A Domain Description Language for Sketch Recognitions. Student Oxygen Workshop 2002, July 2002.

Hammond, Tracy and Davis, Randall (2002). Tahuti: A Geometrical Sketch Recognition System for UML Class Diagrams. 2002 AAAI Spring Symposium on Sketch Understanding.

Hammond, Tracy and Gajos, Krzysztof and Davis, Randall and Shrobe, Howard (2002). An Agent-Based System for Capturing and Indexing Software Design Meetings. Proceedings of 2002 International Workshop on Agents in Design (WAID 2002), August 28-30 2002.

Hammond, Tracy and Sezgin, Metin, and Veselova, Olya and Adler, Aaron and Oltmans, Michael and Alvarado, Christine and Hitchcock, Rebecca (2002). Multi-Domain Sketch Recognition. Student Oxygen Workshop 2002, July 2002.

Alvarado, Christine and Davis, Randall (2001). Resolving ambiguities to create a natural sketch based interface. Proceedings of IJCAI-2001, August 2001.

Alvarado, Christine and Davis, Randall (2001). Preserving the freedom of paper in a computer-based sketch tool. Proceedings of HCI International 2001.

Foltz, Mark and Davis, Randall (2001). Query By Attention: Visually Searchable Information Maps. Proceedings of Fifth International Conference on Information Visualization (InfoVis 2001), London.

Foltz, Mark (2001). Ligature: Gesture-Based Configuration of the E21 Intelligent Environment. Proceedings of MIT Student Oxygen Workshop, July 2001.

Hammond, Tracy (2001). Natural Sketch Recognition in UML Class Diagrams. Student Oxygen Workshop 2001.

Oltmans, Michael and Davis, Randall (2001). Naturally Conveyed Explanations of Device Behavior. Proceedings of PUI-2001, November 2001.

Sezgin, Metin; Stahovich, Thomas and Davis, Randall (2001). Sketch Based Interfaces: Early Processing for Sketch Understanding. Proceedings of PUI-2001, November 2001.

Sezgin, Metin and Davis, Randall (2001). Free-Hand Stroke Approximation for Intelligent Sketching. MIT Student Oxygen Workshop 2001.

Lab Abstracts

Foltz, Mark. Dr. Jones: A Software Design Explorer's Crystal Ball. MIT Artificial Intelligence Laboratory, September 2002.

Alvarado, Christine; Sezgin, Metin; Scott, Dana; Hammond, Tracy; Kasheff, Zardosht; Oltmans, Michael; and Davis, Randall (2001). A Framework for Multi-Domain Sketch Recognition. MIT Artificial Laboratory, September 2001.

Alvarado, Christine and Davis, Randall (2000). Intelligent mechanical engineering design environment: From sketching to simulation. MIT Artificial Intelligence Laboratory Annual Abstract, September 2000.

Foltz, Mark, Weisman, Luke, and Gajos, Krzysztof (2001). Ligature: Gesture-Based Configuration of the E21 Intelligent Environment. MIT Artificial Intelligence Laboratory Annual Abstract, September 2001.

Foltz, Mark (2001). Graph Exploration for Software Archeology. MIT Artificial Intelligence Laboratory Annual Abstract, September 2001.

Foltz, Mark, and Davis, Randall. Query by Attention: Visually Searchable Information Maps. MIT Artificial Intelligence Laboratory Annual Abstract, September 2001.

Foltz, Mark and Davis, Randall (2000). Supporting Group Brainstorming. MIT Artificial Intelligence Laboratory Annual Abstract, September 2000.

Hammond, Tracy and Davis, Randall(2002). A Domain Description Language for Sketch Recognitions. MIT Artificial Intelligence Laboratory Annual Abstract, September 2002.

Hammond, Tracy and Gajos, Krzysztof and Davis, Randall and Shrobe, Howard (2002). Sketch Recognition in Software Design. MIT Artificial Intelligence Laboratory Annual Abstract, September 2002.

Hammond, Tracy and Oshiro, Kalani and Davis, Randall (2001). Natural Editing and Recognition of UML Class Diagrams. MIT Artificial Intelligence Laboratory Annual Abstract, September 2001.

Oltmans, Michael and Davis, Randall (2000). Understanding Naturally Conveyed Explanations of Device Behavior. MIT Artificial Intelligence Laboratory Annual Abstract, September 2000.

Sezgin, Metin and Davis, Randall (2000). Early Processing in Sketch Understanding. MIT Artificial Intelligence Laboratory Annual Abstract, September 2000.

Theses and Proposals

Alvarado, Christine (2003). Multi-domain Sketch Understanding. PhD Thesis Proposal, Department of EECS, MIT. January, 2003. Alvarado, Christine (2000). A natural sketching environment: Bringing the computer into early stages of mechanical design. Master's thesis, Massachusetts Institute of Technology, 2000.

Foltz, Mark (2003). Dr. Jones: A Software Design Explorer's Crystal Ball. Ph.D. Thesis, Massachusetts Institute of Technology, August 2003.

Foltz, Mark (2001). Dr. Jones: A Software Archaeologist's Magic Lens. PhD Thesis Proposal, Department of EECS, MIT. May, 2001.

Foltz, Mark (1998). Designing Navigable Information Spaces. Master's thesis, Massachusetts Institute of Technology.

Oltmans, Michael (2000). Understanding Naturally Conveyed Explanations of Device Behavior. Master's thesis, Massachusetts Institute of Technology, September 2000.

Sezgin, Metin (2001). Feature Point Detection and Curve Approximation for Early Processing of Free-Hand Sketches. Master's thesis, Massachusetts Institute of Technology, 2001.

Movies

Demonstration of the Assist sketch understanding system.