How can we speak math?

Saturday, December 9, 2006 - 2:00pm - 2:30pm
EE/CS 3-180
Richard Fateman (University of California, Berkeley)
Surprisingly, we can speak mathematics to a computer probably more rapidly and accurately than handwriting. Even better is to speak and use pointing or handwriting. A combination may allow us to identify and cancel errors in one mode or another. In some cases speaking may be more convenient than typing, even for rapid typists: many mathematical symbols missing on the keyboard can be easily spoken. Even without venturing into Greek, handwriting or even typing fifty million is probably slower and more error-prone than speaking it.

Pursuing the goal of effectively speaking small pieces of mathematics, we wondered how hard it would be to speak arbitrarily long sections of mathematics, including nested complex expressions.

We first describe programs for the inverse problem: computer generation of mathematical speech. In so doing we find that we need to suggest a few speaking conventions to overcome the unfortunately ambiguous and inconsistent common usages of mathematics.

Then we consider tools and guidelines to make it more plausible for humans to speak full mathematical formulas so they can be recognized by a computer using a speech recognizer program.

We describe our prototype programs which do somewhat less than we propose, but are effective in that speech can either be used alone, or used to fill in boxes (superscripts, etc.) or larger pieces, or for choosing alternatives from plausible symbol recognition from handwriting. We believe the principle barriers to engineering a more complete program can be overcome, though a driving application may be essential for refining prototypes into useful programs. This paper is not intended to be the last word on the subject, but to expose problems and approaches relevant to the task.