One of the hallmarks of the subject is the incredible variety of mathematical tools and theories that arise. Although perhaps daunting to the novice, this diversity enhances the range of possible interactions and cross fertilizations among the different mathematical disciplines, as each is called into play in quantifying, analyzing and developing practical algorithms. A key goal of the multimedia year at the IMA is to foster the interaction between researchers using similar tools in different multimedia modalities, and, on the other hand, providing researchers utilizing different mathematical techniques to study the same modality to compare results and combine promising methods. The year will play an important role in exposing the mathematical community to a new range of challenging and timely mathematical problems and applications. Particular attention will be paid to the training of postdoctoral researchers to be familiar with a wide variety of mathematical tools and techniques that will hopefully lay the foundations for a genuinely mathematical discipline that will become known as "multimedia."
The underlying mathematical theories can be broadly divided into several important, overlapping categories. First, since vision, language, music, video and other sensory processes are much less deterministic than ordinary physical phenomena, a large class of methods rely on probabilistic and stochastic paradigms. For example, Markov random fields form the basis of much current research in speech and feature recognition, in visual tracking, in segmentation, and in signal processing and storage. Statistical estimation and Bayesian methods feature in pattern recognition, in language modeling, in optical character recognition, as well as signal image reconstruction. Monte Carlo methods have been successfully used in the numerical analysis of some processes, as well as in computer design and graphics.
Wavelets and other transform methods appear in a wide variety of contexts, including image and signal compression and enhancement, computer graphics, texture analysis, and the recovery of degraded audio signals. A particularly striking example is the FBI's adoption of a wavelet-based system for compression of fingerprint images. (On the other hand, image enhancement techniques based on nonlinear differential equations have also been dramatically successfully in forensics and video enhancement.) Wavelet methods are particularly useful in the as yet poorly understood theory of image textures, but successfully incorporating them into the variational and partial differential equation approaches to segmentation has yet to be completed.
Since digital imagery is a fundamentally discrete data system, a particular surprise is the relevance of both nonlinear partial differential equations and variational methods in low level image and video processing. The mathematical techniques include the various recent approaches to curve shortening and mean curvature flows (originally studied by differential geometers) and the variational approaches of the type used in minimal surfaces. These methods are used in denoising images and in edge detecion of features, and have seen success in visual tracking and recognition of objects, medical imaging, video enhancement, and collision avoidance. Implementation of the relevant nonlinear diffusion equations has led to the development of fast, sophisticated numerical algorithms that have broad applicability. The passage from discrete pixel-based image to continuous partial differential equation and back to discrete finite difference numerical implementation sounds paradoxical, until one realizes that the fundamental processes in continuum mechanics and fluid dynamics work in precisely the same paradigm - the discrete molecular system is modeled by a continuous system, e.g. the Navier-Stokes equations, which are themselves integrated numerically via a discrete approximation.
The nonlinear partial differential equation approach to image processing relies on classical differential geometry. Modern fractal geometry, which arose in the study of natural phenomena, also has immediate applications in computer graphics, animation and the construction of artificial scenes. Fractal-based algorithms has had dramatic results in practical image compression, while very recent wavelet-fractal hybrids indicate the potentialities for cross disciplinary approaches to multimedia processes.
Algebraic theories arise in cryptography and information theoretic approaches to signal processing. Secure computing, copyright protection, and language grammars also rely on fundamental algebraic methods for implementation.
In computer graphics and geometric design, the digital representation of fully three-dimensional objects has been the subject of intense research activity in the past decade, leading to powerful new algorithms. Algebraic geometry, splines, wavelets, optics and solid geometry are but a few of the mathematical tools brought to bear on this vital area. Applications include computer graphics, surface and solid rendering, animation, flight simulation, virtual reality, and scientific and medical imagery. But real-time display of fully three-dimensional images remains problematic, necessitating further research in the fundamental mathematical issues. Inevitable technological advances in graphics performance will continuously change the requirements for real-time rendering and the complexity of models to be visualized.
Group theory is recognized to play an increasingly important role in media analysis; for instance, both human and camera visual systems naturally incorporate certain symmetries, including translations, rotations, scalings, and then progressing on to the less familiar groups of affine and projective transformations. Galilean invariance plays an important role in movie restoration and recognition of moving objects. In general, the reliable recognition of objects must form an integral part of any functional multimedia system. These include recognition of visual objects, target recognition, satellite images, speech and language recognition, optical character recognition, and language. Most of these are still in their infancy, with the underlying mathematical theory still in need of development. Applying symmetry groups to recognition problems in a natural fashion requires the adaptation of classical geometric theories of differential invariants and algebraic invariants, subjects that formed the core of pure mathematical research in the last century. Recent work on symmetry-incorporating numerical approximations hold significant promise for computation of invariants and object recognition in physical images.
Use of multimedia in manufacturing holds great potential for industrial applications. Visual control of the robots used in industrial processes, such as semiconductor manufacturing and etching, requires efficient, real-time processing of images, incorporating the denoising, enhancement, segmentation, and object recognition techniques from a computer vision system into a broader control-theoretic loop. Some striking experiments, including an image-based vehicular navigational system, show significant promise. Fast and efficient numerical implementation of the analytical algorithms provides the key to real-time applications and automatic control. Both finite difference and finite element methods have been successfully employed. For example, the use of level set methods for front tracking and interface evolution in phase transitions requires fast marching algorithms and efficient numerical implementations. However, to date the marriage of computer vision and control theory has yet to be properly consummated, requiring a new synthesis of the underlying mathematical theories.
In summary, while much progress can be seen on individual aspects, their synthesis into a mathematical theory of multimedia is as yet unexplored. This year at the I.M.A. will provide a unique and unprecedented opportunity to bring together researchers in a wide variety of mathematical disciplines and applications. Forging these disparate subjects into a new and vital subject will have long range effects, both within mathematics, and in the practical applications to multimedia science.
Special note about the worksohps on Mathematical Methods in Geometric Design and Computer Graphics: Most multimedia content today consists primarily of text, 2D images, video, and audio. Multimedia of the near future will additionally rely heavily on 3D graphics. The creation and use of 3D graphics in turn draws upon the two closely related disciplines of geometric design (the creation of the geometric objects), and computer graphics (the animation and display of the objects). These disciplines have been extraordinarily vibrant and diverse, with current applications including flight simulation, medical imaging, scientific visualization, computer-aided design, and entertainment. Results obtained in research laboratories over the past twenty years have given rise to a multi-billion dollar industry. The pending 3D graphics revolution is due in large part to impressive improvements in processor speeds and memory capacities. More impressive still are numerous algorithmic improvements that have opened up new application areas and have provided remarkable asymptotic reductions in both running time and memory requirements. Many of the algorithmic advances are based on mathematical methods such as wavelets, finite element analysis, Monte Carlo methods, and algebraic geometry. There is every reason to suspect that this trend can continue, so it is essential to stimulate collaboration between the mathematics and graphics communities. The goal of this pair of workshops is to do just that.