A moment with Professor Akaike (at right) and Bozdogan (at left) at the Institute of Statistical Mathematics (ISM) of Tokyo Japan during August of 1988, while Bozdogan was visiting ISM as an invited Research Fellow & Visiting Associate Professor as the guest of ISM and the Ministry of Education, Science and Culture of Japan.

Informational Statistical Modeling

"The grotesque emphasis on significance tests in statistics courses of all kinds... is
taught to people, who if they come away with no other notion, will remember that
statistics is about tests for significant differences..."
In John A. Nelder (1985)
Journal of Royal Statistical Society
Section A, 148
Scientists, practitioners, and students are frequently confronted with the problem of statistical model evaluation and selection in their work and area of specialties. Informational statistical modeling is a modern branch of Statistical Science and activity which draws its basis from entropy and information theoretic concepts in statistically analyzing potentially massive data sets and structures with the goal of identifying a model characterized by parsimony. Information criteria are used to estimate expected entropy (analogous to expected uncertainty in the data) by taking simultaneously into account both the "goodness-of-fit" of a model and the complexity of the model required to achieve that fit. Model complexity involves both the number of parameters and the interaction (correlation) between the parameters.
This new approach was pioneered by Professor Hirotugu Akaike in Japan in 1971 in his seminal work based on entropy, information and likelihood and centered around Akaike's Information Criterion (AIC). Following in the footsteps of his mentor, Akaike, Professor Hamparsum Bozdogan extended Akaike's work beginning in 1981. Professor Bozdogan's research and scholarly work based on the use of the covariance complexity index of van Emden (1971), has lead to several important information criteria on his own with excellent performance capabilities as decision rules for model selection. These information-based model selection techniques permit a systematic approach to model evaluation and selection which enable the evalaution of a massive number of models in a single analysis that typically takes less than 2 minutes of computing time on a personal computer for a multi-dimensional data with large number of observations. In the past, a researcher would have to separately analyze each model and use a combination of some objective and subjective criteria in selecting a "best" model.
The developments and the introduction of informational modeling and recent advancements in personal computers have marked the beginning of an era of systematic approach to model evaluation and selection. These statistical and technological advancements are permitting researchers to efficiently address questions such as, given a particular situation involving uncertainty and variability and a collection of possible parametric models of differing degrees of sophistication, how do we decide between these models on the basis of available data? This is the role of the "model selection."
"Model selection" refers to the choice of the "best" model(s) from a portfolio of competing alternative models for a given data set. A "model selection criterion" is a function which provides a figure-of-merit for alternative models. The value of the function for a given model is the figure-of-merit for that model. The criterion value is minimized to choose the best fitting model(s).
Motivated by Akaike's work, Professor Bozdogan developed (1988, 1990, 1994) a new informational complexity criterion called ICOMP ("I" for information and "COMP" for complexity) to choose the best fitting model among competing models. Bozdogan's ICOMP criterion is designed to estimate the loss incurred in modeling, parameter redundancy, parameter stability, and error structure of the models in one criterion function. In model evaluation and selection process, ICOMP trades-off between the "goodness-of-fit" and an entropy measure of the complexity of the covariance matrix of the parameter estimates and the error terms. Also, it trades-off between the "goodness-of-fit" and the accurracy of the parameter estimates via the complexity of the estimated inverse-Fisher information matrix (IFIM) of the model. Empirically, complexity is defined to be a measure of the cohesion or interdependence among the components of a model, not necessarily the number of parameters as in AIC. It is considered an advance over Akaike's work.
This approach to statistics has spawned a new wave of statistical modeling activity previously unattainable with conventional statistical techniques. It has proven very successful for industrial and scientific applications in many diverse fields. Among the numerous applications are: controlling nuclear power plants, paper mills, factory production, detecting brain tumors, breast cancer, causes of heart attacks, predicting earthquakes, enabling image enhancement and signal processing, predicting reliability in software engineering, and dynamically forecasting virtually any economic activity, from state tax revenues to national consumer product demand, just to mention a few.
In general, data analysis is not a onetime activity but rather a dynamic process. Part of the power of informational statistical modeling comes from its ability to compare large numbers of competing statistical models simultaneously. Information criteria assign to each fitted model a comparable scalar value, where this scalar value is a ranking or score that is independent of the units of measurement of the original data. Information criteria balance model complexity with "goodness-of-fit" to select a simple model with highest predictive power in a parsimonious fashion. Comparability and parsimony, therefore, are two major characteristics in informational modeling not found in classical statistical methods.
Professor Bozdogan has been successful in bringing his research activity into the classroom. His graduate courses on statistical modeling have been well received, especially by doctoral students in diverse disciplines, where the informational statistical modeling techniques and attendant computer algorithms have contributed so much to analyzing their data sets and solving their research problems.
Professor Bozdogan, and his team of research colleagues, Professors Peter Bearse and Alan Schlottmann of the CBA, recently won a worldwide competition on research in applied econometrics aimed at forecasting U.S. and Dutch food consumption using these new and novel informational modeling techniques. Such techniques affect public policy in modern society at the local, state, national, and the international level.
Conventional statistical theory regards a model as either true or false. In fact, all models are approximations to an underlying reality and are thus, by definition, false. The informational modeling approach begins data analysis with such a premise, seeking the models that best approximate the real phenomenon at hand for a specific purpose. With this new approach traditional hypothesis testing and table look-up are eliminated by employing the information theoretic principles in model selection. As such, total flexibility and versatility are provided to practitioners, researchers, and students in solving their statistical problems.
Informational statistical modeling is here to stay. Data are there, they need to be analyzed and modeled yielding quicker and more accurate results for knowledge discovery and problem solving in many diverse disciplinesin real time. As L. A. Baxter (1991, p. 356-366) referring to informational statistical modeling in the Journal of the Royal Statistical Society, Volume 154, Part 2, states:
"...this theory is more likely to survive than most, being based on data and common sense rather than dogma."