Measures of Complexity in Complex Adaptive Systems - Prof. Stephanie Forrest, Study notes of Programming Languages

Download Measures of Complexity in Complex Adaptive Systems - Prof. Stephanie Forrest and more Study notes Programming Languages in PDF only on Docsity!

CS 591

Topics in Complex Adaptive Systems

Spring Semester, 2006

Measures of Complexity

Stephanie Forrest

FEC 355E

http://cs.unm.edu/~forrest/cas-class-06.html [email protected] 505-277-

What are Complex Adaptive Systems?

• Collections of agents:
  • Molecules, cells, animals, planets, stars, economic agents.
• Agents interact (locally) with one another and with their environment: - No central controller. - Interactions are nontrivial, I.e. nonlinear. - Chemical reactions, cellular interactions, mating, buy/sell decisions.
• Unanticipated properties often result from the interactions:
  • Immune system responses, flocks of animals, settlement patterns, earthquakes, speculative bubbles and crashes.
• Agents adapt their behavior to other agents and environmental constraints: - Imitation, adaptation, learning.
• System behavior evolves over time:
  • Rules change, unfolding of complex behavior.

Caveat

• Some people believe that there is no general science of complex systems: - “It’s becoming apparent that a theory of complexity in some sense of a great and transforming body of knowledge that applies to a whole range of cases may be untenable.” Sir Robert May (2001)

Understanding Nature and Society Through Computation

• Using information processing methods to learn more about natural systems: - Cognitive science.. - Biological models (e.g., vaccine design, ecological models). - Lattice gas models in physics. - Prisoner’s dilemma in social systems.
• Contrast with computational science:
  • Modeling nature as a mechanical device vs. modeling nature as an information system.
  • E.g., computational biology---genomics and protein folding.

Characteristics of Complex Systems

• What makes a system complex?
  • Nonlinear interactions among components.
  • Multi-scale phenomena.
  • Evolution of underlying components and environments.
• How to measure a system’s complexity?
  • By its unpredictability?
  • By how difficult it is to describe?
    • Length of most concise description.
    • No single model adequate to describe system---the more models that are required, the more complex the system. (Lee Segel)
  • By measuring how long before it halts, if ever? By how long until it repeats itself?
  • Entropy?
  • Multiple levels of organization?
  • Number of interdependencies?
• Is complexity inherent in the system or in our understanding of it?

Representative Measures of Complexity

• Computational complexity (Cook):
  • How long a program runs (or how much memory it uses).
  • Asymptotic.
• Language complexity:
  • Classes of languages that can be computed (recognized) by different kinds of abstract machines.
  • Decidability, computability.
• Logical depth (Bennett).
• Information-theoretic approaches:
  • Algorithmic Complexity (Solomonoff, Komogorov, and Chaitin):
    • Length of the shortest program that can produce the phenomenon.
    • Kolmogorov complexity.
  • Mutual information (many authors)
• Thermodynamic depth (Lloyd and Pagels)
• Effective complexity (Gell-Mann and Lloyd)

Computational Complexity Classes

(from Papadimitriou, 1994)

Algorithmic Complexity (AC)

(also known as Kolmogorov-Chaitin complexity)

• The Kolomogorov-Chaitin complexity K(x) is the length, in bits, of the smallest program that when run on a Universal Turing Machine outputs (prints) x and then halts.
• Example: What is K(x) where x is the first 10 even natural numbers? Where x is the first 5 million even natural numbers?
• Possible representations:
  • 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, … (2n - 2)
  • for (j = 0; j < n: j++) printf(“%d\n”, j * 2);
• How many bits?
  • Alternative 1: O(n log n)
  • Alternative 2: K(x) = O(log n)
• Two problems:
  • Calculation of K(x) depends on the machine we have available (e.g., what if we have a machine with an instruction “print the first 10 even natural numbers”?)
  • In general, it is an uncomputable problem to determine K(x) for arbitrary x.

Structural Complexity

Randomness hμ Randomness hμ

Algorithmic Complexity Measures of Complexity that Capture Properties Distinct from Randomness

• Measures of randomness do not capture pattern, structure, correlation, or organization.
• Mutual information, Wolfram’s CA classification.
• The “edge of chaos.”

Logical Depth

• Bennett 1986;1990:
  • The Logical depth of x is the run time of the shortest program that will cause a UTM to produce x and then halt.
  • Logical depth is not a measure of randomness; it is small both for trivially ordered and random strings.
• Drawbacks:
  • Uncomputable.
  • Loses the ability to distinguish between systems that can be described by computational models less powerful than Turing Machines (e.g., finite-state machines).

Detour into Information Theory

• Shannon Entropy H to measure basic information capacity:
  • For a random variable X with a probability mass function p(x), the entropy of X is defined as:
  • Entropy is measured in bits.
  • H measures the average uncertainty in the random variable.
• Example 1:
  • Consider a random variable with uniform distribution over 32 outcomes.
  • To identify an outcome, we need a label that takes on 32 different values, e.g., 5-bit strings. H ( X)=−∑p(x)log 2 p(x ) log 32 5 bits 32

log 32

( ) ()log ( ) 32 1 32 1 = −∑ =−∑ = = i = i= H X p i p i

What is a Random Variable?

• A function defined on a sample space.
  • Should be called “random function.”
  • Independent variable is a point in a sample space (e.g., the outcome of an experiment).
• A function of outcomes, rather than a single given outcome.
• Probability distribution of the random variable X:
• Example:
  • Toss 3 fair coins.
  • Let X denote the number of heads appearing.
  • X is a random variable taking on one of the values (0,1,2,3).
  • P{X=0} = 1/8; P{X=1} = 3/8; P{X=2} = 3/8; P{X=3} = 1/8.

P{ X = xj }= f(xj) (j= 1,2,...)

Detour into Information Theory cont.

• More generally,
  • the entropy of a random variable is a lower bound on the average number of bits required to represent the random variable.
• The uncertainty (complexity) of a random variable can be extended to define the descriptive complexity of a single string.
• E.g., Kolmogorov (or algorithmic) complexity is the length of the shortest computer program that prints out the string.
• Entropy is the uncertainty of a single random variable.
• Conditional entropy is the entropy of a random variable given another random variable.

Mutual Information

• Measures the amount of information that one random variable contains about another random variable. - Mutual information is a measure of reduction of uncertainty due to another random variable. - That is, mutual information measures the dependence between two random variables. - It is symmetric in X and Y, and is always non-negative.
• Recall: Entropy of a random variable X is H(X).
• Conditional entropy of a random variable X given another random variable Y = H(X | Y).
• The mutual information of two random variables X and Y is: = − = ∑ x y p x p y p x y I X Y H X H X Y p x y , ( ) ( )

( , ) ( ) ( | ) ( , ) log