\[\newcommand{\eq}[1]{\begin{align*}#1\end{align*}}\]

Approach

 

 

State Space Reconstruction of Time Series

 

  Example: Sinus curve as signal
 

 

State Space Function
Input vector, dimensions, and time lag

      f.state.space = function(vec, dim = 2, lag = 1){
            lag.seq = seq(from = 0, by = lag, length.out = dim)
            #print(lag.seq)
            df = data.frame(vec, sapply(lag.seq ,function(x) lead(vec, n = x) ))
            df
      }

 

 

State Space
Dimensions = 3 (X1, X2, X3), Time Lag = 4 | vec = original vector, and is equal to X1
vec X1 X2 X3
0.0000000 0.0000000 0.1986693 0.3894183
0.0499792 0.0499792 0.2474040 0.4349655
0.0998334 0.0998334 0.2955202 0.4794255
0.1494381 0.1494381 0.3428978 0.5226872
0.1986693 0.1986693 0.3894183 0.5646425

 

  Visualization

 

 

 

Embedded Dimenstion and Time Lag

Embedded Dimension:

Method False Nearest Neighbor
The rate of separation can be calculated by taking the ration of the difference in the distanced between the vectors at dimension M and dimension m+1 to the distance of these two vectors at dimension. If this ration is beyond a certain threshold value called \(R_{tol} = 15\), the vector is a false neighbor. This threshold is arbitrary. Ararbanel 1996 suggests 15. \[\eq{ R_{tol} &> \frac{|| \hat{V}(t) - \hat{V}^{NN}(t)|| - || {V}(t) - {V}^{NN}(t)||}{|| {V}(t) - {V}^{NN}(t)||} \\ R_{tol} &= 15 }\]
For data data set with short length, Kennth at al (1992) presented an alternative equation to find a false nearest neighbor using and threshold \(A_{tol} = 2\) \[\eq{ A_{tol} &> \frac{|| \hat{V}(t) - \hat{V}^{NN}(t)|| - || {V}(t) - {V}^{NN}(t)||}{\sigma} \\ A_{tol} &= 2 }\]

Time Lag and Autocorrelation

Autocorrelation
The time lag can be selected when the autocorrelation drops to 1/e or 0. However, the autocorrelation function measures linear dependecny and not nonlinear dependency among data points, something that is critical. \

\[\eq{ r(k) = \frac{\sum^{N-k}_{i = 1}(x_i - \bar x)(x_{i+k} - \bar x)}{\sum^{N}_{i=1}(x_i - \bar x)^2} }\]

Nonlinear Dependency
Average mutual information can be thought of as the nonlinear version of the autocorrelation function. We select the first minimum of the average mutual information as the time lag.
\[\eq{ I(k) = \sum^N_{t=1}P(X_t, X_{t+1}) \ log_x \frac{P(X_t, X_{t+1})}{P(X_t) \ P(X_{t+1})} }\]

vec X1 X2 X3
0.0000000 0.0000000 0.7173561 0.9995736
0.3894183 0.3894183 0.9320391 0.9092974
0.7173561 0.7173561 0.9995736 0.6754632
0.9320391 0.9320391 0.9092974 0.3349882
0.9995736 0.9995736 0.6754632 -0.0583741
0.9092974 0.9092974 0.3349882 -0.4425204
0.6754632 0.6754632 -0.0583741 -0.7568025
0.3349882 0.3349882 -0.4425204 -0.9516021
-0.0583741 -0.0583741 -0.7568025 -0.9961646
-0.4425204 -0.4425204 -0.9516021 -0.8834547
-0.7568025 -0.7568025 -0.9961646 -0.6312666
-0.9516021 -0.9516021 -0.8834547 -0.2794155
-0.9961646 -0.9961646 -0.6312666 NA
-0.8834547 -0.8834547 -0.2794155 NA
-0.6312666 -0.6312666 NA NA
-0.2794155 -0.2794155 NA NA

Autocorrelation r(k) vs k(lag)

Autocorrelation with Force Plate Data

k (lags) rk = 0 or 1/exp(1)
1413 1412 -0.0003196
230 229 0.3675089

Nonlinear Dependency

Entropy, Mutual Information

The following equations show how mutual information between two random variables or strings can be described and how to quantify the amount of information they know about each other.

String Context \[\eq{ I(\sigma ; \tau ) &= K(\sigma) + K(\tau) - K(\sigma, \tau)\\ K(\sigma) + K(\tau) &= I(\sigma : \tau ) + K(\sigma, \tau)\\ }\] String context, contexgt of complexity, mutual information of two strings, Information of sigma plus the information content of tau and subtract the information content of the pair of the two, which you can only retrieve when you use both variables. If they sigma and tau do not have information about each other then the information of the pair is the same as komplexity of tau plus the komplexity of sigma

Probabilistic Context

\[\eq{ I(X;Y) &= H(X) + H(Y) - H(X,Y) }\]

Entropy of X plus entropy of Y minus joint entropy of X and Y. To define mutual information Kolbe Likely divergence, by asking how far away is the joint distribution from the product distribution. If the variables are independent they will share no mutual information.

Let P and Qu be two probability distributions on a finite set A.

Definition: The Kullback-Leibler (KL0) divergence between P and Q is given as \[\eq{ D(P||Q) = \sum_{a \in A} P(a) \ log \frac{P(a)}{Q(a)} = \mathbb{E}_P \ log\frac{P}{Q} }\]

Divergence between P and Q is the sum of probabilities of P(a) multiplied by the log P(a) of a divided by Q of a. If and P(a) and Q(a) are identical then log(1) = 0, so the divergence (or distance) for identical probability distributions is zero.

Convention: log(0/0) = 0, \(\infty\) = p log(p/0)

Is the divergence non-negative D(P||Q) >= Q?

Measuring Mutual Information via KL-Divergence

Let X,Y be two A-valued random variables with distributions P(x), P(y) respectively, and joint distribution P(x,y). Gage the distance we use the Kullback-Leibler divergence forumla and measure the divergence of the distributions P(x), P(y), and the joint distribution of P(x,y)

\[\eq{ I(X;Y) &= D(P(x,y) || P(x) P(y))\\ &= \sum_{a,b \in A} P(a,b) \ log \frac{P(a,b)}{P(a) \ P(b)} \\ &=\mathbb{E}p_{(x,y)} log \frac{P(X,Y)}{P(X)P(Y)} \\ }\]  

 

Example:

X,Y independent:

\[\eq{ &P(x,y) = P(x) P(y) = 0 \\ &-> D(P(x) P(y) || P(x) P(y) ) = 0 \ }\]

2) X=Y:

\[\eq{ I(x,x) &= \sum_{a \in A} P(a) \ log \frac{Pa)}{P(a)^2}\\ &= \sum P(a) \ log \frac{1}{P(a)}\\ }\]

Calculating Mutual Information

Joint and Conditional Probability

\[\eq{ I(X;Y) &= D(P(x,y) || P(x) P(y))\\ I(X;Y) &= \sum_{a,b \in A} P(a,b) \ log \frac{P(a,b)}{P(a) \ P(b)} \\ }\] Example: k = 2

\[\eq{ I_{k = 2} = \sum^N_{i=1} \ P(x_i, x_{i+2}) \ log_2 \ \frac{P(x_i, x_{i+2})}{P(x_i)P(x_{i+2})} = I_{1,3} + I_{2,4} + I_{3,5} + ... + I_{N-2, N} }\]

with \[{X = {0,1,0,0,1,0,1,1,0,0}}\]

Joint Probability
yi xi freq Py.x
0 0 2 0.250
0 0 2 0.250
1 0 3 0.375
1 0 3 0.375
Conditional Probability
yi xi Py.x
0 0 0.4
0 0 0.4
1 0 0.6
1 0 0.6

Mutual Information with R’s mutinformation()

Function Calculating PX, PY, P(X|Y), P(Y|X)
Contingency table showing joint P(X= x, Y = y) and marginal (P(X = x), P(Y = y)) probability.

Discretizing the variables

Since the the cut-off filter is set to 10 Hz, it seems reasonable to discretize.

Addendum

Generate a time series using the following equations. Reconstruct the statespace for X(t) using the embedding dimension = 2 and time lag = 1. Identify the state vectors 10 and 20. Plot the two-dimensional state-space plot for this time series:

X(t)

\[\begin{aligned} \frac{dx}{dt} &= -10x + 10y\\ \frac{dy}{dt} &= 28x - y - xy\\ \frac{dz}{dt} &= xy - 2z\\ \end{aligned}\]

Lorenz Equation

##      time         X        Y        Z
## [1,] 0.00 1.0000000 1.000000 1.000000
## [2,] 0.01 0.9848912 1.012567 1.259918
## [3,] 0.02 0.9731148 1.048823 1.523999
## [4,] 0.03 0.9651593 1.107207 1.798314
## [5,] 0.04 0.9617377 1.186866 2.088545
## [6,] 0.05 0.9638068 1.287555 2.400161

Fx.N 2 Dimensions

Fx.N 3 Dimensions