Functions, Variables, and Matrix Completion: The Mathematical Triad for Modeling All Systems

Understanding transformations on an n-dimensional manifold, in the context of linear and non-linear operations (complex systems).

A simple yet intriguing question struck me: What happens when something enters a system? At its core, a system is a collection of interconnected components that evolves over time and produces specific behaviors. Essentially, I wanted to understand how an input transforms as it passes through a system. Since many systems can be mathematically modeled, we can think of them as functions that act on inputs in defined ways. My curiosity lies in unraveling this transformation — breaking down how inputs evolve within a system, and what that reveals about the system itself. A system can be defined as:

\(F_{out} = f(x_1,x_2,...,x_N), x_i \in R^N\)

where x1,x2,…, x_N are input variables and f is a function acting over the input.

What do we need? Something that can help us understand how the system will transform the input feature space. Maybe a Jacobian, or a Hessian? First, let’s understand what they can offer.

Jacobian Matrix & Eigenvectors

The Jacobian matrix of a transformation at a point on an n-dimensional manifold describes the local linear approximation of the transformation. The entries of a Jacobian matrix are partial derivatives, which capture how each output component responds to changes in each input component.

\(J_{ij} = \frac{\delta f_i} {\delta x_j} \implies J_f = \begin{bmatrix} \frac{\delta f} {\delta x_1} \frac{\delta } {\delta x_2}.....\frac{\delta f} {\delta x_N} \end{bmatrix} \implies J_{f: n \rightarrow m} = \begin{bmatrix} \frac{\delta f_1} {\delta x_1}........\frac{\delta f_1} {\delta x_N} \\ ........\\ ........ \\ ........\\ \frac{\delta f_m} {\delta x_1}........\frac{\delta f_m}{\delta x_N}\end{bmatrix}\)

What can we do with the Jacobian and its eigenvectors?

The Jacobian provides a linear approximation of the transformation near a point. Multiplying an input vector by the Jacobian, J.v, gives the approximate change in the output, revealing how the input is transformed locally.

Now, what are those one or two things that play the larger role in the transformation of an input → eigenvectors and eigenvalues. The eigenvectors of the Jacobian indicate the directions in the input space where the transformation has the most significant effect (or least resistance). The corresponding eigenvalues quantify the scaling or contraction along these directions. So, that’s it, right? We know the catalysts now. Let’s understand this with an example:

• Dominant eigenvectors (those with the largest absolute eigenvalues) highlight the primary directions of transformation or sensitivity.

• If the eigenvalues are positive, negative, or zero, they indicate expansion, contraction, or stability in those directions, respectively.

This is extremely useful for tracking the behavior of the system, especially in dynamical systems or optimization, where eigenvectors can reveal stable/unstable manifolds.

By analyzing the eigenvectors, we can understand how the manifold’s geometry influences the transformation. For instance, in a neural network, the Jacobian of the network’s output with respect to its inputs can reveal how sensitive the output is to input perturbations, which is critical for stability analysis or adversarial robustness.

Did you notice anything about the nature of the transformations? They are all linear because what we’re doing, essentially, is matrix multiplications of linear vectors.

Handling Non-linear Transformations

The Jacobian is inherently a linear tool, as it approximates the transformation locally via a first-order Taylor expansion. For nonlinear operations (e.g., activation functions like ReLU, sigmoid, or tanh in neural networks), the Jacobian only captures the behavior in a small neighborhood around a point. To address the nonlinearities, as we suggested, we can evaluate the Jacobian at multiple points across the manifold to capture local behavior in different regions. This creates a piecewise linear approximation of the nonlinear transformation.

For example, for a neural network with activation functions, the Jacobian varies depending on the input because activation functions like ReLU are piecewise linear. You can compute the Jacobian at different points to understand how the transformation changes across the input space. This approach is common in analyzing neural network dynamics or in techniques like local linear embedding for manifold learning.

Hessian for Nonlinear Insights

The Hessian matrix, which contains second-order partial derivatives and captures the curvature of the transformation. It provides information about the nonlinear behavior that the Jacobian misses.

\((H_f)_{i,j} = \frac{\delta^2 f}{\delta x_i \delta x_j}\)

The Hessian describes how the gradient (captured by the Jacobian) changes, revealing whether the transformation is locally convex, concave, or saddle-shaped. This is critical for understanding nonlinear effects, such as how an activation function bends the input space.

Similar to the Jacobian, the Hessian’s eigenvectors and eigenvalues indicate directions and magnitudes of curvature. Positive eigenvalues suggest convex regions, negative ones suggest concave regions, and a mix indicates saddle points. This is particularly useful in optimization (e.g., analyzing critical points in loss landscapes of neural networks). For highly nonlinear transformations, computing the Hessian at multiple points can help model how the curvature changes across the manifold, providing a second-order approximation that complements the Jacobian’s first-order view.

Matrix Completion

Suppose we don’t have full access to the Jacobian or Hessian (e.g., the network is a black box, or we only observe partial derivatives). There are cases in real-life scenarios where we will have limited measurements, and we can only observe our Jacobian or Hessian for a few points. The goal would be to reconstruct the full Jacobian or Hessian across the manifold if we aim to understand the system’s behavior.

There are some approaches to get out of this muddle, such as the low rank assumption. For neural networks, we can assume that these matrices have low rank due to structured weights or smooth manifolds. Please note that the transformation will be a constrained one in this case. There are many soft imputation algorithms and data-driven completion methods as well. In our case, unlike Jacobians, Hessians are symmetric and higher-order, so completion is more complex. We tend to leverage tensor completion techniques, leveraging symmetry and low-rank structures.

Final Framework: Defining the World with Functions, Variables, and Matrix Completion

We’ll interpret the “world” as a system that maps input to outputs via a transformation defined on an n-dimensional manifold (M) or world. The goal is to understand and predict the output for any input using functions (the transformations (f)), variables (inputs, outputs, and parameters), and matrix completions (to handle incomplete knowledge of the system’s dynamics, e.g., Jacobians or Hessians).

Functions

They represent the system’s transformation (f), which could be a neural network, a physical model, or any process mapping inputs to outputs. The function encodes the rules of the system. If (f) is unknown, we approximate it using data-driven methods (e.g., neural networks) or local/linear approximations via Jacobians and Hessians.

Variables

They are the inputs, outputs, and parameters (e.g., weights in a neural network or physical constants in a model). They define the state and configuration of the system.

• Inputs — points on the manifold, representing the system’s state or observations.

• Outputs — the result of the transformation, e.g., predictions, physical measurements, or transformed coordinates.

• Parameters — variables like weights or biases that define the function, may be partially known or need estimation.

The manifold M constraints the inputs, reflecting the system’s structure (e.g., a low-dimensional subspace in high-dimensional data or a physical constraint like conservation laws).

Matrix Completion

It comes into play when the system’s dynamics (e.g., Jacobians or Hessians) are partially observed or unknown. It allows us to reconstruct the full transformation or its derivatives from incomplete data, enabling us to model the system comprehensively.

An Example

Now, imagine treating activation functions as operators in a Hilbert space, essentially, matrix transformations, but with activation functions baked in.

Everywhere else in a neural network, things are nice and linear. Linear layers? Just linear transformations. So, via eigenvalues, you can easily tell if a transformation is going to expand or contract your feature space. Smooth, elegant, predictable.

But ah — activation functions. These things are nonlinear. No single set of eigenvalues to summarize their behavior across the entire input space. That’s the catch.

So what do we do? We zoom in. We analyze local behavior. Take the derivative that gives us the Jacobian matrix, which acts like a localized linear approximation of our operator (i.e., the activation function). Now, if you look at the eigenvalues of this Jacobian at different points in the input space, you can see how the operator is stretching or shrinking space in that region.

Let’s call the maximum eigenvalue at a point the local radius, the maximum amount of stretch the function applies right there.

• If radius > 1 → space is expanding

• If radius < 1 → space is contracting

• If radius = 0 → space is collapsing

And here’s the punchline: by analyzing these radii, you can predict where and when gradients will likely explode or vanish. That’s not just useful, that’s beautiful.

We’ve juggled functions (the activations), variables (their inputs, outputs, and learned parameters), and leaned on Jacobian-based matrix completion to approximate exactly what we needed.

Summary

The functions encode the system’s rule, whether linear (matrix multiplications) or nonlinear (activation function in neural networks). The variables represent the system’s state (inputs/outputs) and configuration (parameters), allowing us to explore and manipulate the system. The matrix completion handles incomplete knowledge, reconstructing the system’s dynamics (Jacobians/Hessians) from partial data, making the framework robust to real-world simulations.

I feel the triad of functions, variables, and matrix completion is universal because:

• Completeness — any differentiable system can be approximated locally by Jacobians (linear) and Hessians (nonlinear), and globally by combining these across the manifold.

• Flexible — applies to neural networks, physical systems, or abstract manifolds.

• Data-driven — allows us to work with partial observations, common in real-world scenarios like black-box models or sparse measurements.

I am leaving you with a few questions addressing the theoretical foundations of the framework.

Theoretical Questions:

1. Can any system be adequately represented by functions, variables, and their Jacobians/Hessians?

2. How do we define the manifold M for real-world systems?

3. How do nonlinearities impact the framework’s generality?

Computational Questions

4. How do we efficiently compute Jacobians and Hessians for high-dimensional systems?

5. How scalable are matrix completion algorithms for large Jacobians/Hessians?

6. How do we handle the computational costs of evaluating the function (f) across the manifold?

Practical Questions

7. How do we handle incomplete or noisy data in matrix completion?

8. Can the framework generalize across different domains (e.g., AI, physics, biology)?

9. How do we validate the framework’s predictions in real-world scenarios?

10. How do we make the framework real-time or interactive for large systems?

Specific Challenges for Nonlinear Systems

11. How many local approximations (Jacobians/Hessians) are needed to model a highly nonlinear system?

12. Can we approximate nonlinear transformations without computing Hessians?

The framework of functions, variables, and matrix completion is highly scalable in theory, as it can model any differentiable system and handle incomplete data. However, practical scalability requires addressing the above questions, particularly around computational efficiency, handling nonlinearities, and robust completion. By leveraging approximations, sampling, and domain-specific optimizations, the framework can scale to large systems like deep neural networks, physical simulations, or biological models.

~Ashutosh