How Stiffness Constraints and Information Entropy Can Work Against Deep Learning, with Further Implications: An Interactive Illustration

Many difficulties in deep learning can be described as propagation problems. The deeper the network, the more layers hidden states and gradients must traverse, and the more likely local errors are to be amplified through repeated composition. Exploding gradients, vanishing gradients, inter-layer oscillation, feature concentration and representation collapse appear to be distinct phenomena, yet they share an underlying question: can information travel through a deep network in a stable form?

The FE-E, or Finite Element-Entropy, method arose from this question. It treats network depth as a one-dimensional mesh and the gradients at each layer as a vector field on the mesh nodes, then borrows the finite-element concepts of stiffness and mass together with information entropy to constrain propagation. Formally, the idea is natural. The stiffness term suppresses abrupt changes between adjacent layers, the mass term constrains total gradient energy, and information entropy prevents energy from concentrating in only a few layers or feature directions.

Judged from the equations alone, FE-E appears likely to make training more stable and efficient. The experiment produced a more revealing result: a constraint may improve the local shape of propagation without improving learning itself. By making the system too stable, it may even postpone the genuine phase transition in learning.

1. What the Constraints Were Intended to Do

Let the backpropagated gradient at layer l be g_l. The finite-element stiffness energy can be written as:

It penalises sharp changes in gradients across layers. If the gradient at one layer suddenly grows, rotates rapidly or develops high-frequency oscillation along the depth axis, the stiffness energy rises.

The mass energy is:

It describes the total gradient energy along the depth dimension. Keeping it near a reference value can suppress overall explosion or decay.

The depth-wise information entropy is defined from the proportion of gradient energy

as:

Entropy that is too low means that the gradient is concentrated in a small number of layers. Entropy that is too high may mean that every layer is being forced to play nearly the same role. The original FE-E proposal was to keep entropy within a suitable range while maintaining gradient magnitude and continuity between layers.

This proposal contains an easily overlooked assumption: smoother propagation, more stable energy and a more even distribution will produce better learning. The later experiments show that this assumption does not always hold.

2. The Experiment Produced the Opposite Result

In a set of MLX experiments using 128 layers, width 32 and four attention heads, the baseline combined AdamW with Gradient Smoothing. Each FE-E intervention was limited to 5 per cent of the task-gradient norm and applied at freeze frequencies of 1, 3 and 5 per cent. The experiments covered four environments: no propagation noise, high-frequency inter-layer disturbance, global amplification of residual energy and concentration of energy in middle layers. The endpoint was token accuracy at or above 99 per cent in three consecutive validations.

Across 12 pairings between an FE-E dose and a SHAM baseline in the same environment, only three FE-E runs reached the endpoint earlier, while the other nine were slower. In the normal environment, pure GS was confirmed at 1,536 steps, while FE-E at 1, 3 and 5 per cent required 1,600, 1,632 and 1,888 steps respectively. In the absence of a clear propagation abnormality, every tested dose was therefore a negative optimisation.

One local positive result appeared in the middle-layer energy-concentration environment. GS in the same environment required 1,952 steps, while 3 per cent FE-E required 1,824, an improvement of 128 steps. Yet this was still slower than the 1,536 steps needed by GS in the normal environment. It was also a conditional result from a single seed, selected from 12 pairings, and cannot be described as an established optimum intervention rate.

The global energy-amplification environment was more instructive. Before the phase transition, pure GS had a training-loss slope of about -0.00062 per 100 steps, while 5 per cent FE-E reached about -0.01386 and appeared to decline much faster. Yet pure GS confirmed 99 per cent accuracy at step 1,440, whereas 5 per cent FE-E did not do so until step 1,600. The local loss slope became dramatically steeper, but final convergence arrived 160 steps later.

The computational cost also matters. On this small model, the median duration of an FE-E intervention step was about 2.24 times that of a normal step. Across the experiments, the mean cosine similarity between the FE-E gradient and the task gradient was negative. FE-E was generally not continuing along the task objective, but spending part of the optimisation capacity resisting the task gradient in exchange for a more orderly propagation pattern.

3. Why a Steeper Slope Did Not Produce Earlier Convergence

The time required to reach 99 per cent accuracy can be decomposed approximately as:

T_plateauis the time the model remains in the low-accuracy region,T_transitionis the time required to jump from low to high accuracy after entering the phase transition, andT_confirmcomes from consecutive validation confirmations. The short-term loss slope mainly reflects local descent around the phase transition, but says nothing about when the model enters that transition.

In the global energy-amplification experiment, 5 per cent FE-E did shorten the width of the transition from 10 to 90 per cent, from 96 steps to 64. But it delayed the sustained crossing of 10 per cent from step 1,280 to step 1,440. The model spent 160 additional steps before learning truly began, and the 32 steps saved within the transition could not compensate for that delay.

This exposes the ambiguity in saying that the descent is faster. A model may move faster along a direction it has already found, yet find that direction later. It may also lower cross-entropy by increasing its confidence on easy tokens or tokens it already predicts correctly, without pushing the remaining difficult tokens across the decision boundary. Continuous loss and a discrete phase transition in accuracy are therefore not the same measurement.

The short-term slope in the present experiment has another methodological limitation. It is calculated retrospectively over each trajectory's own pre-transition window, not as the causal difference before and after an FE-E intervention at the same model state. Different methods correspond to different absolute time intervals. The slope can describe a trajectory, but should not serve directly as a gating threshold.

4. Why Stiffness and Entropy Can Work Against Learning

The stiffness term suppresses high-frequency variation between layers, but high-frequency variation is not necessarily noise. Different layers in a deep network perform different functions. A large update occurring first in particular layers may be necessary for creating a new representation, redistributing functions or crossing an optimisation barrier. An overly strong stiffness constraint can mistake necessary structural reorganisation for unstable propagation.

Information entropy has the same double edge. Excessive gradient concentration may mean that a few layers are carrying all the pressure of learning, but it may also mean that the model is developing effective functional specialisation. An entropy constraint can determine whether a distribution is concentrated. Concentration alone cannot tell it whether that pattern is collapse or productive focus.

Mass energy keeps the overall magnitude stable, but learning may not require constant energy. Some phase transitions may depend on brief gradient pulses. If the reference energy cannot distinguish harmful explosion from a beneficial leap, the mass term suppresses both.

FE-E is therefore not a neutral mathematical repair tool. It encodes explicit structural preferences: continuity, stability and limited concentration. The constraint can yield a net benefit only when an abnormality in the task violates those preferences and that violation is genuinely harmful. Otherwise, it merely pushes the model towards a more orderly state that is not necessarily better at learning.

5. From Controller to Observer

These results do not mean that stiffness, mass and entropy have no value. They suggest that their role should change. Instead of adding FE-E to the gradient by default, use these quantities first as observations of propagation state:

Normalised stiffness describes abrupt changes between layers. Mass energy detects overall explosion or decay. Depth-wise entropy reveals whether propagation remains concentrated over time. Observation can be performed with gradients stopped, without calculating higher-order derivatives of the FE-E regulariser with respect to the parameters, making it much cheaper and less intrusive.

An observer must answer a stricter question than whether a number looks abnormal. Does the state persist? Can it predict worsening loss, uncontrolled gradients or delayed phase transition tens of steps later? What is its false-positive rate? Intervention is justified only when the indicator has prospective predictive power.

Different abnormalities should not receive the same treatment. For energy explosion, first try gradient clipping or a lower learning rate. For high-frequency abrupt changes, try brief smoothing. Only persistent concentration of energy between layers may justify a trial window with a small FE-E dose. If the validation set, difficult-token margins or precursors of the phase transition do not improve after intervention, stop or roll back. FE-E then changes from a controller that continuously corrects the model into a sensor that helps decide whether the model needs assistance.

6. From Deep Networks to Cognition and Education

If a Transformer is compared to a multi-stage system of cognitive processing, the FE-E experiment also offers an educational insight. The analogy does not claim that network layers correspond to particular brain regions, or that artificial gradients equal neural activity. It treats both as processes in which information passes through multiple stages and gradually takes shape under limited resources.

Within this analogy, training loss resembles the current degree of error in practice, the short-term loss slope resembles the rate at which fluency improves, and the phase transition in accuracy more closely resembles the point at which a learner truly grasps a rule and forms a transferable concept. A stiffness constraint resembles a demand for continuous, orderly cognitive steps. An information-entropy constraint resembles an attempt to prevent attention and strategy from becoming too concentrated. An FE-E intervention resembles a teacher's prompt, scaffolding or metacognitive control.

The counter-effect in the experiment then acquires a clear educational meaning. A learning process that looks smoother does not necessarily produce understanding sooner. Extensive hints, fixed steps and repeated exercises can quickly reduce immediate errors while shrinking the space available for trial and error, changes of strategy and conceptual reorganisation. A student may perform more fluently yet show no better transfer when the hints are removed or the problem type changes.

Nor should all high-frequency variation be treated as harmful. Hesitation, repeated attempts and jumps between strategies are sometimes not cognitive instability, but a necessary stage in exploring new structure. The real difficulty of educational intervention is not eliminating variation, but distinguishing productive exploration from persistent impasse.

A sound educational extension of FE-E is therefore not the continuous disciplining of a learner, but a framework for metacognitive observation. It asks whether errors remain concentrated in one concept, cognitive load stays excessive, strategies collapse into a single path, or difficulty persists long enough to show that help is needed. The goal of intervention should not merely be faster answers on the current problem. It should examine unaided mastery, transfer tests, delayed retention and dependence on hints.

A testable educational experiment could compare three approaches: hints at a fixed frequency, no hints, and adaptive hints triggered by an observer. If adaptive hints improve only the practice-phase slope without improving transfer and delayed retention, they repeat the problem of the FE-E optimiser. The observational framework has educational value only if it shortens the time to genuine mastery while reducing dependence on hints.

7. Conclusion

FE-E originally sought to stabilise gradient propagation with finite elements and prevent information concentration through entropy. The experiment did not support the strong claim that it is a general-purpose optimiser. It revealed a broader problem instead: any stability constraint can mistake a beneficial non-stationary process for a fault.

Stability is not synonymous with learning, and smoothness is not sufficient for understanding. This is true of deep networks and may also be true of educational intervention. What matters is not keeping every local trajectory orderly, but determining which variation is damaging learning and which variation is giving birth to new structure.

The value of stiffness, energy and entropy may therefore lie not in deciding how the system should change, but in helping us see how it is changing. When a constraint becomes an observation rather than a command, its question changes from how instability can be eliminated to when intervention is necessary and when exploration should be allowed. This may be the most important lesson of the FE-E experiment.