Model Capacity
The amount of information a model's parameters can hold, which affects how knowledge representations are compressed and integrated. When capacity is limited, frequently occurring and consistent knowledge is represented first.
Timeline (last 90 days)
When model capacity is limited, consistently common and consistent knowledge (low TVS) is compressed into representations first, while the time-fluctuating fine-grained signals carried by high-TVS items may require more parameters and longer training to be fully encoded.
If the data distribution lies on a d-dimensional manifold, the model's effective capacity should be determined by d rather than by the observed dimensionality.
The researchers propose a new supervision scheme that shifts model capacity from 'scene appearance' to 'object dynamics', improving object fidelity, but a significant gap still remains.
'Shifting model capacity from scene appearance to object dynamics' is a reallocation of the supervision signal and can be understood as an injection of inductive bias, but it cannot create information that does not exist in the data out of thin air.
All Facts (4)
When model capacity is limited, consistently common and consistent knowledge (low TVS) is compressed into representations first, while the time-fluctuating fine-grained signals carried by high-TVS items may require more parameters and longer training to be fully encoded.
50%UnverifiedIf the data distribution lies on a d-dimensional manifold, the model's effective capacity should be determined by d rather than by the observed dimensionality.
50%UnverifiedThe researchers propose a new supervision scheme that shifts model capacity from 'scene appearance' to 'object dynamics', improving object fidelity, but a significant gap still remains.
50%Unverified'Shifting model capacity from scene appearance to object dynamics' is a reallocation of the supervision signal and can be understood as an injection of inductive bias, but it cannot create information that does not exist in the data out of thin air.
50%