.. _sec_deferred_init:
Deferred Initialization
=======================
So far, it might seem that we got away with being sloppy in setting up
our networks. Specifically, we did the following unintuitive things,
which might not seem like they should work:
- We defined the network architectures without specifying the input
dimensionality.
- We added layers without specifying the output dimension of the
previous layer.
- We even "initialized" these parameters before providing enough
information to determine how many parameters our models should
contain.
You might be surprised that our code runs at all. After all, there is no
way the deep learning framework could tell what the input dimensionality
of a network would be. The trick here is that the framework *defers
initialization*, waiting until the first time we pass data through the
model, to infer the sizes of each layer on the fly.
Later on, when working with convolutional neural networks, this
technique will become even more convenient since the input
dimensionality (i.e., the resolution of an image) will affect the
dimensionality of each subsequent layer. Hence, the ability to set
parameters without the need to know, at the time of writing the code,
what the dimensionality is can greatly simplify the task of specifying
and subsequently modifying our models. Next, we go deeper into the
mechanics of initialization.
Instantiating a Network
-----------------------
To begin, let us instantiate an MLP.
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from mxnet import np, npx
from mxnet.gluon import nn
npx.set_np()
def get_net():
net = nn.Sequential()
net.add(nn.Dense(256, activation='relu'))
net.add(nn.Dense(10))
return net
net = get_net()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import tensorflow as tf
net = tf.keras.models.Sequential([
tf.keras.layers.Dense(256, activation=tf.nn.relu),
tf.keras.layers.Dense(10),
])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(net.collect_params)
print(net.collect_params())
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
256, Activation(relu))
(1): Dense(-1 -> 10, linear)
)>
sequential0_ (
Parameter dense0_weight (shape=(256, -1), dtype=float32)
Parameter dense0_bias (shape=(256,), dtype=float32)
Parameter dense1_weight (shape=(10, -1), dtype=float32)
Parameter dense1_bias (shape=(10,), dtype=float32)
)
Note that while the parameter objects exist, the input dimension to each
layer is listed as -1. MXNet uses the special value -1 to indicate that
the parameter dimension remains unknown. At this point, attempts to
access ``net[0].weight.data()`` would trigger a runtime error stating
that the network must be initialized before the parameters can be
accessed. Now let us see what happens when we attempt to initialize
parameters via the ``initialize`` function.
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net.initialize()
net.collect_params()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
sequential0_ (
Parameter dense0_weight (shape=(256, -1), dtype=float32)
Parameter dense0_bias (shape=(256,), dtype=float32)
Parameter dense1_weight (shape=(10, -1), dtype=float32)
Parameter dense1_bias (shape=(10,), dtype=float32)
)
As we can see, nothing has changed. When input dimensions are unknown,
calls to initialize do not truly initialize the parameters. Instead,
this call registers to MXNet that we wish (and optionally, according to
which distribution) to initialize the parameters.
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
[net.layers[i].get_weights() for i in range(len(net.layers))]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[[], []]
Note that each layer objects exist but the weights are empty. Using
``net.get_weights()`` would throw an error since the weights have not
been initialized yet.
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = np.random.uniform(size=(2, 20))
net(X)
net.collect_params()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
sequential0_ (
Parameter dense0_weight (shape=(256, 20), dtype=float32)
Parameter dense0_bias (shape=(256,), dtype=float32)
Parameter dense1_weight (shape=(10, 256), dtype=float32)
Parameter dense1_bias (shape=(10,), dtype=float32)
)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = tf.random.uniform((2, 20))
net(X)
[w.shape for w in net.get_weights()]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[(20, 256), (256,), (256, 10), (10,)]
.. raw:: html
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html