.. _sec_pooling:
Pooling
=======
Often, as we process images, we want to gradually reduce the spatial
resolution of our hidden representations, aggregating information so
that the higher up we go in the network, the larger the receptive field
(in the input) to which each hidden node is sensitive.
Often our ultimate task asks some global question about the image, e.g.,
*does it contain a cat?* So typically the units of our final layer
should be sensitive to the entire input. By gradually aggregating
information, yielding coarser and coarser maps, we accomplish this goal
of ultimately learning a global representation, while keeping all of the
advantages of convolutional layers at the intermediate layers of
processing.
Moreover, when detecting lower-level features, such as edges (as
discussed in :numref:`sec_conv_layer`), we often want our
representations to be somewhat invariant to translation. For instance,
if we take the image ``X`` with a sharp delineation between black and
white and shift the whole image by one pixel to the right, i.e.,
``Z[i, j] = X[i, j + 1]``, then the output for the new image ``Z`` might
be vastly different. The edge will have shifted by one pixel. In
reality, objects hardly ever occur exactly at the same place. In fact,
even with a tripod and a stationary object, vibration of the camera due
to the movement of the shutter might shift everything by a pixel or so
(high-end cameras are loaded with special features to address this
problem).
This section introduces *pooling layers*, which serve the dual purposes
of mitigating the sensitivity of convolutional layers to location and of
spatially downsampling representations.
Maximum Pooling and Average Pooling
-----------------------------------
Like convolutional layers, *pooling* operators consist of a fixed-shape
window that is slid over all regions in the input according to its
stride, computing a single output for each location traversed by the
fixed-shape window (sometimes known as the *pooling window*). However,
unlike the cross-correlation computation of the inputs and kernels in
the convolutional layer, the pooling layer contains no parameters (there
is no *kernel*). Instead, pooling operators are deterministic, typically
calculating either the maximum or the average value of the elements in
the pooling window. These operations are called *maximum pooling* (*max
pooling* for short) and *average pooling*, respectively.
In both cases, as with the cross-correlation operator, we can think of
the pooling window as starting from the upper-left of the input tensor
and sliding across the input tensor from left to right and top to
bottom. At each location that the pooling window hits, it computes the
maximum or average value of the input subtensor in the window, depending
on whether max or average pooling is employed.
.. _fig_pooling:
.. figure:: ../img/pooling.svg
Maximum pooling with a pooling window shape of :math:`2\times 2`. The
shaded portions are the first output element as well as the input
tensor elements used for the output computation:
:math:`\max(0, 1, 3, 4)=4`.
The output tensor in :numref:`fig_pooling` has a height of 2 and a
width of 2. The four elements are derived from the maximum value in each
pooling window:
.. math::
\max(0, 1, 3, 4)=4,\\
\max(1, 2, 4, 5)=5,\\
\max(3, 4, 6, 7)=7,\\
\max(4, 5, 7, 8)=8.\\
A pooling layer with a pooling window shape of :math:`p \times q` is
called a :math:`p \times q` pooling layer. The pooling operation is
called :math:`p \times q` pooling.
Let us return to the object edge detection example mentioned at the
beginning of this section. Now we will use the output of the
convolutional layer as the input for :math:`2\times 2` maximum pooling.
Set the convolutional layer input as ``X`` and the pooling layer output
as ``Y``. Whether or not the values of ``X[i, j]`` and ``X[i, j + 1]``
are different, or ``X[i, j + 1]`` and ``X[i, j + 2]`` are different, the
pooling layer always outputs ``Y[i, j] = 1``. That is to say, using the
:math:`2\times 2` maximum pooling layer, we can still detect if the
pattern recognized by the convolutional layer moves no more than one
element in height or width.
In the code below, we implement the forward propagation of the pooling
layer in the ``pool2d`` function. This function is similar to the
``corr2d`` function in :numref:`sec_conv_layer`. However, here we have
no kernel, computing the output as either the maximum or the average of
each region in the input.
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from mxnet import np, npx
from mxnet.gluon import nn
from d2l import mxnet as d2l
npx.set_np()
def pool2d(X, pool_size, mode='max'):
p_h, p_w = pool_size
Y = np.zeros((X.shape[0] - p_h + 1, X.shape[1] - p_w + 1))
for i in range(Y.shape[0]):
for j in range(Y.shape[1]):
if mode == 'max':
Y[i, j] = X[i: i + p_h, j: j + p_w].max()
elif mode == 'avg':
Y[i, j] = X[i: i + p_h, j: j + p_w].mean()
return Y
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import torch
from torch import nn
from d2l import torch as d2l
def pool2d(X, pool_size, mode='max'):
p_h, p_w = pool_size
Y = torch.zeros((X.shape[0] - p_h + 1, X.shape[1] - p_w + 1))
for i in range(Y.shape[0]):
for j in range(Y.shape[1]):
if mode == 'max':
Y[i, j] = X[i: i + p_h, j: j + p_w].max()
elif mode == 'avg':
Y[i, j] = X[i: i + p_h, j: j + p_w].mean()
return Y
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import tensorflow as tf
def pool2d(X, pool_size, mode='max'):
p_h, p_w = pool_size
Y = tf.Variable(tf.zeros((X.shape[0] - p_h + 1, X.shape[1] - p_w +1)))
for i in range(Y.shape[0]):
for j in range(Y.shape[1]):
if mode == 'max':
Y[i, j].assign(tf.reduce_max(X[i: i + p_h, j: j + p_w]))
elif mode =='avg':
Y[i, j].assign(tf.reduce_mean(X[i: i + p_h, j: j + p_w]))
return Y
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = np.array([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])
pool2d(X, (2, 2))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[4., 5.],
[7., 8.]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = torch.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])
pool2d(X, (2, 2))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[4., 5.],
[7., 8.]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = tf.constant([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])
pool2d(X, (2, 2))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d(X, (2, 2), 'avg')
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[2., 3.],
[5., 6.]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d(X, (2, 2), 'avg')
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[2., 3.],
[5., 6.]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d(X, (2, 2), 'avg')
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = np.arange(16, dtype=np.float32).reshape((1, 1, 4, 4))
X
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[[[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.],
[12., 13., 14., 15.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = torch.arange(16, dtype=torch.float32).reshape((1, 1, 4, 4))
X
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[[[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.],
[12., 13., 14., 15.]]]])
.. raw:: html
.. raw:: html
It is important to note that tensorflow prefers and is optimized for
*channels-last* input.
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = tf.reshape(tf.range(16, dtype=tf.float32), (1, 4, 4, 1))
X
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2D(3)
# Because there are no model parameters in the pooling layer, we do not need
# to call the parameter initialization function
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[[[10.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2d(3)
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[[[10.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = tf.keras.layers.MaxPool2D(pool_size=[3, 3])
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2D(3, padding=1, strides=2)
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[[[ 5., 7.],
[13., 15.]]]])
Of course, we can specify an arbitrary rectangular pooling window and
specify the padding and stride for height and width, respectively.
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2D((2, 3), padding=(0, 1), strides=(2, 3))
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[[[ 5., 7.],
[13., 15.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2d(3, padding=1, stride=2)
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[[[ 5., 7.],
[13., 15.]]]])
Of course, we can specify an arbitrary rectangular pooling window and
specify the padding and stride for height and width, respectively.
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2d((2, 3), stride=(2, 3), padding=(0, 1))
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[[[ 5., 7.],
[13., 15.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
paddings = tf.constant([[0, 0], [1,0], [1,0], [0,0]])
X_padded = tf.pad(X, paddings, "CONSTANT")
pool2d = tf.keras.layers.MaxPool2D(pool_size=[3, 3], padding='valid',
strides=2)
pool2d(X_padded)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Of course, we can specify an arbitrary rectangular pooling window and
specify the padding and stride for height and width, respectively.
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
paddings = tf.constant([[0, 0], [0, 0], [1, 1], [0, 0]])
X_padded = tf.pad(X, paddings, "CONSTANT")
pool2d = tf.keras.layers.MaxPool2D(pool_size=[2, 3], padding='valid',
strides=(2, 3))
pool2d(X_padded)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = np.concatenate((X, X + 1), 1)
X
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[[[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.],
[12., 13., 14., 15.]],
[[ 1., 2., 3., 4.],
[ 5., 6., 7., 8.],
[ 9., 10., 11., 12.],
[13., 14., 15., 16.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = torch.cat((X, X + 1), 1)
X
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[[[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.],
[12., 13., 14., 15.]],
[[ 1., 2., 3., 4.],
[ 5., 6., 7., 8.],
[ 9., 10., 11., 12.],
[13., 14., 15., 16.]]]])
.. raw:: html
.. raw:: html
Note that this will require a concatenation along the last dimension for
TensorFlow due to the channels-last syntax.
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X = tf.concat([X, X + 1], 3) # Concatenate along `dim=3` due to channels-last syntax
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2D(3, padding=1, strides=2)
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[[[ 5., 7.],
[13., 15.]],
[[ 6., 8.],
[14., 16.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
pool2d = nn.MaxPool2d(3, padding=1, stride=2)
pool2d(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[[[ 5., 7.],
[13., 15.]],
[[ 6., 8.],
[14., 16.]]]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
paddings = tf.constant([[0, 0], [1,0], [1,0], [0,0]])
X_padded = tf.pad(X, paddings, "CONSTANT")
pool2d = tf.keras.layers.MaxPool2D(pool_size=[3, 3], padding='valid',
strides=2)
pool2d(X_padded)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Note that the output for the tensorflow pooling appears at first glance
to be different, however numerically the same results are presented as
MXNet and PyTorch. The difference lies in the dimensionality, and
reading the output vertically yields the same output as the other
implementations.
.. raw:: html
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html