[EECS 498] Assignment 3. Convolutional Networks...(1)

본 포스팅은 Michigan Univ.의 EECS 498 강의를 수강하면서 공부한 내용을 정리하는 포스팅입니다.

---

https://jinwoo-jung.tistory.com/135

[EECS 498] Assignment 3. Fully Connected Networks...(2)

---

Deep Fully Connected Networks에서 나아가 Convolutional Networks로 확장시켜보자.

 

Convolutional layer

Convolutional layer : forward

def forward(x, w, b, conv_param):
    """
    A naive implementation of the forward pass for a convolutional layer.
    The input consists of N data points, each with C channels, height H and
    width W. We convolve each input with F different filters, where each
    filter spans all C channels and has height HH and width WW.

    Input:
    - x: Input data of shape (N, C, H, W)
    - w: Filter weights of shape (F, C, HH, WW)
    - b: Biases, of shape (F,)
    - conv_param: A dictionary with the following keys:
      - 'stride': The number of pixels between adjacent receptive fields
        in the horizontal and vertical directions.
      - 'pad': The number of pixels that is used to zero-pad the input.

    During padding, 'pad' zeros should be placed symmetrically (i.e equally
    on both sides) along the height and width axes of the input. Be careful
    not to modfiy the original input x directly.

    Returns a tuple of:
    - out: Output data of shape (N, F, H', W') where H' and W' are given by
      H' = 1 + (H + 2 * pad - HH) / stride
      W' = 1 + (W + 2 * pad - WW) / stride
    - cache: (x, w, b, conv_param)
    """
    out = None
    ####################################################################
    # TODO: Implement the convolutional forward pass.                  #
    # Hint: you can use function torch.nn.functional.pad for padding.  #
    # You are NOT allowed to use anything in torch.nn in other places. #
    ####################################################################

    # Extract Info
    pad = conv_param['pad']
    stride = conv_param['stride']
    N, C, H, W = x.shape
    F, C, HH, WW = w.shape

    # Output Size
    H_out = int((H + 2 * pad - HH) / stride) + 1
    W_out = int((W + 2 * pad - WW) / stride) + 1

    # Pad Size 만큼 상,하,좌,우 Zero Padding
    x = torch.nn.functional.pad(x, (pad, pad, pad, pad))

    # Output
    out = torch.zeros((N, F, H_out, W_out), dtype=x.dtype, device = x.device)

    for n in range(N):      # Input 개수
      for f in range(F):    # Filter 개수
        for height in range(H_out):
          for width in range(W_out):
            # Convolution 
            out[n,f,height,width] = (x[n,:,height*stride:height*stride+HH,width*stride:width*stride+WW] *w[f]).sum() + b[f]

    #####################################################################
    #                          END OF YOUR CODE                         #
    #####################################################################
    cache = (x, w, b, conv_param)
    return out, cache

 

forward 방식은 Convolution 연산을 구현 해 주면 된다.

 

conv_param에 저장된 Padding, Stride Size 및 Input $x$, Weight $w$의 크기 정보를 추출한다. 이후 Convolution 연산이 진행되면서 Stride, Padding을 통해 쵱종적인 Output Size를 계산 해 준다. 하나의 층에 대한 Output Size는 다음 공식에 기반하여 쉽게 계산할 수 있다. 

 

$$ \left \lfloor \frac{I-K+2*P}{S} + 1\right \rfloor$$

 

이때, $I$는 입력 이미지의 너비(높이), $K$는 커널의 너비(높이), $P$는 Padding Size, $S$는 Stride Size 이다. 만약 나누어 떨어지지 않는 경우 소숫점 아래를 버린다. 이를 기반으로 최종적인 Output은 $N, F, H_out, W_out$의 크기를 가짐을 계산할 수 있다. 

 

Convolution 연산을 위해 torch.nn.functional.pad를 사용하여 상, 하, 좌, 우로 Zero Padding을 한다. 이후 각각의 Input을 각각의 Filter에 대하여 Convolution 연산을 다음과 같이 진행한다.

out[n,f,height,width] = (x[n,:,height*stride:height*stride+HH,width*stride:width*stride+WW] *w[f]).sum() + b[f]

 

수식이 약간 생소할 수 있지만 stride를 고려하여 $x$의 Local Region과 $w$를 Matrix Multiply 한 뒤, bias를 더해주면 된다. 또한 한가지 생소할 수 있는 점이 $x[n,:,height*stride:height*stride+HH,width*stride:width*stride+WW]$인데, Depth에 대해서는 " : "로 전체에 대해서 적용한다는 점이다. 이는 실제로 Convolution이 수행될 때, Input의 Depth와 Filter(Weight)의 Depth가 같기 때문이다. 

 

Convolutional layer : backward

def backward(dout, cache):
        """
        A naive implementation of the backward pass for a convolutional layer.
          Inputs:
        - dout: Upstream derivatives.
        - cache: A tuple of (x, w, b, conv_param) as in conv_forward_naive

        Returns a tuple of:
        - dx: Gradient with respect to x
        - dw: Gradient with respect to w
        - db: Gradient with respect to b
        """
        dx, dw, db = None, None, None
        ###############################################################
        # TODO: Implement the convolutional backward pass.            #
        ###############################################################
        
        # Extract Info
        x, w, b, conv_param = cache
        pad = conv_param['pad']
        stride = conv_param['stride']
        F, C, HH, WW = w.shape
        N, F, H_dout, W_dout = dout.shape

        # Initialize Grdient
        db = torch.zeros_like(b)
        dw = torch.zeros_like(w)
        dx = torch.zeros_like(x)

        # Calculate Gradient
        for n in range(N):
          for f in range(F):
            for height in range(H_dout):
              for width in range(W_dout):
                # Bias의 Local Gradient = 1
                db[f] += dout[n, f, height, width]  
                # Weight의 Local Gradient = X
                # 특정 Weight에 의한 Activation Value는 전체 Input Depth 모두가 영향을 준다(Convolution 과정 생각)
                dw[f] += x[n,:,height*stride:height*stride + HH, width*stride:width*stride + WW] * dout[n, f, height, width]
                # Input의 Local Gradient = W
                dx[n,:,height*stride:height*stride + HH, width*stride:width*stride + WW] += w[f] * dout[n, f, height, width]

        # Padding 제거
        dx = dx[:,:,pad:-pad,pad:-pad]

        ###############################################################
        #                       END OF YOUR CODE                      #
        ###############################################################
        return dx, dw, db

 

backward 역시 지난 과제와 유사한 방향으로 구현 가능하다. 먼저 각각의 필요한 정보를 가져온 뒤, db, dw, dx를 0으로 초기화 한다. 이후 각각의 Input을 각각의 Filter에 대하여 Convolution 될 때 마다 db, dw, dx를 계산하면 된다. 지역적으로는 Convolution 연산이 수행되기에 Bias의 Local Gradient는 1, W의 Local Gradient는 Input $x$, X의 Local Gradient는 Weight $w$이다. 따라서 Local Gradient와 Upstream Gradient로 각각의 Gradient를 계산할 수 있으며, 이때 Input $x$의 경우 전체 Depth 모두 관여하기에 " : "임을 주의하자.

 

Gradient 계산 이후, Padding 된 픽셀에 대한 Gradient 정보는 제거 해 주면 된다.

 

Max pooling

Max pooling : forward

def forward(x, pool_param):
    """
    A naive implementation of the forward pass for a max-pooling layer.

    Inputs:
    - x: Input data, of shape (N, C, H, W)
    - pool_param: dictionary with the following keys:
      - 'pool_height': The height of each pooling region
      - 'pool_width': The width of each pooling region
      - 'stride': The distance between adjacent pooling regions
    No padding is necessary here.

    Returns a tuple of:
    - out: Output of shape (N, C, H', W') where H' and W' are given by
      H' = 1 + (H - pool_height) / stride
      W' = 1 + (W - pool_width) / stride
    - cache: (x, pool_param)
    """
    out = None
    ####################################################################
    # TODO: Implement the max-pooling forward pass                     #
    ####################################################################

    # Extract Info
    pool_height = pool_param['pool_height']
    pool_width = pool_param['pool_width']
    stride = pool_param['stride']
    N, C, H, W = x.shape
    H_out = int((H - pool_height) / stride) + 1
    W_out = int((W - pool_width) / stride) + 1

    out = torch.zeros((N, C, H_out, W_out), dtype=x.dtype, device = x.device)

    for n in range(N):
      for c in range(C):
        for height in range(H_out):
          for width in range(W_out):
            # Pooling Region
            pool_region = x[n, c, height*stride:height*stride+pool_height, width*stride:width*stride+pool_width]
            # Max Pooling
            out[n, c, height, width] = pool_region.max()
    ####################################################################
    #                         END OF YOUR CODE                         #
    ####################################################################
    cache = (x, pool_param)
    return out, cache

 

Pooling Width, Height를 고려해 pool_region을 설정한 뒤, torch.max로 Max 값을 찾아 Output으로 할당한다. 

 

def backward(dout, cache):
    """
    A naive implementation of the backward pass for a max-pooling layer.
    Inputs:
    - dout: Upstream derivatives
    - cache: A tuple of (x, pool_param) as in the forward pass.
    Returns:
    - dx: Gradient with respect to x
    """
    dx = None
    #####################################################################
    # TODO: Implement the max-pooling backward pass                     #
    #####################################################################

    # Extract Info
    x, pool_param = cache
    pool_height = pool_param['pool_height']
    pool_width = pool_param['pool_width']
    stride = pool_param['stride']
    N, C, H, W = x.shape

    H_out = int((H - pool_height) / stride) + 1
    W_out = int((W - pool_width) / stride) + 1

    # Initialize dx
    dx = torch.zeros_like(x)

    for n in range(N):              
      for c in range(C):         
          for height in range(H_out):  
              for width in range(W_out): 
                # Pooling Region
                pool_region = x[n, c, height*stride:height*stride+pool_height, width*stride:width*stride+pool_width]

                # Max Pooling인 뉴런 확인
                max_value = pool_region.max()
                max_mask = (pool_region == max_value)
                # Max 값을 가지는 뉴런만 계산
                dx[n, c, height*stride:height*stride+pool_height, width*stride:width*stride+pool_width] += max_mask * dout[n,c,height,width]

    ####################################################################
    #                          END OF YOUR CODE                        #
    ####################################################################
    return dx

 

Max Pooling의 Gradient 계산은, Max Value를 가지는 뉴런에 대해서만 진행하면 된다. 따라서 Forward와 유사한 방식으로 Pool_region에 대하여 max 값을 가지는 mask를 생성하고, Upstream Gradient와 mask를 곱하여 Max 값을 가지는 뉴런에게만 Gradient가 전달되록 구현하면 된다. 

 

Fast layers

class FastConv(object):

  @staticmethod
  def forward(x, w, b, conv_param):
    N, C, H, W = x.shape
    F, _, HH, WW = w.shape
    stride, pad = conv_param['stride'], conv_param['pad']
    # 2D Convolution Layer 정의
    layer = torch.nn.Conv2d(C, F, (HH, WW), stride=stride, padding=pad)
    # 해당 Layer의 Learnable parameter 설정
    layer.weight = torch.nn.Parameter(w)
    layer.bias = torch.nn.Parameter(b)
    # 입력 Tensor 설정 및 Gradient 계산이 가능하도록 설정
    tx = x.detach()
    tx.requires_grad = True
    # forward
    out = layer(tx)
    # backward에 사용하도록 cache에 저장
    cache = (x, w, b, conv_param, tx, out, layer)
    return out, cache

  @staticmethod
  def backward(dout, cache):
    try:
      x, _, _, _, tx, out, layer = cache
      # Backpropagation 수행 : Gradient 계산
      out.backward(dout)
      # 각각의 Gradient 분리
      dx = tx.grad.detach()
      dw = layer.weight.grad.detach()
      db = layer.bias.grad.detach()
      # 다음 Backward pass를 위한 초기화(누적방지)
      layer.weight.grad = layer.bias.grad = None
    except RuntimeError:
      dx, dw, db = torch.zeros_like(tx), torch.zeros_like(layer.weight), torch.zeros_like(layer.bias)
    return dx, dw, db


class FastMaxPool(object):

  @staticmethod
  def forward(x, pool_param):
    N, C, H, W = x.shape
    pool_height, pool_width = pool_param['pool_height'], pool_param['pool_width']
    stride = pool_param['stride']
    layer = torch.nn.MaxPool2d(kernel_size=(pool_height, pool_width), stride=stride)
    tx = x.detach()
    tx.requires_grad = True
    out = layer(tx)
    cache = (x, pool_param, tx, out, layer)
    return out, cache

  @staticmethod
  def backward(dout, cache):
    try:
      x, _, tx, out, layer = cache
      out.backward(dout)
      dx = tx.grad.detach()
    except RuntimeError:
      dx = torch.zeros_like(tx)
    return dx

 

이전까지 열심히 구현 하였지만, 결국에는 더 똑똑한 사람들이 이미 효율적인 연산을 기반으로 Modulation을 해놨다. 따라서 우리는 torch.nn Module을 적재적소에 잘 활용하여 forward, backward를 구현하면 된다. 이는 기존에 구현한 흐름과 동일하게 진행되지만, 단지 Module로 구현된 것을 활용하여 쉽게 구현 가능하다. 

 

Three Layer Convolutional Network

Three Layer Convolutional Network는 다음과 같은 구조로 이루어진다.

$$ Conv \to ReLU \to 2x2 Max Pooling \to Linear \to ReLU \to Linear \to Softmax$$ 

 

class ThreeLayerConvNet(object):
  
    """
    A three-layer convolutional network with the following architecture:
    conv - relu - 2x2 max pool - linear - relu - linear - softmax
    The network operates on minibatches of data that have shape (N, C, H, W)
    consisting of N images, each with height H and width W and with C input
    channels.
    """

    def __init__(self,
                 input_dims=(3, 32, 32),
                 num_filters=32,
                 filter_size=7,
                 hidden_dim=100,
                 num_classes=10,
                 weight_scale=1e-3,
                 reg=0.0,
                 dtype=torch.float,
                 device='cpu'):
        """
        Initialize a new network.
        Inputs:
        - input_dims: Tuple (C, H, W) giving size of input data
        - num_filters: Number of filters to use in the convolutional layer
        - filter_size: Width/height of filters to use in convolutional layer
        - hidden_dim: Number of units to use in fully-connected hidden layer
        - num_classes: Number of scores to produce from the final linear layer.
        - weight_scale: Scalar giving standard deviation for random
          initialization of weights.
        - reg: Scalar giving L2 regularization strength
        - dtype: A torch data type object; all computations will be performed
          using this datatype. float is faster but less accurate, so you
          should use double for numeric gradient checking.
        - device: device to use for computation. 'cpu' or 'cuda'
        """
        self.params = {}
        self.reg = reg
        self.dtype = dtype

        ######################################################################
        # TODO: Initialize weights，biases for the three-layer convolutional #
        # network. Weights should be initialized from a Gaussian             #
        # centered at 0.0 with standard deviation equal to weight_scale;     #
        # biases should be initialized to zero. All weights and biases       #
        # should be stored in thedictionary self.params. Store weights and   #
        # biases for the convolutional layer using the keys 'W1' and 'b1';   #
        # use keys 'W2' and 'b2' for the weights and biases of the hidden    #
        # linear layer, and key 'W3' and 'b3' for the weights and biases of  #
        # the output linear layer                                            #
        #                                                                    #
        # IMPORTANT: For this assignment, you can assume that the padding    #
        # and stride of the first convolutional layer are chosen so that     #
        # **the width and height of the input are preserved**. Take a        #
        # look at the start of the loss() function to see how that happens.  #
        ######################################################################

        # Size 설정
        conv_param = {'stride': 1, 'pad': (filter_size - 1) // 2}     # Preserve Resolution
        pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}

        # Extract Info
        C, H, W = input_dims
        HH = filter_size
        WW = filter_size

        # Output Size
        H_out_conv = int((H + 2*conv_param['pad'] - HH)/conv_param['stride']) + 1
        W_out_conv = int((W + 2*conv_param['pad'] - WW)/conv_param['stride']) + 1
        H_out = int((H_out_conv - pool_param['pool_height']) / pool_param['stride']) + 1
        W_out = int((W_out_conv - pool_param['pool_width']) / pool_param['stride']) + 1

        # First Layer : Conv + ReLU + Pooling
        self.params['W{}'.format(1)] = torch.zeros(num_filters, C, filter_size, filter_size, dtype=dtype,device = device)
        self.params['W{}'.format(1)] += weight_scale*torch.randn(num_filters, C, filter_size, filter_size, dtype=dtype,device= device)
        self.params['b{}'.format(1)] = torch.zeros(num_filters, dtype=dtype,device= device)        

        # Second Layer : Linear + ReLU
        self.params['W{}'.format(2)] = torch.zeros(num_filters*H_out*W_out, hidden_dim, dtype=dtype,device = device)
        self.params['W{}'.format(2)] += weight_scale*torch.randn(num_filters*H_out*W_out, hidden_dim, dtype=dtype,device= device)
        self.params['b{}'.format(2)] = torch.zeros(hidden_dim, dtype=dtype,device= device)        

        # Third Layer : Linear
        self.params['W{}'.format(3)] = torch.zeros(hidden_dim, num_classes, dtype=dtype,device = device)
        self.params['W{}'.format(3)] += weight_scale*torch.randn(hidden_dim, num_classes, dtype=dtype,device= device)
        self.params['b{}'.format(3)] = torch.zeros(num_classes, dtype=dtype,device= device)   

        ######################################################################
        #                            END OF YOUR CODE                        #
        ######################################################################

 

ThreeLayerConvNEt의 초기 Parameter는 다음과 같이 설정할 수 있다. Weight는 평균이 0, 표준편차가 weight_scale으로, Bias는 모두 0으로 설정하였다. 

 

주의해야 할 점은 각각의 Weight, Bais의 크기이다. 첫번째 Layer는 Conv, ReLU, Max Pooling을 거친다. 이때의 Weight의 경우, Filter의 개수는 num_filters이고, 각각의 Filter는 C x filter_size x filter_size의 크기를 가진다. 두번째는 Linear이기 때문에, Max Pooling을 거친 Tensor는 num_filters x H_out x W_out의 크기를 가지므로 Weight는 이를 Flatten  시킨 ( num_filters*H_out*W_out)를 Height로, hidden_dim을 Width로 하는 num_filters*H_out*W_out x hidden_dim의 크기를 가진다. 마지막의 경우 각각의 클래스에 대한 Softmax 확률을 계산하기위해 Linear로 연결되기에 Weight는 hidden_dim x num_classes의 크기를 가진다. 

 

def loss(self, X, y=None):
        """
        Evaluate loss and gradient for the three-layer convolutional network.
        Input / output: Same API as TwoLayerNet.
        """
        X = X.to(self.dtype)
        W1, b1 = self.params['W1'], self.params['b1']
        W2, b2 = self.params['W2'], self.params['b2']
        W3, b3 = self.params['W3'], self.params['b3']

        # pass conv_param to the forward pass for the convolutional layer
        # Padding and stride chosen to preserve the input spatial size
        filter_size = W1.shape[2]
        conv_param = {'stride': 1, 'pad': (filter_size - 1) // 2}

        # pass pool_param to the forward pass for the max-pooling layer
        pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}

        scores = None
        ######################################################################
        # TODO: Implement the forward pass for three-layer convolutional     #
        # net, computing the class scores for X and storing them in the      #
        # scores variable.                                                   #
        #                                                                    #
        # Remember you can use functions defined in your implementation      #
        # above                                                              #
        ######################################################################

        # Forward
        out_CRP, cache_CRP = Conv_ReLU_Pool.forward(X, self.params['W1'], self.params['b1'], conv_param, pool_param)
        out_LR, cache_LR = Linear_ReLU.forward(out_CRP, self.params['W2'], self.params['b2'])
        out_L, cache_L = Linear.forward(out_LR, self.params['W3'], self.params['b3'])
        scores = out_L

        ######################################################################
        #                             END OF YOUR CODE                       #
        ######################################################################

        if y is None:
            return scores

        loss, grads = 0.0, {}
        ####################################################################
        # TODO: Implement backward pass for three-layer convolutional net, #
        # storing the loss and gradients in the loss and grads variables.  #
        # Compute data loss using softmax, and make sure that grads[k]     #
        # holds the gradients for self.params[k]. Don't forget to add      #
        # L2 regularization!                                               #
        #                                                                  #
        # NOTE: To ensure that your implementation matches ours and you    #
        # pass the automated tests, make sure that your L2 regularization  #
        # does not include a factor of 0.5                                 #
        ####################################################################
        
        #Calaculate Loss
        loss, dout = softmax_loss(scores, y)
        
        # Regularization Term
        for i in range(1, 4):
          loss += (self.params['W{}'.format(i)] ** 2).sum() * self.reg

        # Backpropagation
        # Thrid Layer : Linear
        last_dout, dw, db = Linear.backward(dout, cache_L)
        grads['W{}'.format(i)] = dw + 2*self.params['W{}'.format(i)]*self.reg
        grads['b{}'.format(i)] = db
        i -=1

        # Second Layer : Linear + ReLU
        last_dout, dw, db = Linear_ReLU.backward(last_dout, cache_LR)
        grads['W{}'.format(i)] = dw + 2*self.params['W{}'.format(i)]*self.reg
        grads['b{}'.format(i)] = db
        i -=1

        # Thrid Layer : Conv + ReLU + Pool
        last_dout, dw, db = Conv_ReLU_Pool.backward(last_dout, cache_CRP)
        grads['W{}'.format(i)] = dw + 2*self.params['W{}'.format(i)]*self.reg
        grads['b{}'.format(i)] = db
        
        ###################################################################
        #                             END OF YOUR CODE                    #
        ###################################################################

        return loss, grads

 

Score의 경우 앞서 정의한 Conv_ReLU_Pool, Linear_ReLU, Linear Class의 forward Module을 활용하여 쉽게 계산할 수 있다. 최종 Loss는 softmax_loss Module을 활용하며 Regularization Term을 고려 해 준다. 각각에 대한 Gradient는 각 Class의 backward Module을 활용하여 계산할 수 있다. 이때, 각각에 해당되는 적절한 Class를 사용하고, Weight에 대한 Gradient의 경우 Regularization Term을 고려해 준다. 

 

from convolutional_networks import ThreeLayerConvNet

reset_seed(0)
model = ThreeLayerConvNet(dtype=torch.float64, device='cuda')

N = 50
X = torch.randn(N, 3, 32, 32, dtype=torch.float64, device='cuda')
y = torch.randint(10, size=(N,), dtype=torch.int64, device='cuda')
loss, grads = model.loss(X, y)
print('Initial loss (no regularization): ', loss.item())

model.reg = 0.5
loss, grads = model.loss(X, y)
print('Initial loss (with regularization): ', loss.item())

# Initial loss (no regularization):  2.3025849594193004
# Initial loss (with regularization):  2.7153541136268458

 

Regularization을 고려하지 않은 경우 초기 Random Initialization 된 Weight에 대한 Loss는 $log(C)  \approx 2.3$임을 확인함으로써 잘 구했음을 알 수 있다. 

 

from convolutional_networks import ThreeLayerConvNet

num_inputs = 2
input_dims = (3, 16, 16)
reg = 0.0
num_classes = 10
reset_seed(0)
X = torch.randn(num_inputs, *input_dims, dtype=torch.float64, device='cuda')
y = torch.randint(num_classes, size=(num_inputs,), dtype=torch.int64, device='cuda')

model = ThreeLayerConvNet(num_filters=3, filter_size=3,
                          input_dims=input_dims, hidden_dim=7,
                          weight_scale=5e-2, dtype=torch.float64, device='cuda')
loss, grads = model.loss(X, y)

for param_name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    param_grad_num = eecs598.grad.compute_numeric_gradient(f, model.params[param_name])
    print('%s max relative error: %e' % (param_name, eecs598.grad.rel_error(param_grad_num, grads[param_name])))
    
# W1 max relative error: 2.851376e-08
# W2 max relative error: 7.622434e-08
# W3 max relative error: 4.168769e-09
# b1 max relative error: 1.252231e-08
# b2 max relative error: 1.551252e-08
# b3 max relative error: 3.205744e-09

 

Gradient 역시 Numeric Gradient와 비교한 결과 매우 작은 오차로 적절히 잘 구현되었음을 확인하였다. 

 

Overfit small data

 

이를 바탕으로 모델에 비해 아주 작은 데이터 셋에 대하여 모델을 학습시켜본 결과 다음과 같다. 

 

Training Dataset에 대한 Accruacy와 Validation Dataset에 대한 Accuracy의 차이가 매우 큰 것을 보아 Overfitting 되었음을 확인할 수 있다. 

 

Visualize Filters

Solver를 활용하여 Validation Accuracy를 50%가 넘도록 학습시킨 뒤, 첫번째 Weight Matrix를 시각화 한 결과 다음과 같다. 

 

CNN 모델의 첫번째 Layer에서 학습되는 Filter는 Edge, 색상 대비, Texture 등을 감지하는 경향이 있음을 확인할 수 있다.