TensorFlow에서 정규화를 추가하는 방법은 무엇입니까?

TensorFlow에서 정규화를 추가하는 방법은 무엇입니까?

---

TensorFlow를 사용하여 구현 된 많은 사용 가능한 신경망 코드에서 정규화 용어는 손실 값에 추가 용어를 수동으로 추가하여 구현되는 경우가 많습니다.

내 질문은 다음과 같습니다.

1. 수동으로 수행하는 것보다 더 우아하거나 권장되는 정규화 방법이 있습니까?

2. 나는 또한 그것이 get_variable논쟁 을 가지고 있음 을 발견했다 regularizer. 어떻게 사용해야합니까? 내 관찰에 따르면 정규화를 전달하면 (예 : tf.contrib.layers.l2_regularizer정규화 된 용어를 나타내는 텐서가 계산되어라는 그래프 컬렉션에 추가됩니다 tf.GraphKeys.REGULARIZATOIN_LOSSES. 해당 컬렉션이 TensorFlow에서 자동으로 사용됩니까 (예 : 학습시 최적화 프로그램에서 사용됨)? 이 컬렉션을 혼자서 사용해야하나요?

---

두 번째 요점에서 말했듯이 regularizer인수를 사용하는 것이 권장되는 방법입니다. 에서 사용 get_variable하거나 한 번 설정하고 variable_scope모든 변수를 정규화 할 수 있습니다.

손실은 그래프에 수집되며 이와 같이 비용 함수에 수동으로 추가해야합니다.

  reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
  reg_constant = 0.01  # Choose an appropriate one.
  loss = my_normal_loss + reg_constant * sum(reg_losses)

도움이 되었기를 바랍니다.

---

기존 답변의 몇 가지 측면이 즉시 명확하지 않았으므로 다음은 단계별 가이드입니다.

1. 정규화기를 정의하십시오. 여기에서 정규화 상수를 설정할 수 있습니다. 예 :

   regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
   

2. 다음을 통해 변수를 만듭니다.

       weights = tf.get_variable(
           name="weights",
           regularizer=regularizer,
           ...
       )
   

   마찬가지로 일반 weights = tf.Variable(...)생성자 를 통해 변수를 생성 한 다음 tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, weights).

3. loss용어를 정의 하고 정규화 용어를 추가합니다.

   reg_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
   reg_term = tf.contrib.layers.apply_regularization(regularizer, reg_variables)
   loss += reg_term
   

   참고 : tf.contrib.layers.apply_regularization으로 구현 된 것처럼 보이 AddN므로 sum(reg_variables).

---

정답을 찾지 못 했으므로 간단한 정답을 제공하겠습니다. 두 가지 간단한 단계가 필요하며 나머지는 tensorflow 매직에 의해 수행됩니다.

1. 변수 또는 레이어를 만들 때 정규화를 추가합니다.

   tf.layers.dense(x, kernel_regularizer=tf.contrib.layers.l2_regularizer(0.001))
   # or
   tf.get_variable('a', regularizer=tf.contrib.layers.l2_regularizer(0.001))
   

2. 손실을 정의 할 때 정규화 용어를 추가하십시오.

   loss = ordinary_loss + tf.losses.get_regularization_loss()
   

---

contrib.learn라이브러리 로이를 수행하는 또 다른 옵션 은 Tensorflow 웹 사이트 의 Deep MNIST 자습서 를 기반으로하는 다음과 같습니다 . 먼저 관련 라이브러리 (예 :)를 가져 왔다고 가정하면 import tensorflow.contrib.layers as layers별도의 방법으로 네트워크를 정의 할 수 있습니다.

def easier_network(x, reg):
    """ A network based on tf.contrib.learn, with input `x`. """
    with tf.variable_scope('EasyNet'):
        out = layers.flatten(x)
        out = layers.fully_connected(out, 
                num_outputs=200,
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = tf.nn.tanh)
        out = layers.fully_connected(out, 
                num_outputs=200,
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = tf.nn.tanh)
        out = layers.fully_connected(out, 
                num_outputs=10, # Because there are ten digits!
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = None)
        return out 

그런 다음 기본 메서드에서 다음 코드 조각을 사용할 수 있습니다.

def main(_):
    mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)
    x = tf.placeholder(tf.float32, [None, 784])
    y_ = tf.placeholder(tf.float32, [None, 10])

    # Make a network with regularization
    y_conv = easier_network(x, FLAGS.regu)
    weights = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, 'EasyNet') 
    print("")
    for w in weights:
        shp = w.get_shape().as_list()
        print("- {} shape:{} size:{}".format(w.name, shp, np.prod(shp)))
    print("")
    reg_ws = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES, 'EasyNet')
    for w in reg_ws:
        shp = w.get_shape().as_list()
        print("- {} shape:{} size:{}".format(w.name, shp, np.prod(shp)))
    print("")

    # Make the loss function `loss_fn` with regularization.
    cross_entropy = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
    loss_fn = cross_entropy + tf.reduce_sum(reg_ws)
    train_step = tf.train.AdamOptimizer(1e-4).minimize(loss_fn)

To get this to work you need to follow the MNIST tutorial I linked to earlier and import the relevant libraries, but it's a nice exercise to learn TensorFlow and it's easy to see how the regularization affects the output. If you apply a regularization as an argument, you can see the following:

- EasyNet/fully_connected/weights:0 shape:[784, 200] size:156800
- EasyNet/fully_connected/biases:0 shape:[200] size:200
- EasyNet/fully_connected_1/weights:0 shape:[200, 200] size:40000
- EasyNet/fully_connected_1/biases:0 shape:[200] size:200
- EasyNet/fully_connected_2/weights:0 shape:[200, 10] size:2000
- EasyNet/fully_connected_2/biases:0 shape:[10] size:10

- EasyNet/fully_connected/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0
- EasyNet/fully_connected_1/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0
- EasyNet/fully_connected_2/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0

Notice that the regularization portion gives you three items, based on the items available.

With regularizations of 0, 0.0001, 0.01, and 1.0, I get test accuracy values of 0.9468, 0.9476, 0.9183, and 0.1135, respectively, showing the dangers of high regularization terms.

---

I tested tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES) and tf.losses.get_regularization_loss() with one l2_regularizer in the graph, and found that they return the same value. By observing the value's quantity, I guess reg_constant has already make sense on the value by setting the parameter of tf.contrib.layers.l2_regularizer.

---

If you have CNN you may do the following:

In your model function:

conv = tf.layers.conv2d(inputs=input_layer,
                        filters=32,
                        kernel_size=[3, 3],
                        kernel_initializer='xavier',
                        kernel_regularizer=tf.contrib.layers.l2_regularizer(1e-5),
                        padding="same",
                        activation=None) 
...

In your loss function:

onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=num_classes)
loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=logits)
regularization_losses = tf.losses.get_regularization_losses()
loss = tf.add_n([loss] + regularization_losses)

---

If anyone's still looking, I'd just like to add on that in tf.keras you may add weight regularization by passing them as arguments in your layers. An example of adding L2 regularization taken wholesale from the Tensorflow Keras Tutorials site:

model = keras.models.Sequential([
    keras.layers.Dense(16, kernel_regularizer=keras.regularizers.l2(0.001),
                       activation=tf.nn.relu, input_shape=(NUM_WORDS,)),
    keras.layers.Dense(16, kernel_regularizer=keras.regularizers.l2(0.001),
                       activation=tf.nn.relu),
    keras.layers.Dense(1, activation=tf.nn.sigmoid)
])

There's no need to manually add in the regularization losses with this method as far as I know.

Reference: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit#add_weight_regularization

---

Some answers make me more confused.Here I give two methods to make it clearly.

#1.adding all regs by hand
var1 = tf.get_variable(name='v1',shape=[1],dtype=tf.float32)
var2 = tf.Variable(name='v2',initial_value=1.0,dtype=tf.float32)
regularizer = tf.contrib.layers.l1_regularizer(0.1)
reg_term = tf.contrib.layers.apply_regularization(regularizer,[var1,var2])
#here reg_term is a scalar

#2.auto added and read,but using get_variable
with tf.variable_scope('x',
        regularizer=tf.contrib.layers.l2_regularizer(0.1)):
    var1 = tf.get_variable(name='v1',shape=[1],dtype=tf.float32)
    var2 = tf.get_variable(name='v2',shape=[1],dtype=tf.float32)
reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
#here reg_losses is a list,should be summed 

Then,it can be added into the total loss

---

cross_entropy = tf.losses.softmax_cross_entropy(
  logits=logits, onehot_labels=labels)

l2_loss = weight_decay * tf.add_n(
     [tf.nn.l2_loss(tf.cast(v, tf.float32)) for v in tf.trainable_variables()])

loss = cross_entropy + l2_loss

---

tf.GraphKeys.REGULARIZATION_LOSSES will not be added automatically, but there is a simple way to add them:

reg_loss = tf.losses.get_regularization_loss()
total_loss = loss + reg_loss

tf.losses.get_regularization_loss() uses tf.add_n to sum the entries of tf.GraphKeys.REGULARIZATION_LOSSES element-wise. tf.GraphKeys.REGULARIZATION_LOSSES will typically be a list of scalars, calculated using regularizer functions. It gets entries from calls to tf.get_variable that have the regularizer parameter specified. You can also add to that collection manually. That would be useful when using tf.Variable and also when specifying activity regularizers or other custom regularizers. For instance:

#This will add an activity regularizer on y to the regloss collection
regularizer = tf.contrib.layers.l2_regularizer(0.1)
y = tf.nn.sigmoid(x)
act_reg = regularizer(y)
tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, act_reg)

(In this example it would presumably be more effective to regularize x, as y really flattens out for large x.)

참고URL : https://stackoverflow.com/questions/37107223/how-to-add-regularizations-in-tensorflow