Dan Erez danFromTelAviv
danFromTelAviv
/ keras2-highway-network.py
Last active
April 24, 2018 13:46
— forked from iskandr/keras2-highway-network.py
Since Keras 2.0 removed the Highway Network layer, here's my attempt at implementing something equivalent using the functional API
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import keras.backend as K | |
| from keras.layers import Dense, Activation, Multiply, Add, Lambda | |
| import keras.initializers | |
| def highway_layers(value, n_layers, activation="tanh", gate_bias=-3): | |
| dim = K.int_shape(value)[-1] | |
| gate_bias_initializer = keras.initializers.Constant(gate_bias) | |
| for i in range(n_layers): | |
| gate = Dense(units=dim, bias_initializer=gate_bias_initializer)(value) | |
| gate = Activation("sigmoid")(gate) |