Mfcttrf

If I like to write a LSTM network and feed it by different input array sizes, how is it possible?

For example I want to get voice messages or text messages in a different language and translate them. So the first input maybe is "hello" but the second is "how are you doing". How can I design a LSTM that can handle different input array sizes?

I am using Keras implementation of LSTM.

We use LSTM layers with multiple input sizes. But, you need to process them before they are feed to the LSTM.

Padding the sequences:

You need the pad the sequences of varying length to a fixed length. For this preprocessing, you need to determine the max length of sequences in your dataset.

The values are padded mostly by the value of 0. You can do this in Keras with :

y = keras.preprocessing.sequence.pad_sequences( x , maxlen=10 )

• If the sequence is shorter than the max length, then zeros will appended till it has a length equal to the max length.




• If the sequence is longer than the max length then, the sequence will be trimmed to the max length.

The easiest way is to use Padding and Masking.

There are three general ways to handle variable-length sequences:

1. Batch size = 1,


2. Batch size > 1, with equi-length samples in each batch, and


3. Padding and masking (which can be used for (2))

For cases (1) and (2) you need to set the timesteps of LSTM to None, e.g.

model.add(LSTM(units, input_shape=(None, dimension)))

this way LSTM accepts batches that have different lengths; although samples inside each batch must be the same length. Then, you need to feed a custom data generator to model.fit_generator.

I have provided a complete example for simple case (1) at the end. Based on this example and the link, you should be able to build a generator for case (2). Specifically, we either (a) return batch_size of sequences with the same length, or (b) select sequences with almost the same length, and pad the shorter ones as will be illustrated for case (3), and use a Masking layer before LSTM layer to ignore the padded timestamps, e.g.

model.add(Masking(mask_value=special_value, input_shape=(None, dimension)))
model.add(LSTM(lstm_units))

First dimension of input_shape in Masking (for timestamps) is again None for the same aforementioned reason.

Padding and masking

In this approach, we pad the shorter sequences with a special value to be masked (skipped) later. For example, suppose each timestamp has dimension 2, and -10 is the special value, then

X = [

 [[1, 1.1],
 [0.9, 0.95]],

 [[2, 2.2],
 [1.9, 1.95],
 [1.8, 1.85]],

]

should be converted to

X2 = [

 [[1, 1.1],
 [0.9, 0.95],
 [-10, -10]],

 [[2, 2.2],
 [1.9, 1.95],
 [1.8, 1.85]],
]

This way, all instances would have the same length. Then, we use a Masking layer that skips those special timestamps like they don't exist.

Here is the code for cases (1) and (3):

from keras import Sequential
from keras.utils import Sequence
from keras.layers import LSTM, Dense, Masking
import numpy as np


class MyBatchGenerator(Sequence):
 'Generates data for Keras'
 def __init__(self, X, y, batch_size=1, shuffle=True):
 'Initialization'
 self.X = X
 self.y = y
 self.batch_size = batch_size
 self.shuffle = shuffle
 self.on_epoch_end()

 def __len__(self):
 'Denotes the number of batches per epoch'
 return int(np.floor(len(self.y)/self.batch_size))

 def __getitem__(self, index):
 return self.__data_generation(index)

 def on_epoch_end(self):
 'Shuffles indexes after each epoch'
 self.indexes = np.arange(len(self.y))
 if self.shuffle == True:
 np.random.shuffle(self.indexes)

 def __data_generation(self, index):
 Xb = np.empty((self.batch_size, *X[index].shape))
 yb = np.empty((self.batch_size, *y[index].shape))
 # naively use the same sample over and over again
 for s in range(0, self.batch_size):
 Xb[s] = X[index]
 yb[s] = y[index]
 return Xb, yb


# Parameters
N = 1000
halfN = int(N/2)
dimension = 2
lstm_units = 3

# Data
np.random.seed(123) # to generate the same numbers
timestamps = np.random.randint(1, 10, halfN) # sequences with timestamps between 1 to 10
X_zero = np.array([np.random.normal(0, 1, size=(timestamp, dimension)) for timestamp in timestamps])
y_zero = np.zeros((halfN, 1))
X_one = np.array([np.random.normal(1, 1, size=(timestamp, dimension)) for timestamp in timestamps])
y_one = np.ones((halfN, 1))
p = np.random.permutation(N) # to shuffle zero and one classes
X = np.concatenate((X_zero, X_one))[p]
y = np.concatenate((y_zero, y_one))[p]

# Batch = 1
model = Sequential()
model.add(LSTM(lstm_units, input_shape=(None, dimension)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())
model.fit_generator(MyBatchGenerator(X, y, batch_size=1), epochs=2)

# Padding and Masking
special_value = -10.0
max_timestamp = max(timestamps)
Xpad = np.full((N, max_timestamp, dimension), fill_value=special_value)
for i, x in enumerate(X):
 timestamp = x.shape[0]
 Xpad[i, 0:timestamp, :] = x
model2 = Sequential()
model2.add(Masking(mask_value=special_value, input_shape=(None, dimension)))
model2.add(LSTM(lstm_units))
model2.add(Dense(1, activation='sigmoid'))
model2.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model2.summary())
model2.fit(Xpad, y, epochs=50, batch_size=32)