deepctr.layers.sequence module

Author:

Weichen Shen,wcshen1994@163.com

class deepctr.layers.sequence.AttentionSequencePoolingLayer(att_hidden_units=(80, 40), att_activation='sigmoid', weight_normalization=False, return_score=False, supports_masking=False, **kwargs)

The Attentional sequence pooling operation used in DIN.

Input shape

• A list of three tensor: [query,keys,keys_length]
• query is a 3D tensor with shape: (batch_size, 1, embedding_size)
• keys is a 3D tensor with shape: (batch_size, T, embedding_size)
• keys_length is a 2D tensor with shape: (batch_size, 1)

Output shape

• 3D tensor with shape: (batch_size, 1, embedding_size).

Arguments

• att_hidden_units:list of positive integer, the attention net layer number and units in each layer.
• att_activation: Activation function to use in attention net.
• weight_normalization: bool.Whether normalize the attention score of local activation unit.
• supports_masking:If True,the input need to support masking.

References

• [Zhou G, Zhu X, Song C, et al. Deep interest network for click-through rate prediction[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2018: 1059-1068.](https://arxiv.org/pdf/1706.06978.pdf)

build(input_shape)

Creates the variables of the layer.

call(inputs, mask=None, training=None, **kwargs)

This is where the layer’s logic lives.

Arguments:

inputs: Input tensor, or list/tuple of input tensors. **kwargs: Additional keyword arguments.

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask)

Computes an output mask tensor.

Arguments:

inputs: Tensor or list of tensors. mask: Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,

one per output tensor of the layer).

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

class deepctr.layers.sequence.BiLSTM(units, layers=2, res_layers=0, dropout_rate=0.2, merge_mode='ave', **kwargs)

A multiple layer Bidirectional Residual LSTM Layer.

Input shape

• 3D tensor with shape (batch_size, timesteps, input_dim).

Output shape

• 3D tensor with shape: (batch_size, timesteps, units).

Arguments

• units: Positive integer, dimensionality of the output space.
• layers:Positive integer, number of LSTM layers to stacked.
• res_layers: Positive integer, number of residual connection to used in last res_layers.
• dropout_rate: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs.
• merge_mode: merge_mode: Mode by which outputs of the forward and backward RNNs will be combined. One of { 'fw' , 'bw' , 'sum' , 'mul' , 'concat' , 'ave' , None }. If None, the outputs will not be combined, they will be returned as a list.

build(input_shape)

Creates the variables of the layer.

call(inputs, mask=None, **kwargs)

This is where the layer’s logic lives.

Arguments:

inputs: Input tensor, or list/tuple of input tensors. **kwargs: Additional keyword arguments.

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask)

Computes an output mask tensor.

Arguments:

inputs: Tensor or list of tensors. mask: Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,

one per output tensor of the layer).

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

class deepctr.layers.sequence.BiasEncoding(sess_max_count, seed=1024, **kwargs)

build(input_shape)

Creates the variables of the layer.

call(inputs, mask=None)

Parameters:concated_embeds_value – None * field_size * embedding_size Returns:None*1

compute_mask(inputs, mask=None)

Computes an output mask tensor.

Arguments:

inputs: Tensor or list of tensors. mask: Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,

one per output tensor of the layer).

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

class deepctr.layers.sequence.DynamicGRU(num_units=None, gru_type='GRU', return_sequence=True, **kwargs)

build(input_shape)

Creates the variables of the layer.

call(input_list)

Parameters:concated_embeds_value – None * field_size * embedding_size Returns:None*1

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

class deepctr.layers.sequence.KMaxPooling(k=1, axis=-1, **kwargs)

K Max pooling that selects the k biggest value along the specific axis.

Input shape

• nD tensor with shape: (batch_size, ..., input_dim).

Output shape

• nD tensor with shape: (batch_size, ..., output_dim).

Arguments

• k: positive integer, number of top elements to look for along the axis dimension.
• axis: positive integer, the dimension to look for elements.

build(input_shape)

Creates the variables of the layer.

call(inputs)

This is where the layer’s logic lives.

Arguments:

inputs: Input tensor, or list/tuple of input tensors. **kwargs: Additional keyword arguments.

Returns:

A tensor or list/tuple of tensors.

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

class deepctr.layers.sequence.SequencePoolingLayer(mode='mean', supports_masking=False, **kwargs)

The SequencePoolingLayer is used to apply pooling operation(sum,mean,max) on variable-length sequence feature/multi-value feature.

Input shape

• A list of two tensor [seq_value,seq_len]
• seq_value is a 3D tensor with shape: (batch_size, T, embedding_size)
• seq_len is a 2D tensor with shape : (batch_size, 1),indicate valid length of each sequence.

Output shape

• 3D tensor with shape: (batch_size, 1, embedding_size).

Arguments

• mode:str.Pooling operation to be used,can be sum,mean or max.
• supports_masking:If True,the input need to support masking.

build(input_shape)

Creates the variables of the layer.

call(seq_value_len_list, mask=None, **kwargs)

This is where the layer’s logic lives.

Arguments:

inputs: Input tensor, or list/tuple of input tensors. **kwargs: Additional keyword arguments.

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask)

Computes an output mask tensor.

Arguments:

inputs: Tensor or list of tensors. mask: Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,

one per output tensor of the layer).

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

class deepctr.layers.sequence.Transformer(att_embedding_size=1, head_num=8, dropout_rate=0.0, use_positional_encoding=True, use_res=True, use_feed_forward=True, use_layer_norm=False, blinding=True, seed=1024, supports_masking=False, **kwargs)

Simplified version of Transformer proposed in 《Attention is all you need》

Input shape

• a list of two 3D tensor with shape (batch_size, timesteps, input_dim) if supports_masking=True.
• a list of two 4 tensors, first two tensors with shape (batch_size, timesteps, input_dim),last two tensors with shape (batch_size, 1) if supports_masking=False.

Output shape

• 3D tensor with shape: (batch_size, 1, input_dim).

Arguments

• att_embedding_size: int.The embedding size in multi-head self-attention network.
• head_num: int.The head number in multi-head self-attention network.
• dropout_rate: float between 0 and 1. Fraction of the units to drop.
• use_positional_encoding: bool. Whether or not use positional_encoding
• use_res: bool. Whether or not use standard residual connections before output.
• use_feed_forward: bool. Whether or not use pointwise feed foward network.
• use_layer_norm: bool. Whether or not use Layer Normalization.
• blinding: bool. Whether or not use blinding.
• seed: A Python integer to use as random seed.
• supports_masking:bool. Whether or not support masking.

References

• [Vaswani, Ashish, et al. “Attention is all you need.” Advances in Neural Information Processing Systems. 2017.](https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf)

build(input_shape)

Creates the variables of the layer.

call(inputs, mask=None, training=None, **kwargs)

This is where the layer’s logic lives.

Arguments:

inputs: Input tensor, or list/tuple of input tensors. **kwargs: Additional keyword arguments.

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask=None)

Computes an output mask tensor.

Arguments:

inputs: Tensor or list of tensors. mask: Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,

one per output tensor of the layer).

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

class deepctr.layers.sequence.WeightedSequenceLayer(weight_normalization=True, supports_masking=False, **kwargs)

The WeightedSequenceLayer is used to apply weight score on variable-length sequence feature/multi-value feature.

Input shape

• A list of two tensor [seq_value,seq_len,seq_weight]
• seq_value is a 3D tensor with shape: (batch_size, T, embedding_size)
• seq_len is a 2D tensor with shape : (batch_size, 1),indicate valid length of each sequence.
• seq_weight is a 3D tensor with shape: (batch_size, T, 1)

Output shape

• 3D tensor with shape: (batch_size, T, embedding_size).

Arguments

• weight_normalization: bool.Whether normalize the weight score before applying to sequence.
• supports_masking:If True,the input need to support masking.

build(input_shape)

Creates the variables of the layer.

call(input_list, mask=None, **kwargs)

This is where the layer’s logic lives.

Arguments:

inputs: Input tensor, or list/tuple of input tensors. **kwargs: Additional keyword arguments.

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask)

Computes an output mask tensor.

Arguments:

inputs: Tensor or list of tensors. mask: Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,

one per output tensor of the layer).

compute_output_shape(input_shape)

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Arguments:

input_shape: Shape tuple (tuple of integers)

or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

Python dictionary.

deepctr.layers.sequence.positional_encoding(inputs, pos_embedding_trainable=True, zero_pad=False, scale=True)

Sinusoidal Positional_Encoding.

Args:

• inputs: A 2d Tensor with shape of (N, T).
• num_units: Output dimensionality
• zero_pad: Boolean. If True, all the values of the first row (id = 0) should be constant zero
• scale: Boolean. If True, the output will be multiplied by sqrt num_units(check details from paper)
• scope: Optional scope for variable_scope.
• reuse: Boolean, whether to reuse the weights of a previous layer by the same name.

Returns:

• A ‘Tensor’ with one more rank than inputs’s, with the dimensionality should be ‘num_units’