ASR工业级代码-WeNet代码逐行分析（4）Decoder的forward

目前的位置：

encoder的forward走完了

在所有12层ConformerEncoderLayer的forward执行完毕之后，最后还有个after_norm 来最后搞一次layer normalization。

至此，Encoder的部分，就算完成了。

_calc_att_loss

[wenet/transformer/asr_model.py:120]

下面开启的就是和目标文本序列相关的，decoder，以及各种loss的计算的方面了。

来自语音方面的信息，一般以memory来标识。

decoder（left to right以及right to left），以及其他

上面是_calc_att_loss函数的一部分逻辑。

add_sos_eos

先看它的add_sos_eos方法：

[wenet/utils/common.py]

构造前后的sos, eos从而构造成类似自回归的形式，即：从y0 y1 y2 来预测y1 y2 y3。

上面也给出了一个例子，对于一个文本序列，里面有四个单词：

1969, 2306, 1715, 1737

我们搞出来的ys_in是类似：【5501, 1969, 2306, 1715, 1737】，

而ys_out则是类似：【1969, 2306, 1715, 1737,5501】

接着细看，pad_list：

补全到本batch的最长长度7，ys_in补充pad的是5501，而ys_out补充的pad的则是-1。

下面是具体的例子：

ys_in

ipdb>pad
tensor([[5501,1969,2306,1715,1737,5501,5501],
[5501,1527,1548,1609,1762,1741,5501],
[5501,1715,1737,1527,1548,1845,1867],
[5501,2973,1631,1795,5501,5501,5501],
[5501,1741,1885,1673,5501,5501,5501],
[5501,1677,1640,1737,5501,5501,5501],
[5501,1677,1609,1885,1741,5501,5501],
[5501,2277,4240,1845,5501,5501,5501],
[5501,1762,1737,1908,5501,5501,5501],
[5501,1527,1548,1609,1762,1969,5501],
[5501,1554,1516,1980,5501,5501,5501],
[5501,3500,3555,1741,1670,5501,5501]],device='cuda:0')

以及：ys_out

ipdb>pad
tensor([[1969,2306,1715,1737,5501,-1,-1],
[1527,1548,1609,1762,1741,5501,-1],
[1715,1737,1527,1548,1845,1867,5501],
[2973,1631,1795,5501,-1,-1,-1],
[1741,1885,1673,5501,-1,-1,-1],
[1677,1640,1737,5501,-1,-1,-1],
[1677,1609,1885,1741,5501,-1,-1],
[2277,4240,1845,5501,-1,-1,-1],
[1762,1737,1908,5501,-1,-1,-1],
[1527,1548,1609,1762,1969,5501,-1],
[1554,1516,1980,5501,-1,-1,-1],
[3500,3555,1741,1670,5501,-1,-1]],device='cuda:0')

然后是类似的方法，来构造逆序序列的输入和输出并搞padding：

逆序序列的准备。目的是为right-to-left decoder准备的。

deocder的概要

[wenet/transformer/decoder.py: 232]

decoder的概要

上面给出了decoder的概要，

接受encoder的输出，作为memory；

以及pad好的ys_in_pad，以及倒序的r_ys_in_pad。

然后就是调用left_decoder和right_decoder了。

left_decoder

[wenet/transformer/decoder.py] class TransformerDecoder

left-decoder的主要流程

mask 两种

看看mask的例子吧，这个相对比较简单：

mask的例子

一则是关于目标文本序列长度在一个batch中的mask，另外一个是关于causal masking的。

embed的细节：

ipdb>self
Sequential(
(0):Embedding(5502,512)
(1):PositionalEncoding(
(dropout):Dropout(p=0.1,inplace=False)
)
)

embedding: 文本token编码和位置编码

decoder layer:

[wenet/transformer/decoder_layer.py] class DecoderLayer

一层decoder layer，包括三个大的模块，和三个layer norm，以及两个linear层：

(0):DecoderLayer(
(self_attn):MultiHeadedAttention(
(linear_q):Linear(in_features=512,out_features=512,bias=True)
(linear_k):Linear(in_features=512,out_features=512,bias=True)
(linear_v):Linear(in_features=512,out_features=512,bias=True)
(linear_out):Linear(in_features=512,out_features=512,bias=True)
(dropout):Dropout(p=0.1,inplace=False)
)
(src_attn):MultiHeadedAttention(
(linear_q):Linear(in_features=512,out_features=512,bias=True)
(linear_k):Linear(in_features=512,out_features=512,bias=True)
(linear_v):Linear(in_features=512,out_features=512,bias=True)
(linear_out):Linear(in_features=512,out_features=512,bias=True)
(dropout):Dropout(p=0.1,inplace=False)
)
(feed_forward):PositionwiseFeedForward(
(w_1):Linear(in_features=512,out_features=2048,bias=True)
(activation):ReLU()
(dropout):Dropout(p=0.1,inplace=False)
(w_2):Linear(in_features=2048,out_features=512,bias=True)
)
(norm1):LayerNorm((512,),eps=1e-12,elementwise_affine=True)
(norm2):LayerNorm((512,),eps=1e-12,elementwise_affine=True)
(norm3):LayerNorm((512,),eps=1e-12,elementwise_affine=True)
(dropout):Dropout(p=0.1,inplace=False)
(concat_linear1):Linear(in_features=1024,out_features=512,bias=True)
(concat_linear2):Linear(in_features=1024,out_features=512,bias=True)
)

一个DecoderLayer里面有三个子模块

输入参数：

DecoderLayer的输入参数列表

输入包括了target text 序列，以及source wav的memory。

1 self-attn

[wenet/transformer/attention.py] class MultiHeadedAttention

self_attn函数，主要就是封装出来Q, K, V了。

因为是对文本序列使用了绝对位置编码，所以上面的pos_emb=[]没有赋值。

然后就是调用self.forward_qkv，打分scores，以及self.forward_attention了。

这两个函数都是在MultiHeadedAttention类里面被定义的。在encoder里面也用过它们。

1.1 forward_qkv

三个线性层，这个没啥说的。

1.2 forward_attention

再看下这个函数的输入，特别是mask，是长度mask + causal mask的结合：

>/workspace/asr/wenet/wenet/transformer/attention.py(68)forward_attention()
67
--->68defforward_attention(self,value:torch.Tensor,scores:torch.Tensor,
69        mask:Optional[torch.Tensor])->torch.Tensor:

ipdb>value.shape,scores.shape,mask.shape
(torch.Size([12,8,7,64]),torch.Size([12,8,7,7]),torch.Size([12,7,7]))
ipdb>mask[0]
tensor([[True,False,False,False,False,False,False],
[True,True,False,False,False,False,False],
[True,True,True,False,False,False,False],
[True,True,True,True,False,False,False],
[True,True,True,True,True,False,False],
[True,True,True,True,True,True,False],
[True,True,True,True,True,True,True]],device='cuda:0')
ipdb>mask[-1]
tensor([[True,False,False,False,False,False,False],
[True,True,False,False,False,False,False],
[True,True,True,False,False,False,False],
[True,True,True,True,False,False,False],
[True,True,True,True,False,False,False],
[True,True,True,True,False,False,False],
[True,True,True,True,False,False,False]],device='cuda:0')

这里面有关键的一行：

通过mask对scores改造！

ipdb>scores[-1,0]
tensor([[0.5888,0.4690,-0.5331,-0.2052,0.5192,0.6984,0.6824],
[0.0623,0.0422,0.3149,-0.2785,0.2940,0.2956,0.4160],
[-0.2210,-0.3464,-0.2069,-0.2525,-0.3247,-0.2746,-0.1471],
[0.3421,-0.5054,0.2017,0.2765,0.2724,0.1311,0.3179],
[0.4926,0.5492,-0.5834,-0.0920,0.4683,0.6382,0.6231],
[0.5342,0.5590,-0.5261,-0.1105,0.5625,0.7140,0.7838],
[0.4247,0.6325,-0.7112,-0.0731,0.4031,0.5676,0.5104]],
device='cuda:0',grad_fn=)
ipdb>mask[-1,0]
tensor([[False,True,True,True,True,True,True],
[False,False,True,True,True,True,True],
[False,False,False,True,True,True,True],
[False,False,False,False,True,True,True],
[False,False,False,False,True,True,True],
[False,False,False,False,True,True,True],
[False,False,False,False,True,True,True]],device='cuda:0')
ipdb>n
>/workspace/asr/wenet/wenet/transformer/attention.py(90)forward_attention()
89       scores=scores.masked_fill(mask,-float('inf'))
--->90   attn=torch.softmax(scores,dim=-1).masked_fill(
91         mask,0.0)# (batch, head, time1, time2)
ipdb>scores[-1,0]
tensor([[0.5888,-inf,-inf,-inf,-inf,-inf,-inf],
[0.0623,0.0422,-inf,-inf,-inf,-inf,-inf],
[-0.2210,-0.3464,-0.2069,-inf,-inf,-inf,-inf],
[0.3421,-0.5054,0.2017,0.2765,-inf,-inf,-inf],
[0.4926,0.5492,-0.5834,-0.0920,-inf,-inf,-inf],
[0.5342,0.5590,-0.5261,-0.1105,-inf,-inf,-inf],
[0.4247,0.6325,-0.7112,-0.0731,-inf,-inf,-inf]],
device='cuda:0',grad_fn=)

scores的形状为[12, 8, 7, 7]。

上面的scores[-1, 0]表示的是当前batch中的最后一个序列，head=0的attention score 矩阵的经过mask之后的结果。这个序列的长度为4，所以最右边三列都是-inf。

【最好：最下面三行也应该是都是-inf !】

然后是：

--->90           attn=torch.softmax(scores,dim=-1).masked_fill(
91                mask,0.0)# (batch, head, time1, time2)

我们拆开来看看，

其一，torch.softmax(scores, dim=-1)，得到的是：

ipdb>temp=torch.softmax(scores,dim=-1)
ipdb>temp.shape
torch.Size([12,8,7,7])
ipdb>temp[-1,0]tensor([[1.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000],
[0.5050,0.4950,0.0000,0.0000,0.0000,0.0000,0.0000],
[0.3453,0.3046,0.3502,0.0000,0.0000,0.0000,0.0000],
[0.3092,0.1325,0.2687,0.2896,0.0000,0.0000,0.0000],
[0.3382,0.3579,0.1153,0.1885,0.0000,0.0000,0.0000],
[0.3453,0.3539,0.1196,0.1812,0.0000,0.0000,0.0000],
[0.3165,0.3895,0.1016,0.1924,0.0000,0.0000,0.0000]],
device='cuda:0',grad_fn=)

可以看到，-inf的地方，都是0了。最后三列也都是0.

其二，masked_fill(mask, 0.0)那个其实感觉没有用：

ipdb>temp[-1,0]
tensor([[1.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000],
[0.5050,0.4950,0.0000,0.0000,0.0000,0.0000,0.0000],
[0.3453,0.3046,0.3502,0.0000,0.0000,0.0000,0.0000],
[0.3092,0.1325,0.2687,0.2896,0.0000,0.0000,0.0000],
[0.3382,0.3579,0.1153,0.1885,0.0000,0.0000,0.0000],
[0.3453,0.3539,0.1196,0.1812,0.0000,0.0000,0.0000],
[0.3165,0.3895,0.1016,0.1924,0.0000,0.0000,0.0000]],
device='cuda:0',grad_fn=)
ipdb>temp=temp.masked_fill(mask,0.0)
ipdb>temp[-1,0]
tensor([[1.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000],
[0.5050,0.4950,0.0000,0.0000,0.0000,0.0000,0.0000],
[0.3453,0.3046,0.3502,0.0000,0.0000,0.0000,0.0000],
[0.3092,0.1325,0.2687,0.2896,0.0000,0.0000,0.0000],
[0.3382,0.3579,0.1153,0.1885,0.0000,0.0000,0.0000],
[0.3453,0.3539,0.1196,0.1812,0.0000,0.0000,0.0000],
[0.3165,0.3895,0.1016,0.1924,0.0000,0.0000,0.0000]],
device='cuda:0',grad_fn=)

执行上面的masked_fill(mask, 0.0)没有改变啥。

如此，forward_attention的完整的逻辑就如下图所示了：

[wenet/transformer/attention.py] class MultiHeadedAttention

forward_attention函数的细节

通过memory-mask，把scores的最后两列赋值为-inf：

ipdb>scores[-1,0]
tensor([[-0.0635,-0.0746,-0.1542,-0.0638,-0.2013,-0.0418,-0.1145,-0.1504,
-0.1842,-0.0661,-0.3125,0.1422,-0.0963],
[0.2473,-0.0016,-0.2346,0.0759,-0.1538,-0.0522,-0.1290,-0.0740,-0.1654,-0.1559,-0.0379,-0.1297,-0.0999],
[-0.1114,0.0246,0.5050,0.2623,0.3198,0.3314,0.3654,0.4987,0.3887,0.3395,0.2231,0.3547,0.3265],
[-0.6373,-0.8985,-0.6710,-0.2674,-0.3900,-0.3370,-0.5896,-0.4747,-0.5338,-0.5168,-0.2306,-0.2827,-0.3717],
[0.1051,0.1630,0.1033,0.1525,0.0340,0.0996,0.0902,0.0140,-0.0556,0.0738,-0.1748,0.3400,0.0802],
[0.0810,0.2666,0.1509,0.1975,-0.0071,0.1362,0.1721,0.1444,0.0991,0.2025,-0.0953,0.3370,0.2114],
[0.0400,0.1195,0.0041,0.1349,-0.0089,0.0849,0.0815,-0.0257,-0.0710,0.0894,-0.2177,0.2357,0.1076]],
device='cuda:0',grad_fn=)
ipdb>n
>/workspace/asr/wenet/wenet/transformer/attention.py(90)forward_attention()
89   scores=scores.masked_fill(mask,-float('inf'))
--->90   attn=torch.softmax(scores,dim=-1).masked_fill(
91   mask,0.0)# (batch, head, time1, time2)
ipdb>scores[-1,0]
tensor([[-0.0635,-0.0746,-0.1542,-0.0638,-0.2013,-0.0418,-0.1145,-0.1504,-0.1842,-0.0661,-0.3125,-inf,-inf],
[0.2473,-0.0016,-0.2346,0.0759,-0.1538,-0.0522,-0.1290,-0.0740,-0.1654,-0.1559,-0.0379,-inf,-inf],
[-0.1114,0.0246,0.5050,0.2623,0.3198,0.3314,0.3654,0.4987,0.3887,0.3395,0.2231,-inf,-inf],
[-0.6373,-0.8985,-0.6710,-0.2674,-0.3900,-0.3370,-0.5896,-0.4747,-0.5338,-0.5168,-0.2306,-inf,-inf],
[0.1051,0.1630,0.1033,0.1525,0.0340,0.0996,0.0902,0.0140,-0.0556,0.0738,-0.1748,-inf,-inf],
[0.0810,0.2666,0.1509,0.1975,-0.0071,0.1362,0.1721,0.1444,0.0991,0.2025,-0.0953,-inf,-inf],
[0.0400,0.1195,0.0041,0.1349,-0.0089,0.0849,0.0815,-0.0257,-0.0710,0.0894,-0.2177,-inf,-inf]],
device='cuda:0',grad_fn=)

2 src-attn

交叉注意力：

基本逻辑和self-attn类似，只不过是query是text, key-value都是来自wav；mask用的是source wave frame length的masking

我们展开来看看就好了：

src-attn中的forward_qkv方法

以及：

src-attn中的forward_attention函数的细节

3. FFN

这个就是普通transformer里面的FFN，和编码器那边的1/2的两个FFN有不同。

FFN的细节，512 -&amp;amp;gt; 2048 -&amp;amp;gt; 512

3层DecoderLayer之后

循环3层DecoderLayer之后：

后续的简单处理，包括512 -&amp;amp;gt; 5502的映射！

这里面有个有意思的：olens：

ipdb>olens[-1]
tensor([7,6,5,4,0,0,0],device='cuda:0')
ipdb>olens
tensor([[7,6,5,4,3,2,1],
[7,6,5,4,3,2,0],
[7,6,5,4,0,0,0],
[7,6,5,4,3,0,0],
[7,6,5,4,3,0,0],
[7,6,5,4,3,0,0],
[7,6,5,4,3,0,0],
[7,6,5,4,3,0,0],
[7,6,5,4,3,0,0],
[7,6,5,4,0,0,0],
[7,6,5,4,3,0,0],
[7,6,5,4,0,0,0]],device='cuda:0')
ipdb>tgt.shape
torch.Size([12,7])
ipdb>tgt
tensor([[5501,1762,1527,1548,1609,1762,1741],
[5501,1969,2403,1952,1950,1762,5501],
[5501,1885,2929,5290,5501,5501,5501],
[5501,2062,2006,2089,2146,5501,5501],
[5501,4906,1945,1701,1762,5501,5501],
[5501,1554,2161,1885,2161,5501,5501],
[5501,1640,1640,1928,1548,5501,5501],
[5501,1885,1694,1516,1845,5501,5501],
[5501,1677,1845,2895,1816,5501,5501],
[5501,1565,1795,1945,5501,5501,5501],
[5501,1527,1885,1640,1694,5501,5501],
[5501,2277,4240,1845,5501,5501,5501]],device='cuda:0')

后续用到它的时候，还会再提。

目前的位置：

left_decoder完成；right_decoder等待开始

right_decoder

[wenet/transformer/decoder.py] class TransformerDecoder

这个的逻辑和left_decoder基本相同。不过是逆序。

基本结构：

ipdb>self.decoders
ModuleList(
(0,1,2):DecoderLayer(
(self_attn):MultiHeadedAttention(
(linear_q):Linear(in_features=512,out_features=512,bias=True)
(linear_k):Linear(in_features=512,out_features=512,bias=True)
(linear_v):Linear(in_features=512,out_features=512,bias=True)
(linear_out):Linear(in_features=512,out_features=512,bias=True)
(dropout):Dropout(p=0.1,inplace=False)
)
(src_attn):MultiHeadedAttention(
(linear_q):Linear(in_features=512,out_features=512,bias=True)
(linear_k):Linear(in_features=512,out_features=512,bias=True)
(linear_v):Linear(in_features=512,out_features=512,bias=True)
(linear_out):Linear(in_features=512,out_features=512,bias=True)
(dropout):Dropout(p=0.1,inplace=False)
)
(src_attn): MultiHeadedAttention(
      (linear_q): Linear(in_features=512, out_features=512, bias=True)
      (linear_k): Linear(in_features=512, out_features=512, bias=True)
      (linear_v): Linear(in_features=512, out_features=512, bias=True)
      (linear_out): Linear(in_features=512, out_features=512, bias=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
(feed_forward): PositionwiseFeedForward(
      (w_1): Linear(in_features=512, out_features=2048, bias=True)
      (activation): ReLU()
      (dropout): Dropout(p=0.1, inplace=False)
      (w_2): Linear(in_features=2048, out_features=512, bias=True)
    )
(norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
    (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
    (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
    (concat_linear1): Linear(in_features=1024, out_features=512, bias=True)
    (concat_linear2): Linear(in_features=1024, out_features=512, bias=True)
  )
 (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
    (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
    (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
    (concat_linear1): Linear(in_features=1024, out_features=512, bias=True)
    (concat_linear2): Linear(in_features=1024, out_features=512, bias=True)
  )
)

memory还是正序。

关于left-to-right decoder和right-to-left decoder就先到这里。

ASR工业级代码-WeNet代码逐行分析（4）Decoder的forward

目前的位置：

_calc_att_loss

add_sos_eos

deocder的概要

left_decoder

mask 两种

embed的细节：

decoder layer:

输入参数：

1 self-attn

1.1 forward_qkv

1.2 forward_attention

2 src-attn

3. FFN

3层DecoderLayer之后

目前的位置：

right_decoder

评论 0