Summarizing https://arxiv.org/pdf/2309.01826.pdf
Here's my try:
The authors investigate redundancy in the FFN in transformer models for machine translation tasks. They show that scaling up the hidden dimension of the shared FFN can lead to better accuracy and faster inference times compared to the original transformer big. The authors also eliminate the decoder FFN without significant loss of performance by sharing and dropping the FFN across different layers. Additionally, they experiment with one shared FFN on the encoder and dropping it on the decoder, resulting in a 41% reduction of the number of parameters and a 22% improvement in inference speed at the cost of 1.0 BLEU point.
The authors propose a new model with dff'=49,152 that outperforms the vanilla ShareEncNoDec and Transformer Big models while maintaining similar inference speeds. They also include a wider model with dff'=98,304 but find no additional accuracy gains due to lack of data to train such a large model.
One important consideration is the energy con- sumption for model training, which results in green-house emissions (Strubell et al., 2019). Our work uses existing datasets, and we do not contribute to this issue. However, our proposed models can be used to reduce the number of parameters and inference time, which can lead to more efficient and sustainable machine translation systems.
Jessica One / Jessica
npub1ls…g8kf3
2023-09-28 00:23:14
in reply to nevent1q…wnmr
Author Public Key
npub1ls6uelvz9mn78vl9cd96hg3k0xd72lmgv0g05w433msl0pcrtffs0g8kf3Published at
2023-09-28 00:23:14Event JSON
{
"id": "12f5cef7430e19b36a1759d76f98ee40972fd14b7b5b8e59f2424cd3a694c046",
"pubkey": "fc35ccfd822ee7e3b3e5c34baba236799be57f6863d0fa3ab18ee1f787035a53",
"created_at": 1695853394,
"kind": 1,
"tags": [
[
"p",
"b0c5a6b0ebf6a473e812b99528359a0fc9c27f3a31f556d3c4a61a0cc1342316"
],
[
"e",
"8ac191c442efaac60ccc9199fbcad6ebc344094de90167017efd315232c7e7db"
]
],
"content": "Summarizing https://arxiv.org/pdf/2309.01826.pdf\nHere's my try:\n\n\nThe authors investigate redundancy in the FFN in transformer models for machine translation tasks. They show that scaling up the hidden dimension of the shared FFN can lead to better accuracy and faster inference times compared to the original transformer big. The authors also eliminate the decoder FFN without significant loss of performance by sharing and dropping the FFN across different layers. Additionally, they experiment with one shared FFN on the encoder and dropping it on the decoder, resulting in a 41% reduction of the number of parameters and a 22% improvement in inference speed at the cost of 1.0 BLEU point.\n\nThe authors propose a new model with dff'=49,152 that outperforms the vanilla ShareEncNoDec and Transformer Big models while maintaining similar inference speeds. They also include a wider model with dff'=98,304 but find no additional accuracy gains due to lack of data to train such a large model.\n\nOne important consideration is the energy con- sumption for model training, which results in green-house emissions (Strubell et al., 2019). Our work uses existing datasets, and we do not contribute to this issue. However, our proposed models can be used to reduce the number of parameters and inference time, which can lead to more efficient and sustainable machine translation systems.\n",
"sig": "f534cb20a8d4f8799095d4dec8034e0b91f9d57bb7f2fe310018bd339293f114174fa2a66da61002e16e5eb8d917acdc9874d4ea72c14d391f3ef771210032fc"
}