T5

Modern LLMs (such as the GPT series, Claude, and Llama) are all based on the Transformer architecture, and their decoding process is essentially a Markov chain.

T5 was proposed by Google in 2019, unifying all NLP tasks into a text-to-text format.

BERT uses an encoder architecture, the GPT series uses a decoder architecture, and T5 and BART use an encoder-decoder architecture

BERT was proposed by Google in 2018 and is a bidirectional encoder; GPT was proposed by OpenAI in 2018 and is an autoregressive decoder.

Text models supported by the Transformers framework include GPT, BERT, LLaMA, and T5; vision models include ViT, CLIP, and Stable Diffusion; audio models include Whisper and Wav2Vec; and multimodal models include LLaVA and Qwen-VL.

Encoder-only architectures are suited for understanding tasks, decoder-only for generation tasks, and encoder-decoder for sequence-to-sequence tasks like translation

Transformers come in three variants: encoder-only (like BERT), decoder-only (like GPT), and encoder-decoder (like T5)

The text encoder required by this solution is about 14 GB in size, and a larger T5-series encoder is usually used to support more complex prompt understanding.

When T5 does summarization it typically uses abstractive generation, i.e. reorganizing the language rather than extracting original sentences, so it must work in tandem with transcription quality—typos and spoken-word redundancy in the transcription directly affect the coherence of the summary.

AccessLearnAI's pipeline consists of four stages: Whisper handles speech-to-text, T5 handles summary compression, edge-tts converts the summary back to speech, and finally the text is converted to braille.

T5 is a unified text-to-text framework proposed by Google that formulates tasks such as summarization, translation and classification as 'input text → output text'; the largest original version has about 11 billion parameters, while smaller variants such as T5-small, T5-base or Flan-T5 are commonly used in actual deployments.

FastModel can now fine-tune text encoder-decoder models such as T5, T5Gemma, BART, and Marian

The Transformer architecture is the foundation of modern large language models such as BERT, T5, and LLaMA.

Compared to pure decoder architectures (such as GPT), GLM achieves an average improvement of 8–15 percentage points on Chinese NLU tasks; compared to encoder-decoder architectures (such as T5), inference efficiency is improved by approximately 40%.

Text encoders such as CLIP or T5 tend to encode numerical concepts as vague quantities rather than precise counting instructions

After the backend was deployed to Render's free tier, out-of-memory (OOM) errors occurred repeatedly, because Whisper and T5 were loaded into memory at the same time and their combined size exceeded the free tier's limit.