The Viral Bilibili 'Big Fat Fish': A Build Guide for a QQ Chatbot with Local Memory and Swappable Voices

Starting with 'Big Fat Fish': It's More Than Just Replying to Messages

Recently, the "DeepSeek^Ad fish-raising"玩法 (a playful trend of giving a model a persona) has been gaining traction on Bilibili—giving the model a persona, connecting it to QQ, letting it chat, remember things, and even sing. Various repos, rule sets, and玩法 versions have already appeared online. But before starting, this UP主 (content creator) first pointed out a problem: most versions only solve the single issue of "being able to reply to messages." Once you use it as a long-term chat companion, its shortcomings become obvious.

It forgets things after chatting for a while. To make the model remember history, you can only keep stuffing chat logs into the context—the more you stuff, the longer it gets, and the calling cost rises accordingly. Although DeepSeek is relatively cheap, high-frequency daily chatting still adds up to an expense. In addition, fixed voice timbre, making it sing a segment, and freely switching between text and voice—these can't be achieved just by tweaking the persona.

So the UP主 reorganized the entire workflow according to his own needs. It should be noted that he explicitly stated this is not taking someone else's pre-built bot, changing the persona and renaming it: the underlying tools, connection protocols, and models can of course be referenced, but how QQ messages are received, how text and voice are switched, how models and voices cooperate, how translation and memory are saved, and how song requests work—these steps were organized and connected by himself.

Architecture Breakdown: One Listens, One Thinks, One Speaks

The core idea of this system is "division of labor," not letting one model do everything.

Message Pipeline: QQ as the Entry Point, a Middle Layer for Scheduling

The UP主 mainly uses DeepSeek Harness (DSH for short). He suggests not being intimidated by the name—think of it as "the layer that organizes models and chat tasks." QQ is the entry point; components and interfaces like NapCat and OneBot handle message ingress and egress; the self-built workflow in the middle decides "what is received, who it's handed to, and in what form to reply."

He emphasizes that just because others provide useful underlying tools doesn't mean the whole system is built for you. Everyone's repo is different, message formats differ, and the handling of role rules, permissions, voice, and memory also varies. Copying someone else's repo together and expecting it to do everything is unrealistic—the real brainwork is making the parts cooperate according to your needs.

What Are OneBot and NapCat?

OneBot is an application-layer protocol standard widely used in the Chinese QQ bot community. It abstracts "sending and receiving messages" into unified HTTP/WebSocket interfaces, so that upper-layer business logic doesn't have to directly handle the QQ client's underlying protocol. NapCat is a concrete implementation end of the OneBot protocol (based on the NTQQ client), responsible for logging in, maintaining connections, and pushing message events to callers. Together they act as a "message gateway": interfacing with QQ on one side and converting messages into program-readable JSON events on the other. This way, the middle-layer self-built workflow only needs to subscribe to events and call send interfaces to achieve text/voice switching and scheduling, without reverse-engineering the protocol itself. Understanding this layer explains why voice and memory modules can be retained as-is when changing entry points (e.g., to a desktop character).

Voice Pipeline: Local Transcription + Local Synthesis

When you send a voice message, the system doesn't throw it directly to a text model and expect it to "magically understand." The process is: first use Whisper to convert the voice to text locally, then hand it to the chat model for understanding and generating a reply, then hand the reply to a local voice model to speak it out, and finally send it back to QQ. Switching between text and voice modes and language preferences can be retained once set—no need to re-teach every sentence.

For voice, the UP主 mainly uses VoxCPM2, while also keeping Qwen3-TTS as a comparable alternative route. You can swap in whatever voice you like without rebuilding the entire chat system just to change voices.

The Roles of Whisper and TTS

Whisper is an open-source speech recognition (ASR) model from OpenAI that supports multilingual transcription and can run locally without internet calls, making it suitable for handling private chat voice and avoiding per-use billing. Its text output is then handed to the chat LLM, which is only responsible for semantic understanding and generating replies—it doesn't directly "hear" audio. To turn text replies back into voice, a text-to-speech (TTS) model is needed, such as the VoxCPM2 and Qwen3-TTS mentioned in the article, which synthesize text into waveforms of a specified timbre. The whole pipeline is a serial ASR→LLM→TTS structure, where each link can be replaced independently: changing the recognition model doesn't affect reply logic, and changing TTS doesn't affect memory. This separation is also the technical prerequisite for freely switching text/voice modes and saving language preferences.

Voice Cloning: Reference Cloning and Fine-Tuning Are Two Different Things

When it came to voices, the UP主 didn't go overboard: he found an anime video on Bilibili, extracted a few relatively clean voice segments, and after some selection ended up using about five segments as reference tests—not hours of training data, nor training some super model from scratch.

He also clarified two commonly confused concepts: giving a voice model a reference audio clip and having it say new sentences in that timbre is reference cloning; actually taking a batch of data to do LoRA fine-tuning and train new weights is something else entirely. For those who want to try it themselves, it's best to choose clean single-speaker material with little background noise—the more suitable the material, the more玩法 (applications) you can try.

Singing: Timbre Conversion, Not Live Generation

The song-request feature also needs clear boundaries: this is not a text-to-speech model reading lyrics aloud, nor is it generating a whole new song from scratch every time you request one. The UP主 uses VoxCPM2 to process existing vocal performances, converting the selected timbre over, keeping the good-sounding segments and storing them in a local playlist—what plays later when you request a song is the already-processed version. What songs go in the playlist, how long to keep them, and which segment to choose are all left to the user.

Why LoRA Fine-Tuning Is Different

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that inserts small trainable low-rank matrices alongside model weights. After training on a batch of data, new adapter weights are obtained, thereby changing the model's overall behavior or timbre distribution. Reference cloning doesn't change weights at all: it only takes a reference audio clip as conditional input and lets the model imitate that timbre—this is a zero-shot/few-shot capability at inference time. The difference: reference cloning has a low barrier, works with a few minutes of material, and swapping timbres only requires swapping the reference audio; LoRA fine-tuning requires large-scale clean data, training time, and compute, and produces reusable weight files with potentially better timbre stability but less flexibility. The article's choice of five reference audio clips for testing is a typical use of reference cloning, and also explains why "changing voices doesn't require rebuilding the entire chat system."

Memory Is the Key: Local SQLite + Summaries + Freedom to Switch Models

The UP主 admits that what he values more is memory—the most troublesome thing about raising a fish isn't whether it will call you master today, but whether it still remembers past events after chatting for a long time.

Why Not Keep Stuffing Chat Logs into Context

An intuitive analogy: with every sentence you say, the model carries a huge stack of previous chat logs along with it. At first it's fine, but the longer you chat, the more unwieldy it gets—models have context length limits, and calls aren't free. You're clearly only asking what to eat today, yet you still have to bring along last week's small talk—obviously not cost-effective.

Implementing Layered Memory

He stores memory separately on the local machine, mainly using SQLite to save chat logs, key information, and the connecting content between conversations. After chatting to a certain extent, the things truly worth keeping are extracted and organized into shorter records, and then a new session is started. The key point is: the new session doesn't rely on the model magically recalling what happened before—necessary materials are handed to it; when a specific event from long ago is needed, it's retrieved from local memory and given to the model.

This summary is mainly organized locally based on verbatim excerpts, without needing an extra LLM call for each summary—reducing unnecessary context and saving on calling costs. For example, if you previously explicitly said you don't like cilantro, later when chatting about ordering takeout, this information might be retrieved from local memory. What's wanted isn't every word ever said permanently sitting in the chat box, but being able to find the relevant entry when truly needed.

Context Windows and Summary Compression

The context window of an LLM refers to the upper limit of tokens a single call can receive; anything beyond is truncated or causes an error. APIs usually bill by input tokens, so the longer the history, the higher the per-call cost. Therefore, repeatedly stuffing all chat logs into context is both length-limited and continuously expensive. The idea of summary compression: when conversation accumulates to a certain amount, distill the information worth keeping long-term (preferences, facts, agreements) into short text stored in a local database, and new sessions only load summaries and retrieved relevant entries. This is similar to a simplified version of retrieval-augmented generation (RAG)—not aiming to "remember every word," but to "be able to look up that one entry when needed." Since summaries are verbatim excerpts without LLM calls, it saves both tokens and extra inference overhead.

Switching Models Without Losing Memory

Another benefit of storing memory locally is freedom to switch models. Big Fat Fish currently mainly runs on DeepSeek, but the UP主 doesn't want it bound to one name—in the future, connecting Kimi or other suitable models, whichever is more cost-effective and fits the needs, is possible, and different APIs and subscription packages can be compared. After switching models, speaking habits, tone, and even personality expression may change a bit, but the memory database is local and character data remains, so there's no need to throw away previous chat logs just because you switched providers.

The difference from simply writing a long persona for DeepSeek is exactly this: persona is persona, model is model, memory is memory—only after separating them do you have choices. As for extensions like little emotions, small actions, and expression reactions, MODs and plugins have already been shared online; the UP主 didn't specifically include them in this version, believing such features are easy to extend and there's no need to lock everything down at once.

How the Download Packages Are Split: Universal Voice Pack and QQ Workflow Pack

The shared content is split into two packages, both in the description:

• Universal Voice Pack: If you already have your own QQ bot, or later want to connect the voice to a desktop character or other chat entry points, you can keep your existing system and just use this set of interfaces and methods for voice recognition, voice generation, training, and singing voice conversion.
• QQ Connection and Character Workflow Pack: Strings together QQ messages, DSH, text-voice switching, local memory, and song requests into a working workflow, then pairs with the voice pack.

The benefit of this split is not having to overturn everything when changing one part: switch to a non-QQ entry point and the voice pack stays; change voices and memory stays untouched; change chat models and the original records need not be deleted. Each part only handles its own piece, making maintenance easier later. The shared package won't include the UP主's QQ account keys, personal chats, or private character data, nor will the anime voices and others' songs used for testing be directly packaged—what you get is reusable methods and programs; materials must be prepared and configured yourself.

The UP主 also reminds: use it properly, don't do illegal things, and especially don't clone others' voices to impersonate them without permission; get consent before using real people's voices, and check usage authorization for songs and video materials. He also mentions a more meaningful use: if a family member has passed away and left legally usable recordings, and the family is willing, this method can be used for commemoration—but when presenting it, make clear that it's an AI-synthesized voice, not the person themselves responding.

Teach a Man to Fish: From One Fish to a Whole Workflow

The UP主 gives "teach a man to fish" a literal double meaning: this episode does indeed prepare the fish and share it with everyone, but what he wants more is to teach how to make it yourself, and then swap in a different character or task to make something of your own.

From this he introduces the direction for the next episode: since the model handles thinking, the program handles sending and receiving messages, the voice model handles speaking, and the database handles memory—no one needs to do everything—can the same approach be used for something else? People have been asking in the background about those AI comic dramas on Bilibili—cats telling stories, Liaozhai animations, dinosaurs fighting, toy stories—how can someone make so many types at once, what exactly is the platform doing for them, and can this workflow be built without ready-made everything? This question is much bigger than raising a fish: if functions can be linked together today, then tomorrow the video production process can also be broken apart, with different Agents dividing the work—some organizing stories, some generating assets, some checking results, some handling transitions—and in the end you only need to tell one manager clearly what you want.

The longer-term vision is to move toward desktop characters: not just a message in QQ, but with its own appearance, actions, and animations, visible on the computer desktop, able to talk to it, with swappable voices, retained memory, and a chat model you choose yourself. The UP主 says the first version stops here—it's already quite usable for his own needs—and appearance animations and such can be extended when there's demand; he doesn't plan to keep spending time on the same thing just to pile on features.

One-line Summary

A Bilibili creator breaks down a self-built QQ 'fish-raising' bot: ASR+LLM+TTS division of labor, local SQLite layered memory, reference-cloned timbres, and timbre-conversion singing.

Paragraph Summary

The article presents a Bilibili UP主's complete approach to building a "DeepSeek fish-raising" QQ chatbot. Addressing common issues—bots that only reply, forget after long chats, and inflate context leading to rising costs—he reorganizes the workflow by division of labor: QQ connects via NapCat/OneBot, Whisper transcribes voice locally, the chat model handles understanding and generation, and TTS like VoxCPM2 handles speaking, enabling free text-voice switching. Memory is stored separately in local SQLite, summarized on demand, and supports switching models without losing history. Voices use reference cloning rather than LoRA fine-tuning, and singing is timbre conversion of existing performances. The share is split into a universal voice pack and a QQ workflow pack, with emphasis on cloning authorization compliance, and a vision of applying the same division-of-labor idea to AI video and desktop characters.