Adding Minimax Music 3, LLMChat (this is the big one!), Image2Text and improvements in MLX performance

e1536fa verified 27 days ago

14.4 kB

LLM Chat: ComfyUI workflows (local)

A multi-turn chat with a language model, inside ComfyUI: the conversation stays between messages, pasted images stay in it for vision models, replies stream in with Markdown and diagrams, and web tools look things up when you allow them. Snippets turn a few characters into a full prompt, *tm rainbows into a tutorial on rainbows. It's the natural follow-up to Image2Text: paste a picture, ask about it, then keep talking.

Highlights

💬 A chat view in ComfyUI — in the App view's middle, and as the 💬 LLM Chat tab of the sidebar in the graph view: type, press Enter, and the reply streams in, drawn as it comes: Markdown, tables and code as soon as each block is complete, and a Mermaid diagram (with zoom, pan, fit and a full view) as soon as its block closes
🖼️ Images in the conversation — paste, drop or attach pictures; vision models (Qwen3-VL, Qwen3.5, Gemma 4…) keep seeing them in follow-up questions
🧰 Web tools that ask first — latest news, web search, reading a page, image and video search, weather, and your approximate location when you switch it on. Every network call shows a card: Allow, Deny or Allow all for this chat. No accounts, no API keys
✦ Snippets — *tm topic for a tutorial, *tms topic and *tmls topic for stories, and *dwys, *i2t, *i2tl, *id to turn an attached image into a text-to-image prompt; *tm alone picks a topic. Edit them or add your own in the chat view
🍏 Quick on a Mac, and the model stays loaded — the MLX version runs on Apple MLX through LVNodes and keeps the model and each chat's context between messages, so a follow-up only reads what's new: on an M2 Ultra, Qwen3-VL 4B answers at about 85 tokens/s and starts a follow-up to a 2,000-token chat in a fraction of a second. ⏏ frees the model's memory whenever you like; the next message loads it again
📅 Knows the date and time — every message carries when it was sent
◼ Stop keeps what came — stop a reply halfway; what was written stays, and so does the model
🕒 Chats you can come back to — a chat lasts until ComfyUI restarts; 💾 Save chat keeps it and its images in ComfyUI's user folder, to reopen from the 🕒 list
🧠 Any language model — Qwen3-VL, Qwen3.5 / 3.6 / 3.8 and Gemma 4 with their own chat templates and native tool calls, Gemma 3 and other LLMs ComfyUI runs, and on a Mac any MLX model folder mlx-vlm or mlx-lm runs, mixture-of-experts models included
🏠 Local only — it runs in your own ComfyUI, on any GPU or an Apple Silicon Mac; Comfy Cloud can't hold a chat (why)

Screenshots

From the Apple MLX workflow on an M2 Ultra, with Gemma 4 E4B (ComfyUI's own gemma-4-e4b-it_bf16 file); the local PyTorch workflow looks the same. Click a picture to open it full size.

The App view: the chat fills the middle, and the app's inputs (Model, System prompt, Max length) are on the right. A 400-word story about a dragon who loves tea, then an ASCII dragon in a code block; with 💭 Thinking on, each reply's thinking streams into the grey block above it. The status line reads Done in 0:03.

Three pictures sent without a message get "Describe what you see". The model is told how many images the message holds, so its thinking and its reply take them one by one, Image 1 to Image 3. Done in 0:10.

✦ Snippets: each one has a name, a Text or Image input, its prompt and, for text ones, the fallback values *name alone picks from. Edits are saved in ComfyUI's user folder.

A *tm tutorial on Python with diagrams while it streams: a Markdown table, a Mermaid flowchart drawn as soon as its block closed (with zoom, Fit and a full view), then the next section and its code. The status line reads Writing the reply · Generating on MLX · 2572 tokens · 62 tokens/s.

The graph view: the Notes explain the setup and every switch, with Load LLM (MLX) under them; the LLM Chat node in the middle holds every field (the message, System prompt, Max length, the Thinking, Tools, Location and Save chat switches, sampling, and the chat it belongs to); on the right, Conversation shows the whole chat as Markdown. The same chat is in the sidebar's 💬 LLM Chat tab.

Two versions

File	Runs on	Loads the model with
`LLMChat-ComfyUI-local.json`	local ComfyUI: NVIDIA, Apple GPUs, CPU (PyTorch)	LVNodes' Load CLIP (keep loaded): `qwen3vl_4b_bf16`
`LLMChat-ComfyUI-local-MLX.json`	local ComfyUI on an Apple Silicon Mac (Apple MLX)	LVNodes' Load LLM (MLX): `qwen3vl_4b_bf16`

The local versions are three nodes: the loader, LVNodes' LLM Chat and a Preview as Text that shows the conversation as Markdown in the graph view. They open in the App view: the chat in the middle, and Model, System prompt and Max length on the right. The chat's Send runs each message, so the App view's Run button and Number of runs are hidden; the status line above them stays and shows what's happening (starting the MLX helper, loading the model, the tokens so far).

There's no Comfy Cloud version: see why.

Install on local ComfyUI

ComfyUI 0.39 or newer.
LVNodes: copy the LVNodes folder into ComfyUI/custom_nodes/ and restart ComfyUI (see the root README; to update, copy what's inside the new LVNodes into your existing one). LLM Chat installs no Python packages.
Model → models/text_encoders/: qwen3vl_4b_bf16.safetensors (8.9 GB). It's the same file as Krea 2's text encoder, so with Krea 2 installed you already have it. When you open the workflow, ComfyUI offers to download it if it's missing. Smaller, for NVIDIA GPUs: qwen3vl_4b_fp8_scaled.safetensors (5.2 GB).
Open LLMChat-ComfyUI-local.json and type in the chat.

Load CLIP (keep loaded) has keep_loaded on, so the model stays in memory between messages whatever ComfyUI's startup flags; the Keep models loaded switch above Run does the same in the App view. Switch it off to free the memory after the next message.

On an Apple Silicon Mac, use the MLX version below: the same model, about 3× faster, and follow-ups don't read the whole conversation again.

Apple MLX version (local, Apple Silicon)

LLMChat-ComfyUI-local-MLX.json swaps the loader for LVNodes' Load LLM (MLX), which runs the model on the Mac's GPU with Apple MLX.

LVNodes, as above. The first MLX run sets up LVNodes' own MLX environment: about 1.3 GB, a few minutes, once. ComfyUI's own packages aren't touched.
Model → models/text_encoders/: qwen3vl_4b_bf16.safetensors (8.9 GB). It's re-quantized to MLX 8-bit as it loads: about 5 GB in memory.
Open the workflow and type in the chat.

Load LLM (MLX) lists ComfyUI text encoders in models/text_encoders (Qwen3-VL 4B / 8B / 32B, Qwen3.5 / 3.6 / 3.8 dense or mixture-of-experts, Gemma 4 E2B / E4B / 31B; bf16, or ComfyUI's int8 / int6 / int4) and MLX model folders in models/LLM (a folder, or a link to one): any model mlx-vlm or mlx-lm runs, with the folder's own chat template. See Image2Text for the full list and their speeds.

Measured in the chat on an M2 Ultra (64 GB), with the tools on; "follow-up" is a short answer after a story, in a chat of about 2,000 tokens:

Model	Load	Memory	Speed	A follow-up
Qwen3-VL 4B, bf16 file (MLX 8-bit), the default	3 s	4.7–5.4 GB	82–88 tokens/s	0.3 s
Qwen3.5 4B, ComfyUI int8 file	4 s	4.8–6.1 GB	88–90 tokens/s	0.5–0.8 s
Gemma 4 26B-A4B (mixture-of-experts), MLX 6-bit folder	10 s	21–22 GB	61–78 tokens/s	0.6–0.8 s
the same, with an image in the chat	6 s	23.6 GB	67–71 tokens/s	2.5 s

Without the kept context, the same Gemma 4 follow-up took 2.3 s, and the gap grows with the conversation. Chats with images on an MLX folder read the whole conversation again each time.

Using the chat

Messages: Enter sends, Shift+Enter starts a new line, and your message shows in the chat at once. ↑ / ↓ go through the messages you sent in this chat; past the newest, ↓ brings back what you were typing. ◼ stops a reply; what came so far stays, marked as stopped. Until the reply starts, the chat shows what's happening: waiting for ComfyUI's queue, starting the MLX helper, loading the model (with a percentage), reading the conversation.
Queue: a message sent while a reply is on its way waits its turn: ⏳ 2 queued above the message box lists each one (📎 for its images), and they go in order. ✕ Clear Queue drops them. ◼ also pauses the queue; ▶ Send Queue carries on (so does sending a new message).
⏏ Eject (top right): frees the model's memory now, and ComfyUI's other loaded models with it; hovering shows what's loaded and how much memory it holds. The next message loads the model again, and the conversation carries on.
Images: paste, drop or 📎 attach them. They stay in the conversation, so you can keep asking about them. Sent without a message, they get "Describe what you see". The model is told how many images each message has, so with several it answers about each one, not only the last.

Snippets: type * to see them. *name value fills the snippet's placeholder with the value; *name alone picks one of its fallback values.

Snippet	With	Asks for
`*tm topic`	text	a tutorial on the topic, at least 1000 words, with examples and analogies
`*tms topic`	text	a story about the topic
`*tmls topic`	text	a very long story, at least 2000 words
`dwys`, `i2t`	an image	a one-paragraph description to use as a text-to-image prompt, about 800 words per image
`*i2tl`	an image	the same, about 4000 words per image
`*id`	an image	the same, with no set length

✦ edits them, adds your own and saves them in ComfyUI's user folder (default/LVNodes/chat_snippets.json).

🧰 Tools (on by default): latest news (Google News), web search (Bing, then DuckDuckGo), reading a page, image search (DuckDuckGo, then Bing), video search (YouTube) and weather (Open-Meteo). The model decides when it needs one; each network call asks first, and Allow all for this chat stops the asking for the rest of that chat.
📍 Location (off by default): lets the location and weather tools look up where you are, from your IP address (ipapi.co or ipwho.is).
💭 Thinking: models that can think, such as Qwen3.5 and Gemma 4, think before they answer. The thinking streams into a grey block above the reply, open while Thinking is on (click it to fold it).
Chats: ＋ starts a new one (so does a message /clear), 🕒 lists them to reopen or delete. A chat lasts until ComfyUI restarts; 💾 Save chat saves the open chat, and the ones after it, in ComfyUI's user folder (default/LVNodes/chats/). Switching it off removes that copy.
System prompt (right panel): extra instructions for every reply, added after the built-in ones. Max length: the longest reply, in tokens (4096 ≈ 3000 words).
The graph view: the same chat is in the sidebar's 💬 LLM Chat tab. The LLM Chat node's outputs, the reply and the conversation as Markdown, can feed other nodes, such as an image prompt.

Small models sometimes need a nudge to use a tool ("search the web for…").

Why there's no Comfy Cloud version

A chat needs things Comfy Cloud doesn't give a workflow:

No chat view. The chat view is a page add-on of LVNodes. Comfy Cloud runs only the node packs it has installed itself and loads no page add-ons, so neither LVNodes' LLM Chat node nor its view can run there.
No memory between runs. Comfy Cloud keeps nothing from one run to the next, so every message would start a new conversation unless you pasted the whole chat back in by hand each time. Models also read a pasted transcript as text to carry on rather than as a conversation, and small ones answer it badly. ComfyUI's own OpenAI ChatGPT node can't keep a conversation either: its persist context option no longer does anything.
No images, tools or saved chats. Images in the conversation, streaming, web tools that ask first and saved chats all run in LVNodes.

For a single question on Comfy Cloud, ComfyUI's own Generate Text node (as in its Qwen3-VL text generation template) answers one prompt per run, and Image2Text has a Comfy Cloud version for describing an image.

Licence

The workflows are part of this project and under the Apache License 2.0. LLM Chat and its chat view are part of LVNodes (Apache-2.0); the view uses marked, DOMPurify and Mermaid, whose licences are in LVNodes/web/lib/LICENSES.md. The models aren't part of the project: Qwen3-VL, Qwen3.5 and Gemma 4 are Apache-2.0, and every other model is under its own licence, which its model card states.