[2027 Beta] Catalyst – Ask AI popup silently fails to open with Ollama (context window too small)

Summary
With an Ollama model provider, clicking any purple Ask AI icon in the Designer does nothing. No popup and no error message on screen. The cause only appears in the Designer console log: the Designer assistant’s system prompt and tool definitions don’t fit in the provider’s Max Context Tokens.

Environment

  • Ignition: 2027 Beta, build [from Gateway → Status → Overview]
  • Modules: Catalyst Core + Catalyst Agent
  • OS: Windows 11
  • Model provider: Ollama [version, e.g. 0.40.16], same machine as Gateway
  • Model: qwen3:4b-instruct-2507-q4_K_M
  • Hardware: Intel i5-1235U, 8 GB RAM, no dedicated GPU (CPU-only inference)

Steps to reproduce

  1. Create an Ollama model provider with Max Context Tokens = 8192.
  2. Select it in the Designer (Ask AI menu at the bottom of the window).
  3. Click any Ask AI icon, e.g. in a Project Library script.

Expected
The chat popup opens, or a visible message explains why it can’t.

Actual
Nothing happens. The Designer console shows:

WARN AssistEmbedFactoryImpl -- Cannot start assist session for provider 'Catalyst_provider'
ProviderConfigException: The selected model's 8,192-token context window is too small
for this agent. Its system prompt (5,075 tokens) and tool definitions (5,444 tokens),
plus a 4,096-token reserve for the response, leave no room for a conversation.

At Max Context Tokens = 16384 it still fails, because the response reserve grows to 8,192:

The selected model's 16,384-token context window is too small for this agent.
Its system prompt (5,070 tokens) and tool definitions (5,444 tokens),
plus a 8,192-token reserve for the response, leave no room for a conversation.

Observations

  • The Designer assistant needs ~10,500 tokens of built-in prompt and tools before any user input.
  • The response reserve appears to scale with the context (4,096 at 8k, 8,192 at 16k), so it seems to need roughly 32k in practice.
  • On CPU-only hardware, a 32k context isn’t feasible (memory), so the Designer assistant can’t be used with local Ollama on modest machines.
  • My own Catalyst Agent (one small tool) works fine at 8192 via system.ai.sendMessage and the Perspective AI Chat component.

Suggestions

  1. Show the error in the Designer UI instead of failing silently.
  2. Document the minimum Max Context Tokens needed for the Designer assistant.
  3. Consider a lighter assistant mode (fewer tools or a smaller reserve) for small local models.

Logs attached [Designer console excerpt, with tokens and IPs removed].

We'll work on the error messaging here, but generally we're going for the pragmatic view that local models are supported on a 'best effort' basis. By which I mean - we're going to provide the hooks in the gateway and designer to connect to your open model provider, but it's up to you to find a runtime that's sufficiently capable. Restricting tools or cutting down the system prompts means taking away the things that make Catalyst actually useful over just leaning on the base model's own intelligence.

We're very interested in the long term story for local models once hardware prices get less ridiculous to run them at reasonable speeds. For now, pragmatism suggests we focus most of our attention on the cloud providers who are lighting money on fire offering incentivized pricing to get folks in the door.

1 Like

I'm considering a beefy new desktop/GPU combination in spite of the price, due to the uncertainty of API costs. If you screw up with local model, you tie up your machine. If you screw up with a commercial model, you can get badly whacked.

2 Likes

I have a Framework desktop with the Ryzen 495 and 192GB ram due to arrive this month (supposedly...).

I'll be trying to find some time to test local models with Catalyst.

7 Likes

This is starting to sound like everyone is buying a new gaming rig for Christmas and writing it off as a “Business expense”… I am jealous.

3 Likes

I was thinking of getting a new GPU for Christmas but after comparing the prices to msrp I decided my RX 570 can last a year or two longer.

brb, setting up a LLC to expense my 5090 rig :laughing:

2 Likes

I am wondering myself if something like the AMD Ryzen AI Halo (128GB unified memory) would be enough to do simple agents. We have a local AI server with 2 ADA 4000 GPUs and a Ryzen Thread Ripper 32 core, and 512 GB RAM (purchased before the RAM craziness).

It really does seem like it can do the proof of concept at least with a 27B parameter model. I'm not looking for frontier level with this use case. Just something that can do a RAG search over thousands of pages of data. The other use case I would want to try is building trend groups or dashboards the user wants via a prompt. Seems like pretty modest hardware can pull that off. :person_shrugging:

1 Like