prism-ml

/

Bonsai-8B-gguf

Like 802

Follow

Prism ML 4.6k

Thanks for testing it and glad the speed benefits came through.

On quality, we’d love to better understand exactly what you’re seeing. Local GGUF runs can be sensitive to a few variables: the specific GGUF file, prompt/chat template, LM Studio version, and the llama.cpp backend. One useful sanity check would be to try the model through our demo repository and see whether the behavior persists there: https://github.com/PrismML-Eng/Bonsai-demo/

Separately, Bonsai 8B is optimized for local, low-memory, high-throughput use. For harder reasoning and coding tasks, the gap between small local models and larger models will still show up. That said, we want the out-of-box experience to be strong for general local tasks, so concrete examples would be very helpful for us to reproduce and debug.

Hello @bansaltushar92 ,

I have not checked your demo repository yet. Since I saw that the PR for llama.cpp 1-bit support was accepted, I went straight to testing the model in LM Studio.

I am using LM Studio 0.4.18 with the following backend:

{
  "extension_type": "engine",
  "domains": [
    "llm",
    "embedding"
  ],
  "engine": "llama.cpp",
  "target_libraries": [
    {
      "name": "llm_engine_cuda12.node",
      "type": "llm_engine",
      "version": "0.1.2"
    },
    {
      "name": "liblmstudio_bindings_cuda12.node",
      "type": "liblmstudio",
      "version": "0.2.26"
    }
  ],
  "supported_model_formats": [
    "gguf"
  ],
  "engine_protocol_server": {
    "runtime_kind": "llama-server",
    "executable_relative_path": "llama-server.exe"
  },
  "name": "llama.cpp-win-x86_64-nvidia-cuda12-avx2",
  "version": "2.24.0",
  "platform": "win",
  "cpu": {
    "architecture": "x86_64",
    "instruction_set_extensions": [
      "AVX2"
    ]
  },
  "gpu": {
    "make": "Nvidia",
    "framework": "CUDA",
    "targets": [
      "7.5",
      "8.0",
      "8.9",
      "9.0",
      "10.0",
      "12.0"
    ],
    "minimum_driver_version": "12040"
  },
  "manifest_version": "4",
  "minimum_lmstudio_version": "0.4.0+15",
  "minimum_llmster_version": "0.0.1+9",
  "vendor_lib_package_names": [
    "win-llama-cuda12-vendor-v2"
  ]
}

For inference, I used the recommended parameters from the model card:

Temperature: 0.5
Top-K: 20
Top-P: 0.9
Repeat penalty: 1

My hardware:

GPU: RTX 4070 Ti
RAM: 32 GB DDR5
CPU: i7-13700KF

I tested the model with the following scheduling problem:

You are managing a small compute cluster with 4 machines: A, B, C, D.

You must schedule 6 jobs: J1 … J6.

Each job has:
- a duration in hours
- machine constraints
- dependency constraints

Cluster rules:
- A machine can run only one job at a time
- Jobs start at integer hours
- All jobs start at or after hour 0

Objective:
Minimize total completion time (makespan)

Job definitions:

| Job | Duration | Allowed Machines | Dependencies |
|-----|----------|------------------|--------------|
| J1  | 3        | A, B             | none         |
| J2  | 2        | B, C             | J1           |
| J3  | 4        | A, C             | none         |
| J4  | 1        | C, D             | J2           |
| J5  | 2        | B, D             | J1           |
| J6  | 3        | A, D             | J3, J4       |

Required output format:
- For each job: (job, machine, start_time, end_time)
- Final makespan
- One sentence justifying why this schedule is optimal

The Bonsai 8B response was:

Here is the optimal schedule:

- J1 on A, start at 0, end at 3
- J2 on B, start at 3, end at 5
- J4 on C, start at 5, end at 6
- J5 on D, start at 6, end at 7
- J3 on A, start at 7, end at 11
- J6 on D, start at 11, end at 14

Final makespan: 14 hours

Justification: This schedule minimizes the total completion time by carefully sequencing jobs to avoid conflicts on machines and respecting dependencies. Job J6, which has the longest duration and depends on J3 and J4, is scheduled last on D to ensure it completes after all required jobs. The use of A for J1 and J3, and D for J5 and J6, optimally utilizes the cluster resources without overlap.

However, the expected optimal makespan is actually 9 hours, not 14. Smaller models, such as Gemma 4B, correctly identify the 9-hour schedule.

One valid optimal schedule is:

J1: A, start 0, end 3
J3: C, start 0, end 4
J2: B, start 3, end 5
J5: D, start 3, end 5
J4: C, start 5, end 6
J6: D, start 6, end 9

Final makespan: 9 hours

This makespan is optimal because J6 depends on both J3 and J4, while J4 depends on J2, which depends on J1. Therefore, the critical dependency chain is J1 → J2 → J4 → J6, with total duration 3 + 2 + 1 + 3 = 9 hours. Since no valid schedule can finish earlier than this lower bound, the 9-hour schedule is optimal.

Could you please clarify whether this issue is expected at the current stage of the GGUF / llama.cpp implementation, or whether there are specific inference settings or prompt formatting requirements recommended for this model?

you should test the prompt against the demo repo bc now this thread has me questioning whether I shouldn't use the llama.cpp release

Hey,

Just to update you - I've checked your demo repo and got same results as in LM Studio - fast but not correct.

Thanks for following up and for testing this in the demo repo as well. That is very helpful.

For context, Bonsai 8B is optimized for local, low-memory, high-throughput use. It should perform well on many general chat, instruction following and local tasks, but harder thinking and reasoning-style prompts can still be brittle at this model size. That is broadly consistent with other small local models in this class, including the Qwen3 8B model that Bonsai 8B is based on.

That said, we are actively working on more capable Bonsai models with stronger reasoning, coding, and agentic workflow performance. Please stay tuned.

Now that cpu q2_0 PR had been merged into llamacpp main, will you release other pr soon ? ( Cuda )

Yes, for now waiting for the Metal PR to get merged (they allow only one PR open at a time). Sending Cuda after.

llama.cpp and HF Transformers apply sampling in different order and that results in different token probabilities.
Temperature is first for HF Transformers but last for llama.cpp

To fix that, specify the sequence explicitly: --samplers "temperature;top_k;top_p"