Intella Assist: Choosing a Local Model and Hardware

Selecting a local model or hardware setup for Intella Assist depends on your environment, the types and sizes of items in your case, and the complexity of the tasks you run. Different models behave differently depending on whether items are short or long and whether tasks are simple or complex. Because deployments vary widely, Vound does not recommend specific models or hardware.

Intella Assist integrates with models that expose the same API format as OpenAI (POST /v1/chat/completions). 

The runtime does not matter — the model may run on:

• Ollama

• vLLM

• LM Studio

• llama.cpp

• Local Docker deployments

• Cloud instances

• ...

... as long as the API follows the OpenAI Chat Completions standard.

B.) Structured output (JSON)

Intella Assist Tasks require models that support OpenAI-style Structured output, specifically:

response_format: { "type": "json_schema" }

The model must be able to generate responses that follow the provided JSON Schema exactly.

If your large language model does not natively expose an OpenAI-compatible API, you may optionally use a gateway such as LiteLLM.

LiteLLM is:

• open-source (MIT licensed)

• a lightweight translation layer

• capable of exposing a unified OpenAI-style API

• able to route requests to various backends

• capable of normalizing responses to the OpenAI format

This is just an example; we are not affiliated with LiteLLM.

Gateways are optional and used at your discretion.

There is no single “best” model for all environments. Performance depends on many factors:

• Task type (summarization, extraction, categorization, redaction, etc.)

• Clarity and size of the task definition

• Required context window size (we recommend at least 32k tokens)

• Hardware (GPU type, VRAM, ... )

• Quantization level (Q2, Q4, Q5, Q6, Q8, FP16, etc.)

• Inference runtime (Ollama vs vLLM vs llama.cpp)

• Latency and throughput expectations

• Quality expectations

• Legal and ethical considerations (transparency of the training process incl. what data was used to train the model, bias mitigation strategies, …)

Because all these variables differ across customer deployments, fixed recommendations often do not match real-world usage.

We also do not provide specific model or hardware recommendations because quality is not objectively measurable. Performance metrics such as speed or latency can be benchmarked, but the “quality” of the generated output is inherently subjective and varies with the task, prompt wording, domain, and reviewer expectations.

In addition, Intella Assist Tasks are used for a very broad range of use-cases — from simple extractions to complex analyses — and this diversity makes it impossible to predict which model will perform best in every scenario. A model that excels for one organization’s workload may not produce satisfactory results for another.

Try Models in the Cloud First  

If you do not yet have suitable hardware, evaluating models in the cloud is the safest way to understand real-world performance before investing in equipment.

Examples of platforms that let you select models and hardware (examples only; we do not endorse or partner with these services):  

• RunPod

• Vast.ai

• Lambda.ai

These platforms allow experimentation with different GPU types, quantizations, and model families.

What About Specific Models?  

Already Running a Model?  

If your model supports an OpenAI-compatible API and returns Structured output, you can integrate it with Intella Assist immediately.

If the results are slower or less accurate than expected, you can revisit your setup to improve either performance or quality.

Improving Speed vs Improving Quality  

Depending on what you want to optimize, different strategies apply.  

Conclusion  

Important Notes