NUC AI Lab · Local model test plan
One task: how do local, cloud, and hybrid compare?
Before handing part of the work to a NUC, ask whether it works well, what it saves, and what it adds.
Video notes
Some things are easier to show on video.
I'll add walkthroughs and test videos here, alongside written summaries and setup details. The first video is still in preparation.
Start with an everyday task
Use the same documents and questions. The cloud group processes them directly; the local group attempts the task on the NUC; the hybrid group prepares information locally and routes it to the cloud under defined rules. Fix the quality threshold and retain failures, retries, and human revision.
This is a proposed test design. The question is whether a particular task suits local execution, without assuming either deployment approach is inherently better.
Record the hardware and model setup first
NUC; exact specifications to be added
Pin versions before testing
Record the version and startup settings
Keep documents, prompts, and output-length conditions consistent
Check the answer before the speed
- Build a test set with consistent documents and questions. Record sources and usage permissions.
- Define how to score summary accuracy and preservation of context.
- Separate cold-start and warmed-up tests, with a fixed number of repetitions.
- Record end-to-end latency, generation speed, and resource use, keeping every individual run.
- Consider quality and cost together, and record errors and failure cases.
Model comparisons also need output length, context, and quantization details. Numbers measured under different conditions don't make a fair speed ranking.
Record the whole task across all three workflows
Use the same documents and questions. The cloud group processes them directly; the local group attempts the task on the NUC; the hybrid group prepares information locally and routes it to the cloud under defined rules. Fix the quality threshold and retain failures, retries, and human revision.
| Measure | Cloud workflow | NUC local workflow | Hybrid workflow |
|---|---|---|---|
| Result quality and human revision time | To test | To test | To test |
| End-to-end latency, including retries | To test | To test | To test |
| Cloud input / output tokens and number of calls | To test | To test | To test |
| Local peak memory and electricity use | To test | To test | To test |
| API fees, hardware amortization, and maintenance time | To test | To test | To test |
Do not fill in savings percentages before measuring. Even when a workflow is intended to avoid cloud calls, verify network and service records before recording zero. Record model versions and billing dates alongside token and price figures.
What can't this test tell us?
One task cannot represent a model's overall ability, and one hardware setup cannot represent every local deployment. Next, I'll choose test materials and settings that can be shared publicly, run the measurements, and then consider their product implications.