Search
Glossary4
Open the full glossary- InferenceRunning an already-trained model to produce output — the counterpart of training, and the entirety of the computation that happens when you use a model.
- Model RuntimeA Model Runtime loads a model and exposes an inference API.
- CPU inference (running a model without a GPU)Running a model on the CPU alone, with no GPU. For small models, low volume and offline batch work it is genuinely usable; for large models or interactive use it is essentially hopeless.
- Token Cost / Inference CostToken cost / inference cost is input usage × input price + output usage × output price.
Found 5 articles
After You Press Enter: From 0.03 s to 0.85 s, What Your Question Goes Through
One real API call dissects the full chain of a question: 35 ms to establish the connection, microseconds to tokenize, 474 ms for the model to read your whole question, 380 ms generating it token by token — 854 ms in total. Includes the complete reproducible measurement method, and one counterintuitive conclusion: 96% of the time is not in the network but inside the model.
Hallucination Risk in Finance: The Three Most Expensive Mistakes
The cost of AI hallucination in financial scenarios is severely underestimated. This article dissects the three most expensive mistakes — fabricating numbers, confusing periods and companies, and filling gaps with inference — explains what causes each one, and gives four implementable lines of defense: mandatory citation, separate statistics, low-confidence abstention, and human review.
Comparison of Other Inference Frameworks: A Complete Guide to llama.cpp, MLX, and ExLlamaV2
An in-depth comparison of three major local LLM inference frameworks: llama.cpp (GGUF), Apple MLX, and ExLlamaV2, covering installation, performance, and use cases.
LiteLLM Proxy Multi Model Routing in Practice
Use LiteLLM Proxy to centrally manage multiple LLM backends such as DeepSeek, ModelStudio, Ollama, and more: load balancing, cost tracking, fallback configuration.
Complete Guide to vLLM Deployment and Performance Tuning
vLLM high-performance LLM inference engine deployment guide: installation, configuration, PagedAttention principles, performance benchmarking.