eLLM: Run Long-Horizon Inference Faster on CPUs Than on GPUs
-
Updated
Sep 5, 2026 - Rust
#
llm-infernece
Here are 6 public repositories matching this topic...
Efficient LLM inference on Slurm clusters.
inference speech-to-text vlm text-embedding multimodal audio-transcription llm vllm reward-model llm-infernece sglang llm-infrastructure
-
Updated
Sep 7, 2026 - Python
Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability.
-
Updated
Mar 9, 2026 - Python
⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.
macos rust metal self-hosted quantization mlx inference-engine kv-cache apple-silicon openai-api int4 llm local-llm local-ai ollama qwen llm-infernece on-device-inference ollama-alternative llama-cpp-alternative
-
Updated
Sep 7, 2026 - Python
A low-overhead process-pacing tool for Apple Silicon that delays thermal buildup during sustained local LLM inference through process-level duty cycling.
macos macbook pwm thermal-management apple-silicon llama-cpp ollama llm-infernece omlx gpu-duty-cycling
-
Updated
Aug 23, 2026 - Python
Add this topic to your repo
To associate your repository with the llm-infernece topic, visit your repo's landing page and select "manage topics."