Install Local AI.
Step by step.
Want to install local AI on Android? This practical installation guide walks through the three essentials: a runner app, a model file and settings that suit your device. Start small, test the setup, then scale up when everything works.
☘️ ITC
What you’re building
The Local AI Installation Bible lays out a simple sequence: install a runner app, download a compatible .gguf model, then configure context and performance settings for your Android device.
1. Runner app
The application that loads the model and gives you a chat interface. The source guide recommends ChatterUI for the simpler path, or Termux with llama.cpp for a more hands-on Linux-style setup.
2. Model file
A compatible GGUF file downloaded from a model repository. Its size, architecture and quantisation must be supported by your chosen runner.
3. Loadout settings
Context length, thread count, batch size and battery settings influence stability, speed and memory use.
Check your device first
The source playbook targets Android 9 or later, suggests 6 GB RAM as a minimum and says 8–12 GB is more comfortable. It recommends around 8–10 GB free storage before starting. Actual requirements vary by model and app; a 7B model may need substantially more memory than its file size alone.
Do these checks before installing
Free storage
Open Settings → About phone → Storage. The playbook recommends 8–10 GB free to leave room for a model and overhead.
Use trusted sources
Install runner apps from their official release pages or trusted distribution channels. Check the publisher and permissions.
Review battery rules
Android may restrict background work. If the app is repeatedly killed, check its battery settings and test with restrictions adjusted where your device allows.
Keep it cool
Model loading and inference can use CPU and battery. Charge before longer setup tasks and stop if the device becomes unusually hot.
Security note: installing APKs outside the Play Store requires extra care. Grant “install unknown apps” permission only to the specific source app when necessary, verify the download, and turn off that permission again if you no longer need it. Do not disable broad security protections without a clear reason.
Two ways to get local AI running
Path A — ChatterUI
Best for a straightforward start. The source guide describes ChatterUI as a GGUF front end with model management, sessions, presets and character cards.
1. Open the official release page.
2. Choose the APK asset for the current release.
3. Review the source and permissions before installing.
4. Grant file access only when needed to locate your model.
Path B — Termux + llama.cpp
Best for hands-on users. This route uses a terminal and build tools to compile and run llama.cpp on Android. It is more technical and can take longer.
Termux on F-Droid ↗
Official Termux releases ↗
llama.cpp project ↗
Follow the current installation instructions for your Android version and device architecture. The source guide warns against relying on the outdated Play Store Termux build.
Which should you choose?
If your goal is simply to chat with a local model, start with a compatible GUI app. Choose Termux if you specifically want command-line control and are comfortable troubleshooting build dependencies.
You can explore both later, but prove one path works before adding complexity.
Smoke-test with a small GGUF file
The installation playbook recommends using TinyLlama as a smoke test before attempting a larger model. A small model helps you find out whether the app can load and run a model at all.
Start with TinyLlama 1.1B Chat
The source guide lists a Q4_K_M file at approximately 700 MB. Use the exact repository and filename shown on the model card, and verify the completed download against the listed size.
Browser download workflow
Open model page
Use the repository link and inspect the model card and licence.
Find the file
Open “Files and versions” and locate the exact compatible GGUF quantisation.
Download fully
Use a reliable connection and confirm the download completed successfully.
Verify size
Compare the local file size with the repository listing before loading it.
Large downloads: the source playbook suggests using Termux and wget -c for resumable downloads. Confirm the exact URL and filename from the repository before running a command. Never copy commands blindly from an untrusted source.
What the command-line route involves
The supplied guide’s Termux path sets up shared storage, updates packages, installs build tools, then builds llama.cpp. These are reference commands from the source playbook; check the current project instructions if a command or dependency has changed.
Grant Termux shared-storage access
termux-setup-storageThis creates storage links such as ~/storage/shared and ~/storage/downloads after you grant Android’s permission.
Update packages and install build tools
pkg update -y && pkg upgrade -y
pkg install git cmake clang -yClone and build llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j2Build time and supported options vary by phone and current project version. The source playbook uses -j8; this version shows a lower parallel-job count as a cautious starting point to reduce simultaneous build load on a phone. If the project documentation recommends a different command for your environment, follow that documentation.
Don’t start here unless you want the terminal route
For many beginners, the GUI-app path is simpler. The command-line route is optional — you don’t need to compile software just to try local AI if a compatible runner app already meets your needs.
Begin with conservative settings
The source playbook recommends a modest context length first, then tuning only after the model is stable. The values below are starting points from that guide, not universal best settings for every app or device.
Max context: 2048
Start small. Increase towards 4096 only if the model and device remain stable. Context can add significant memory use.
Threads: 6 (example)
The playbook suggests six threads for an eight-core CPU, leaving some capacity for Android. The best value depends on the processor and runtime.
Batch: test gradually
The source guide suggests 2048, but batch support and memory cost vary. If the app crashes or runs out of memory, lower it and consult the runtime guidance.
GPU offload: start at 0
The playbook recommends CPU-first testing because phone GPU support varies. Try acceleration later only if your app supports it reliably.
Local KV saving
The source suggests leaving “Save Local KV” off while tuning, then enabling it only if you need the session-resume feature and understand the storage/battery trade-off.
Auto-load
If your runner offers an auto-load option, decide whether you want models to load automatically when opening a chat. It may increase startup time and memory use.
Performance tip: the first response can be slower while the model loads. Compare the second or third response before deciding whether a setup is too slow.
When something breaks, isolate the cause
App crashes on load
Reduce context length. If it still crashes, check the model file’s size and completeness, then try the small smoke-test model.
Very slow responses
Try a smaller model, close background apps and review the context and batch settings. Compare later responses after the initial load.
Model not found
Check the file path, storage permissions and whether the app can access the folder where the GGUF file is saved.
Download seems incomplete
Compare the local size with the repository listing. Re-download from the trusted source if the file is incomplete or corrupt.
App closes in background
Review battery optimisation settings for that app. Device-specific settings vary; avoid disabling unrelated security features.
Model answers nonsense
Confirm the model type and file, then test with the small model to help separate a model-specific issue from an app or configuration issue.
The golden diagnostic: test small first
If the tiny test model works, focus on the larger model file, memory demands or settings. If the tiny model also fails, investigate the runner app, permissions and setup before downloading more models.
One model isn’t always enough
The source playbook proposes a small collection of models with different strengths. Store compatible files in one organised folder, then switch between them through your runner app rather than repeatedly downloading the same model.
Featherweight
A 1B–2B model for smoke tests, short prompts and lighter tasks.
Quick thinker
A 3B–4B model for everyday questions when your device has enough memory.
Deep thinker
A 7B-class model for more demanding prompts if your device can run it comfortably.
The WispOS connection
The playbook frames model switching as a manual version of a future fetch-on-demand design: use a small model for quick work and load a larger one when a task calls for it. This is a design concept, not a claim that a completed WispOS product is available.
Test the privacy promise for yourself
A model can run on-device after download, but local inference alone doesn’t prove that every app component is offline or telemetry-free. If privacy matters, check the runner’s documentation, permissions and network behaviour.
Download first
Install the runner and model from sources you trust.
Load the model
Confirm the runner can open it and respond to a test prompt.
Go offline
Enable airplane mode and retry a simple prompt after the model has loaded.
Review evidence
Check app documentation and settings before concluding that all network activity is disabled.
Offline operation for a local model does not mean model downloads are offline, and apps may behave differently. Do not enter sensitive data into a tool until you understand its data handling.
Local AI installation: quick answers
Do I need Termux to run local AI?
No. The source playbook presents ChatterUI as the simpler option and Termux plus llama.cpp as a more technical route. Choose a runner compatible with your model and device.
How much RAM do I need?
The source guide suggests 6 GB as a minimum and 8–12 GB as more comfortable. Actual requirements depend on model size, context length, runtime and what else is running.
Why start with TinyLlama?
A small test model helps verify that the app, model format and storage permissions work before you commit time and storage to a larger download.
Why is the first reply slow?
The model may need to load into memory first. The source guide recommends judging speed from the second or third message as well as the first.
Does running locally guarantee privacy?
No. Local model inference can reduce cloud dependence, but the app may have network features or telemetry. Verify offline behaviour and app privacy documentation yourself.
Start small. Verify. Load. Learn.
Pick a runner, test with a lightweight model, use cautious settings and expand only when your device is stable. A reliable setup beats a huge model that won’t load.
Explore ChatterUI releases ↗Irish Tech Connect · Practical technology. Smarter digital choices. ☘️
Use current project documentation
This feature is based on the supplied “The Local AI Installation Bible” file. Commands and setup recommendations may change as Android, Termux, ChatterUI and llama.cpp evolve. Check current official documentation before following installation steps.
- ChatterUI releases
- Termux on F-Droid
- Termux official releases
- llama.cpp project
- TinyLlama GGUF model repository
Disclaimer: This is an educational guide. Sideloading apps, building software and downloading model files involve security, compatibility and licensing considerations. Use trusted sources, inspect permissions and make backups before changing device settings.
Keep building your digital toolkit
Local AI Models on Your Phone
Compare mobile-friendly model sizes, quantisation and model choices in our first AI Toolkit feature.
Website Maintenance & Care
Learn how updates, backups, security checks and performance care support a healthy WordPress website.
Irish Tech Connect
Practical technology guidance and digital services, built for real-world needs.