Wan Musa
All projects
Self-hosted LLMDeployed - personal VPS

VPS-Hosted Gemma 3

A shared Gemma 3 inference service on my VPS, exposing authenticated APIs for structured job extraction and lightweight assistant tasks.

View on GitHub

Problem & context

Different applications need model inference, but each use case has different prompts, output formats, and generation settings. Duplicating the model service per application adds operational work.

My contribution

I built a FastAPI service around a shared Gemma 3 backend, separated task-specific prompts into profiles, and set up Docker deployment behind the platform's HTTPS reverse proxy.

Workflow

  1. Application request

    Job extraction or helper task

  2. Authenticate request

    Bearer token

  3. Select task profile

    Prompt and generation settings

  4. Run Gemma 3

    Ollama, OpenAI-compatible API

  5. Parse model output

    Structured fields and warnings

  6. Return task response

High-level workflow.

Technology

  • Python
  • FastAPI
  • Gemma 3 4B
  • Ollama
  • Pydantic
  • Docker Compose
  • nginx
  • Oracle VPS

Key design decisions

One backend, separate task profiles

Job extraction and helper tasks share the model backend but have their own prompts and generation settings. New use cases can be added without standing up another inference service.

A stable interface to the model

The service talks to an OpenAI-compatible chat-completions endpoint. This keeps application code separate from the model runtime and allows compatible backends to be configured independently.

Treat model output as uncertain

Responses are parsed and normalised, with warnings for malformed or incomplete output. Tolerant parsing helps callers inspect a result, but does not make generated information authoritative.

Capabilities & limitations

Implemented

  • VPS-hosted Gemma 3 inference through authenticated endpoints
  • Separate profiles for structured job extraction and assistant tasks
  • Configurable generation settings, input limits, and timeouts
  • Parsed outputs with warnings and explicit backend-error responses

Limitations

  • A single VPS and model backend constrain throughput and availability
  • Model-generated fields still need review; parsing is not factual validation
  • The current Chrome capture extension works offline and does not call this service

Have a workflow worth automating?

I'm interested in AI-focused roles and conversations about building useful, practical agentic AI systems.

Email meBack to projects