Module: Vangrail::Providers::Llmlite

Defined in:
lib/vangrail/providers/llmlite.rb

Overview

llmlite: a local OpenAI-compatible proxy, and the default this gem builds around.

It is the right default for guardrails specifically. Rails run on every turn, so their latency and their failure modes are the application's; a local endpoint keeps both on this machine, needs no shared credential, and cannot bill anyone. It also means a laptop with the proxy running has working rails with nothing configured.

The proxy serves an instruct model, not a safety classifier, so model(:guard) is nil and the builder puts a policy rail on the input side. That is a real difference between endpoints, and it belongs here rather than in a rail deciding what it is talking to.

Constant Summary collapse

HOST = 
'127.0.0.1'
DEFAULT_PORT = 
8760
DEFAULT_KEY = 
'grok-inside'
DEFAULT_MODEL = 
'grok-4.5'

Class Method Summary collapse

Class Method Details

.base_url(env = ENV) ⇒ Object 



37
38
39
 
# File 'lib/vangrail/providers/llmlite.rb', line 37

def base_url(env = ENV)
  "http://#{host(env)}:#{port(env)}/v1"
end

.embed_model(env = ENV) ⇒ Object

No default. Which embedding model a proxy serves, if any, is deployment knowledge, and a guessed name costs a 404 on every check while looking like a rail that ran.



58
59
60
61
 
# File 'lib/vangrail/providers/llmlite.rb', line 58

def embed_model(env = ENV)
  value = env['LLMLITE_EMBED_MODEL'] || env['GUARDRAILS_EMBED_MODEL']
  value.to_s.strip.empty? ? nil : value.strip
end

.host(env = ENV) ⇒ Object 



33
34
35
 
# File 'lib/vangrail/providers/llmlite.rb', line 33

def host(env = ENV)
  env['LLMLITE_HOST'].to_s.strip.empty? ? HOST : env['LLMLITE_HOST'].strip
end

.key(env = ENV) ⇒ Object 



63
64
65
 
# File 'lib/vangrail/providers/llmlite.rb', line 63

def key(env = ENV)
  env['LLMLITE_API_KEY'] || DEFAULT_KEY
end

.listening?(env = ENV) ⇒ Boolean

A TCP connect, not a request: a proxy that is not running is the common case, and finding that out must cost microseconds rather than a timeout.

Returns:

  • (Boolean) 


43
44
45
46
47
48
49
 
# File 'lib/vangrail/providers/llmlite.rb', line 43

def listening?(env = ENV)
  socket = TCPSocket.new(host(env), port(env))
  socket.close
  true
rescue StandardError
  false
end

.model(env = ENV) ⇒ Object 



51
52
53
 
# File 'lib/vangrail/providers/llmlite.rb', line 51

def model(env = ENV)
  env['LLMLITE_MODEL'] || env['GROK_LLMLITE_MODEL'] || DEFAULT_MODEL
end

.port(env = ENV) ⇒ Object 



29
30
31
 
# File 'lib/vangrail/providers/llmlite.rb', line 29

def port(env = ENV)
  (env['LLMLITE_PORT'] || env['GROK_SHIM_PORT'] || DEFAULT_PORT).to_i
end

.provider(env = ENV) ⇒ Object 



67
68
69
70
71
72
73
74
75
76
 
# File 'lib/vangrail/providers/llmlite.rb', line 67

def provider(env = ENV)
  Provider.new(
    name: 'llmlite',
    base_url: base_url(env),
    models: { judge: model(env), guard: nil, embed: embed_model(env) },
    key_resolver: -> { key(env) },
    local: true,
    probe: -> { listening?(env) },
  )
end