Class: Vangrail::Rails::Grounding

Inherits:
Vangrail::Rail show all
Defined in:
lib/vangrail/rails/grounding.rb

Overview

Does the answer say only what its passages support.

Safety classifiers score hazards. For a retrieval system the failure that matters is different and they cannot see it: an invented partition name, a quota that appears nowhere, a citation pointing at a passage that does not support the sentence in front of it. Those read exactly like real answers to the person asking, which is the whole problem.

Output side only, and never memoized: the verdict depends on the passage set as well as the draft, so a changed retrieval must be judged again.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(provider: nil, model: nil, chat: nil, policy: nil, name: 'grounding', max_tokens: 256, **chat_options) ⇒ Grounding

Returns a new instance of Grounding.



23
24
25
26
27
28
29
30
31
32
33
34
 
# File 'lib/vangrail/rails/grounding.rb', line 23

def initialize(provider: nil, model: nil, chat: nil, policy: nil,
               name: 'grounding', max_tokens: 256, **chat_options)
  super(name: name, sides: [:output])
  @model = model || provider&.model(:judge)
  @policy = policy || Policies.grounding_policy
  @chat = chat || begin
    raise ArgumentError, 'a grounding rail needs a provider or a chat client' unless provider

    Chat.new(model: @model, base_url: provider.base_url, api_key: provider.api_key,
             max_tokens: max_tokens, **chat_options)
  end
end

Instance Attribute Details

#chat ⇒ Object (readonly)

Returns the value of attribute chat.



21
22
23
 
# File 'lib/vangrail/rails/grounding.rb', line 21

def chat
  @chat
end

#model ⇒ Object (readonly)

Returns the value of attribute model.



21
22
23
 
# File 'lib/vangrail/rails/grounding.rb', line 21

def model
  @model
end

#policy ⇒ Object (readonly)

Returns the value of attribute policy.



21
22
23
 
# File 'lib/vangrail/rails/grounding.rb', line 21

def policy
  @policy
end

Instance Method Details

#cache_key(_text, _context) ⇒ Object

Not memoizable. Stated rather than left to a default so the reason is visible where the decision is.



38
39
40
 
# File 'lib/vangrail/rails/grounding.rb', line 38

def cache_key(_text, _context)
  nil
end

#call(text, context) ⇒ Object 



42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
 
# File 'lib/vangrail/rails/grounding.rb', line 42

def call(text, context)
  passages = Array(context[:passages])
  return unchecked('no passages supplied') if passages.empty?

  answer = chat.ask([
                      { 'role' => 'system', 'content' => policy },
                      { 'role' => 'user', 'content' => Policies.grounding_prompt(text, passages) },
                    ])
  parsed = Parsers.policy(answer.text)
  unless parsed[:decided]
    return Result.new(status: :passed, rail: name, certain: false, model: model,
                      latency_ms: answer.latency_ms, raw: answer.raw,
                      reason: "unparsed judge response: #{parsed[:reason]}")
  end

  return pass(model: model, latency_ms: answer.latency_ms, raw: answer.raw) unless parsed[:violated]

  block(reason: parsed[:reason], categories: parsed[:categories], model: model,
        latency_ms: answer.latency_ms, raw: answer.raw)
end