Class: Vangrail::Rails::Grounding

Inherits:
Vangrail::Rail show all
Defined in:
lib/vangrail/rails/grounding.rb

Overview

Does the answer say only what its passages support.

Safety classifiers score hazards. For a retrieval system the failure that matters is different and they cannot see it: an invented partition name, a quota that appears nowhere, a citation pointing at a passage that does not support the sentence in front of it. Those read exactly like real answers to the person asking, which is the whole problem.

Output side only, and never memoized: the verdict depends on the passage set as well as the draft, so a changed retrieval must be judged again.

Constant Summary collapse

DEFAULT_MAX_TOKENS =

1200 rather than 256. A reasoning model writes its working before its verdict, and 256 bought the working: scored against RAGTruth, 125 of 300 checks returned no verdict at all, every one of them a judge that was still thinking when the budget ran out. 54 of those were responses that really were hallucinated, and an unchecked result is not a catch.

1200

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(provider: nil, model: nil, chat: nil, policy: nil, name: 'grounding', max_tokens: DEFAULT_MAX_TOKENS, **chat_options) ⇒ Grounding

Returns a new instance of Grounding.



30
31
32
33
34
35
36
37
38
39
40
41
 
# File 'lib/vangrail/rails/grounding.rb', line 30

def initialize(provider: nil, model: nil, chat: nil, policy: nil,
               name: 'grounding', max_tokens: DEFAULT_MAX_TOKENS, **chat_options)
  super(name: name, sides: [:output])
  @model = model || provider&.model(:judge)
  @policy = policy || Policies.grounding_policy
  @chat = chat || begin
    raise ArgumentError, 'a grounding rail needs a provider or a chat client' unless provider

    Chat.new(model: @model, base_url: provider.base_url, api_key: provider.api_key,
             max_tokens: max_tokens, **chat_options)
  end
end

Instance Attribute Details

#chat ⇒ Object (readonly)

Returns the value of attribute chat.



21
22
23
 
# File 'lib/vangrail/rails/grounding.rb', line 21

def chat
  @chat
end

#model ⇒ Object (readonly)

Returns the value of attribute model.



21
22
23
 
# File 'lib/vangrail/rails/grounding.rb', line 21

def model
  @model
end

#policy ⇒ Object (readonly)

Returns the value of attribute policy.



21
22
23
 
# File 'lib/vangrail/rails/grounding.rb', line 21

def policy
  @policy
end

Instance Method Details

#cache_key(_text, _context) ⇒ Object

Not memoizable. Stated rather than left to a default so the reason is visible where the decision is.



49
50
51
 
# File 'lib/vangrail/rails/grounding.rb', line 49

def cache_key(_text, _context)
  nil
end

#decide(text, context) ⇒ Object 



53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/vangrail/rails/grounding.rb', line 53

def decide(text, context)
  passages = Array(context[:passages])
  return unchecked('no passages supplied') if passages.empty?

  answer = chat.ask([
                      { 'role' => 'system', 'content' => policy },
                      { 'role' => 'user', 'content' => Policies.grounding_prompt(text, passages) },
                    ])
  parsed = Parsers.policy(answer.text)
  unless parsed[:decided]
    return unchecked("unparsed judge response: #{parsed[:reason]}",
                     model: model, latency_ms: answer.latency_ms, raw: answer.raw)
  end

  return pass(model: model, latency_ms: answer.latency_ms, raw: answer.raw) unless parsed[:violated]

  block(reason: parsed[:reason], categories: parsed[:categories], model: model,
        latency_ms: answer.latency_ms, raw: answer.raw)
end

#offline? ⇒ Boolean

Returns:

  • (Boolean) 


43
44
45
 
# File 'lib/vangrail/rails/grounding.rb', line 43

def offline?
  false
end