Module: Vangrail::Parsers

Defined in:
lib/vangrail/parsers.rb

Overview

Readers for what guard and judge models actually answer.

Each returns a hash: violated:, categories:, reason:. decided false means the text was not in a form this code understands, which a rail turns into an uncertain result rather than a pass. Guessing at an unreadable answer is how a guardrail comes to report checks it never made.

Constant Summary collapse

LLAMA_GUARD_CATEGORIES =

Llama Guard 3 hazard codes, the MLCommons taxonomy the model card lists.

{
  'S1' => 'Violent Crimes',
  'S2' => 'Non-Violent Crimes',
  'S3' => 'Sex-Related Crimes',
  'S4' => 'Child Sexual Exploitation',
  'S5' => 'Defamation',
  'S6' => 'Specialized Advice',
  'S7' => 'Privacy',
  'S8' => 'Intellectual Property',
  'S9' => 'Indiscriminate Weapons',
  'S10' => 'Hate',
  'S11' => 'Suicide & Self-Harm',
  'S12' => 'Sexual Content',
  'S13' => 'Elections',
  'S14' => 'Code Interpreter Abuse',
}.freeze

Class Method Summary collapse

Class Method Details

.apriel_guard(text) ⇒ Object

"safe\nnon_adversarial" or "unsafe-O14,O12\nadversarial". Either line can condemn the turn: a jailbreak attempt with no hazard category is still one. With reasoning on the same two verdicts arrive as labelled fields after their assessments, so that form is tried first.



91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/vangrail/parsers.rb', line 91

def apriel_guard(text)
  reasoned = apriel_guard_reasoned(text)
  return reasoned if reasoned

  lines = clean_lines(text)
  safety = lines.detect { |l| l.match?(/\A(safe|unsafe)/i) }
  adversarial = lines.detect { |l| l.match?(/\A(non_adversarial|adversarial)/i) }
  return undecided(text) if safety.nil? && adversarial.nil?

  unsafe = safety.to_s.match?(/\Aunsafe/i)
  attack = adversarial.to_s.match?(/\Aadversarial/i)
  return clean unless unsafe || attack

  codes = codes_in(safety, /O\d{1,2}/i)
  codes += ['adversarial'] if attack
  reason = unsafe ? describe(codes - ['adversarial'], {}, 'unsafe') : 'adversarial input'
  { decided: true, violated: true, categories: codes, reason: reason }
end

.apriel_guard_reasoned(text) ⇒ Object

Reasoning mode:

safety_risks_assessment_reasoning: ## Step 1 ...
safety_risks_class: unsafe,
safety_risks_categories: ['O15'],
adversarial_attacks_assessment_reasoning: ## Step 1 ...
adversarial_attacks_class: adversarial

nil when the text is not in this form, so the caller can try the short one.



119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/vangrail/parsers.rb', line 119

def apriel_guard_reasoned(text)
  body = text.to_s
  return nil unless body.include?('safety_risks_class')

  fields = {}
  body.each_line do |line|
    m = line.chomp.match(
      /\A(safety_risks_class|safety_risks_categories|adversarial_attacks_class)\s*:\s*(.*)\z/,
    )
    fields[m[1]] = m[2].strip.delete_suffix(',') if m
  end
  unsafe = fields['safety_risks_class'].to_s.match?(/unsafe/i)
  attack = fields['adversarial_attacks_class'].to_s.match?(/\Aadversarial/i)
  return clean unless unsafe || attack

  codes = codes_in(fields['safety_risks_categories'], /O\d{1,2}/i)
  codes += ['adversarial'] if attack
  block = unsafe ? 'safety_risks' : 'adversarial_attacks'
  { decided: true, violated: true, categories: codes, reason: rationale(body, block) }
end

.clean ⇒ Object 



197
198
199
 
# File 'lib/vangrail/parsers.rb', line 197

def clean
  { decided: true, violated: false, categories: [], reason: nil }
end

.clean_lines(text) ⇒ Object 



209
210
211
 
# File 'lib/vangrail/parsers.rb', line 209

def clean_lines(text)
  text.to_s.strip.lines.map(&:strip).reject(&:empty?)
end

.codes_in(text, pattern) ⇒ Object 



213
214
215
 
# File 'lib/vangrail/parsers.rb', line 213

def codes_in(text, pattern)
  text.to_s.scan(pattern).map(&:upcase).uniq
end

.describe(codes, names, fallback) ⇒ Object 



217
218
219
220
221
 
# File 'lib/vangrail/parsers.rb', line 217

def describe(codes, names, fallback)
  return fallback if codes.empty?

  codes.map { |c| names[c] ? "#{c} #{names[c]}" : c }.join(', ')
end

.first_json_object(text) ⇒ Object

First balanced ..., so a fenced or prefaced verdict still reads.



170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'lib/vangrail/parsers.rb', line 170

def first_json_object(text)
  start = text.index('{')
  return nil unless start

  depth = 0
  text[start..].each_char.with_index do |ch, i|
    depth += 1 if ch == '{'
    next unless ch == '}'

    depth -= 1
    return JSON.parse(text[start, i + 1]) if depth.zero?
  end
  nil
rescue JSON::ParserError
  nil
end

.harmful_yes_no(text) ⇒ Object

"Harmful request: yes" / "Harmful request: no", the WildGuard shape, and the several models that answer a bare "Yes" or "No" to a policy question the way ShieldGemma does.

A bare yes needs the caller to know the question was "is this harmful", which is why this is a preset rather than a guess: the same token means the opposite from a model asked "is this safe". Nothing here infers the polarity from the text.



53
54
55
56
57
58
59
60
61
62
 
# File 'lib/vangrail/parsers.rb', line 53

def harmful_yes_no(text)
  lines = clean_lines(text)
  line = lines.detect { |l| l.match?(/\A(harmful|harmful request|answer)?\s*:?\s*(yes|no)\b/i) } || lines.first
  answer = line.to_s[/\b(yes|no)\b/i]
  return undecided(text) unless answer

  return clean if answer.casecmp('no').zero?

  { decided: true, violated: true, categories: [], reason: 'harmful' }
end

.llama_guard(text) ⇒ Object

"safe" or "unsafe\nS1,S10".



34
35
36
37
38
39
40
41
42
43
 
# File 'lib/vangrail/parsers.rb', line 34

def llama_guard(text)
  lines = clean_lines(text)
  head = lines.first.to_s.downcase
  return undecided(text) unless head.start_with?('safe', 'unsafe')
  return clean if head.start_with?('safe')

  codes = codes_in(lines[1..]&.join(','), /S\d{1,2}/i)
  { decided: true, violated: true, categories: codes,
    reason: describe(codes, LLAMA_GUARD_CATEGORIES, 'unsafe') }
end

.policy(text) ⇒ Object

A JSON verdict first, then a bare 0/1, then Yes/No. All three appear depending on which answer contract a policy prompt asked for.



142
143
144
145
146
147
148
149
150
151
152
153
154
155
 
# File 'lib/vangrail/parsers.rb', line 142

def policy(text)
  body = text.to_s
  from_json = policy_json(body)
  return from_json if from_json

  stripped = body.strip
  return (stripped.start_with?('0') ? clean : violation) if stripped.match?(/\A[01]\b/)

  case stripped
  when /\Ayes\b/i then violation(reason: 'policy judge said yes')
  when /\Ano\b/i then clean
  else undecided(body)
  end
end

.policy_json(body) ⇒ Object 



157
158
159
160
161
162
163
164
165
166
167
 
# File 'lib/vangrail/parsers.rb', line 157

def policy_json(body)
  obj = first_json_object(body)
  return nil unless obj

  value = obj['violation']
  return nil unless [0, 1, true, false, '0', '1'].include?(value)
  return clean unless [1, true, '1'].include?(value)

  cats = [obj['policy_category'], *Array(obj['rule_ids'])].compact.map(&:to_s).reject(&:empty?)
  violation(categories: cats, reason: (obj['rationale'] || cats.join(',')).to_s[0, 240])
end

.rationale(body, block) ⇒ Object

Last step of an assessment block, where the model states its conclusion rather than restating the input.



189
190
191
192
193
194
195
 
# File 'lib/vangrail/parsers.rb', line 189

def rationale(body, block)
  section = body[/#{block}_assessment_reasoning:(.*?)(?=^[a-z_]+:)/m, 1]
  return nil unless section

  steps = section.split(/^##\s*Step\s*\d+\s*$/m).map(&:strip).reject(&:empty?)
  (steps.last || section).gsub(/\s+/, ' ').strip[0, 240]
end

.three_label_guard(text, middle: :unsafe) ⇒ Object

"safe" / "unsafe" / "controversial", the Qwen Guard shape, and any other three-label guard that keeps a middle verdict.

The middle label is a policy decision and it is a large one. Benchmarking fourteen open guard models, Sadeghi et al. (arXiv:2605.28830) moved one model's recall from 46.75% to 83.97% -- 37.2 points -- by counting "controversial" as unsafe rather than dropping it. So it is counted, and middle: says so at the call site: :unsafe blocks, :undecided reports a check that did not conclude, and nothing silently treats it as clean.



73
74
75
76
77
78
79
80
81
82
83
84
85
 
# File 'lib/vangrail/parsers.rb', line 73

def three_label_guard(text, middle: :unsafe)
  lines = clean_lines(text)
  head = lines.first.to_s.downcase
  return clean if head.start_with?('safe')
  return { decided: true, violated: true, categories: [], reason: 'unsafe' } if head.start_with?('unsafe')
  return undecided(text) unless head.start_with?('controversial')

  case middle
  when :unsafe then { decided: true, violated: true, categories: ['controversial'], reason: 'controversial' }
  when :safe then clean
  else undecided(text)
  end
end

.undecided(text) ⇒ Object 



205
206
207
 
# File 'lib/vangrail/parsers.rb', line 205

def undecided(text)
  { decided: false, violated: false, categories: [], reason: text.to_s.strip[0, 120] }
end

.violation(categories: [], reason: nil) ⇒ Object 



201
202
203
 
# File 'lib/vangrail/parsers.rb', line 201

def violation(categories: [], reason: nil)
  { decided: true, violated: true, categories: categories, reason: reason }
end