Class: Vangrail::Evidence

Inherits:
Struct
  • Object
show all
Defined in:
lib/vangrail/evidence.rb

Overview

One rail's measured operating point, read as evidence rather than as a verdict.

Every rail in this gem answers a yes-or-no question, and the engine combines those answers by taking the first "yes". That is what the published defences do too, and it throws away almost everything the rails know. It cannot say how much a hit is worth, cannot add up three near misses, and cannot tell an operator what a block actually means about the text.

What a hit is worth is a ratio, and it is measurable: how much likelier this rail is to fire on an attack than on ordinary documentation. That number, the likelihood ratio, is all a rail needs to contribute to a shared judgement, and it is exactly what the corpora in this repository already measure. Every entry here comes from running a rail over the same attack and benign sets as every other rail, which is what makes the numbers comparable in the first place: a detection rate measured on one paper's corpus and a false-positive rate measured on another's cannot be combined at all.

Rates are smoothed with the Jeffreys prior, (hits + 1/2) / (n + 1), for a reason that is not decoration. A rail that caught 60 of 60 has an unsmoothed detection rate of exactly 1, an unsmoothed likelihood ratio of infinity, and would single-handedly decide every judgement it appears in on the strength of a sixty-item corpus. Smoothing keeps the evidence finite and proportional to how much was actually measured.

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#attacks ⇒ Object

Returns the value of attribute attacks

Returns:

  • (Object) —

    the current value of attacks



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def attacks
  @attacks
end

#attacks_caught ⇒ Object

Returns the value of attribute attacks_caught

Returns:

  • (Object) —

    the current value of attacks_caught



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def attacks_caught
  @attacks_caught
end

#benign ⇒ Object

Returns the value of attribute benign

Returns:

  • (Object) —

    the current value of benign



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def benign
  @benign
end

#benign_flagged ⇒ Object

Returns the value of attribute benign_flagged

Returns:

  • (Object) —

    the current value of benign_flagged



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def benign_flagged
  @benign_flagged
end

#group ⇒ Object

Returns the value of attribute group

Returns:

  • (Object) —

    the current value of group



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def group
  @group
end

#rail ⇒ Object

Returns the value of attribute rail

Returns:

  • (Object) —

    the current value of rail



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def rail
  @rail
end

Instance Method Details

#bits(fired, confidence: nil) ⇒ Object

Evidence in bits, positive towards attack. Bits rather than nats because an operator has to read these: one bit is a doubling of the odds, and "this rail is worth four bits" is a sentence somebody can act on.

With a confidence, the bits are what the corpus can defend at that level rather than what it measured. A table built from a few hundred texts should be read this way; the point estimate is what it would say if the corpus were the world.



87
88
89
90
91
92
93
94
95
 
# File 'lib/vangrail/evidence.rb', line 87

def bits(fired, confidence: nil)
  return Math.log2(fired ? ratio_fired : ratio_silent) if confidence.nil?

  low, high = bits_interval(fired, confidence: confidence)
  return low if low.positive?
  return high if high.negative?

  0.0
end

#bits_interval(fired, confidence:) ⇒ Object

A joint credible interval for the observation's likelihood ratio, in bits. Splitting the excluded mass across both tails of both independent beta posteriors gives the rate rectangle at least confidence posterior mass by the union bound.



101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/vangrail/evidence.rb', line 101

def bits_interval(fired, confidence:)
  unless confidence.is_a?(Numeric) && confidence.finite? && confidence.positive? && confidence < 1
    raise ArgumentError, 'confidence must be finite and strictly between 0 and 1'
  end

  tail = (1 - confidence) / 4.0
  detection_low, detection_high = rate_interval(:detection, attacks_caught, attacks, tail)
  false_alarm_low, false_alarm_high = rate_interval(:false_alarm, benign_flagged, benign, tail)

  ratios = if fired
             [detection_low / false_alarm_high, detection_high / false_alarm_low]
           else
             [(1 - detection_high) / (1 - false_alarm_low),
              (1 - detection_low) / (1 - false_alarm_high)]
           end
  ratios.map { |ratio| Math.log2(ratio) }
end

#bounds ⇒ Object 



75
76
77
 
# File 'lib/vangrail/evidence.rb', line 75

def bounds
  @bounds ||= {}
end

#capability(prior:) ⇒ Object

How much of the question this rail actually answers, at a given base rate.

Detection and false-alarm rates describe a rail; they do not describe what it is worth in a deployment, because they say nothing about how often the thing being detected happens. The intrusion-detection literature settled this with an information-theoretic measure: the fraction of the uncertainty about "is this an attack" that the rail's verdict removes.

Measured on the shipped table, the base rate costs every rail roughly two fifths of its capability between a balanced corpus and one attack in ten thousand: paraphrase falls from 0.52 to 0.28, and every other rail sits under 0.1 at both. many_shot manages 0.001, which is the honest reading of a rail that caught six of 270 because the corpus is mostly not its attack.

Ranking rails by this rather than by detection rate is the point. It is the only number here that changes when the deployment does.



135
136
137
138
139
 
# File 'lib/vangrail/evidence.rb', line 135

def capability(prior:)
  return 0.0 unless measured?

  mutual_information(prior) / entropy(prior)
end

#detection ⇒ Object

Probability the rail fires given the text is an attack.



33
34
35
 
# File 'lib/vangrail/evidence.rb', line 33

def detection
  (attacks_caught + 0.5) / (attacks + 1.0)
end

#detection_bound(confidence) ⇒ Object

The rates a corpus this size can actually defend, rather than the ones it happens to have produced.

A rail that fired on none of 48 benign texts has a point estimate of one in a hundred and a 95% upper bound of one in twenty-six. The difference is two bits of evidence that nobody measured, and reporting the point estimate spends them.

Marginal one-sided rate bounds are useful summaries. A likelihood ratio needs simultaneous bounds on both rates; bits_interval supplies that joint statement.



65
66
67
68
 
# File 'lib/vangrail/evidence.rb', line 65

def detection_bound(confidence)
  bounds[[:detection, confidence]] ||=
    Beta.quantile(1 - confidence, attacks_caught + 0.5, attacks - attacks_caught + 0.5)
end

#false_alarm ⇒ Object

Probability it fires given the text is ordinary.



38
39
40
 
# File 'lib/vangrail/evidence.rb', line 38

def false_alarm
  (benign_flagged + 0.5) / (benign + 1.0)
end

#false_alarm_bound(confidence) ⇒ Object 



70
71
72
73
 
# File 'lib/vangrail/evidence.rb', line 70

def false_alarm_bound(confidence)
  bounds[[:false_alarm, confidence]] ||=
    Beta.quantile(confidence, benign_flagged + 0.5, benign - benign_flagged + 0.5)
end

#measured? ⇒ Boolean

A rail that never fired on either corpus has measured nothing, whatever its detection rate looks like after smoothing.

Returns:

  • (Boolean) 


152
153
154
 
# File 'lib/vangrail/evidence.rb', line 152

def measured?
  attacks.positive? && benign.positive?
end

#ratio_fired ⇒ Object

How much likelier a hit is on an attack than on ordinary text.



43
44
45
 
# File 'lib/vangrail/evidence.rb', line 43

def ratio_fired
  detection / false_alarm
end

#ratio_silent ⇒ Object

And how much likelier silence is on ordinary text than on an attack. This is the half that OR-combination cannot express at all: a sensitive rail staying quiet is evidence too, and it points the other way.



50
51
52
 
# File 'lib/vangrail/evidence.rb', line 50

def ratio_silent
  (1 - detection) / (1 - false_alarm)
end

#to_bits_h(prior:, confidence: nil) ⇒ Object 



141
142
143
144
145
146
147
148
 
# File 'lib/vangrail/evidence.rb', line 141

def to_bits_h(prior:, confidence: nil)
  {
    'rail' => rail,
    'bits_if_fired' => bits(true, confidence: confidence).round(2),
    'bits_if_silent' => bits(false, confidence: confidence).round(2),
    'capability' => capability(prior: prior).round(4),
  }
end

#to_h ⇒ Object 



156
157
158
159
160
161
162
163
164
 
# File 'lib/vangrail/evidence.rb', line 156

def to_h
  {
    'rail' => rail, 'group' => group,
    'attacks_caught' => attacks_caught, 'attacks' => attacks,
    'benign_flagged' => benign_flagged, 'benign' => benign,
    'detection' => detection.round(4), 'false_alarm' => false_alarm.round(4),
    'bits_if_fired' => bits(true).round(2), 'bits_if_silent' => bits(false).round(2)
  }
end