Module: Vangrail::Posterior

Defined in:
lib/vangrail/evidence.rb

Overview

Combines rail evidence into a posterior probability that the text is an attack.

The arithmetic is one line: odds after = odds before times every likelihood ratio. In bits it is addition, which is why the contributions of individual rails can be printed and read.

Three things make this more than a formality, and all three are things the published defences leave on the floor.

The prior is the deployment's, and it dominates. Detector papers evaluate on balanced corpora, where half the traffic is an attack; a documentation desk sees maybe one poisoned page in ten thousand. At that base rate a rail with a one percent false-alarm rate is wrong far more often than it is right when it fires, and no amount of detection rate fixes it. That is not a criticism of the rails: it is the arithmetic every operator inherits and almost none is shown.

Abstention is evidence of nothing, which is different from evidence against. A rail that was off, unreachable, or undecided contributes no term at all, and this gem is unusual in knowing which rails those were: certain? is exactly that fact, and here it finally has arithmetic to feed.

Correlated rails do not each get a vote. Three rails that fire on the same sentence for the same reason are one observation reported three times, and summing them is how naive Bayes talks itself into certainty. Rails measured to agree are grouped, and a group contributes once.

Constant Summary collapse

DEFAULT_CONFIDENCE =

Combines and returns [posterior, contributions].

observations maps a rail name to true (fired), false (ran and did not fire), or nil (did not run). The nils are the point. direct carries rails that computed their own log-likelihood ratio rather than answering yes or no. A rail that can say how sure it is should not be flattened to one bit on the way in, and nothing about the arithmetic changes: bits are bits, whoever produced them. Defensible by default. The point estimate is what a corpus happened to produce and it is unreadable at the edges: a rail that caught none of the published attacks and fired on none of eighteen thousand documents scores +7 bits on the point estimate, from two smoothing constants dividing each other, and -2.7 on the bound. The bound is the number that survives being measured against somebody else's corpus, so it is the one that runs.

0.95

Class Method Summary collapse

Class Method Details

.combine(prior:, observations:, evidence: EvidenceData::TABLE, confidence: DEFAULT_CONFIDENCE, direct: {}) ⇒ Object

Raises:

  • (ArgumentError) 


243
244
245
246
247
248
249
250
 
# File 'lib/vangrail/evidence.rb', line 243

def combine(prior:, observations:, evidence: EvidenceData::TABLE, confidence: DEFAULT_CONFIDENCE,
            direct: {})
  raise ArgumentError, 'prior must be strictly between 0 and 1' unless prior.positive? && prior < 1

  contributions = weigh(observations, evidence, confidence) + quantified(direct)
  total = contributions.sum { |c| c[:bits] }
  [from_odds(to_odds(prior) * (2**total)), contributions]
end

.false_alarm_needed(prior:, detection: 0.75, target: 0.5) ⇒ Object

The false-alarm rate a single rail would need to carry a block on its own.

Rearranged from the same identity: at base rate prior, one rail with detection detection reaches target only if it almost never fires on ordinary text. The answers come out in the region of one in ten thousand, which is below what any hand-built benign corpus can demonstrate: showing a rate that low needs tens of thousands of clean documents on which the rail stayed silent.

That is the practical case for combining rails rather than trusting one, and it is an argument about evidence rather than about taste.



310
311
312
 
# File 'lib/vangrail/evidence.rb', line 310

def false_alarm_needed(prior:, detection: 0.75, target: 0.5)
  detection / (to_odds(target) / to_odds(prior))
end

.from_odds(odds) ⇒ Object 



325
326
327
328
329
 
# File 'lib/vangrail/evidence.rb', line 325

def from_odds(odds)
  return 1.0 if odds.infinite?

  odds / (1 + odds)
end

.quantified(direct) ⇒ Object 



314
315
316
317
318
319
 
# File 'lib/vangrail/evidence.rb', line 314

def quantified(direct)
  direct.map do |rail, bits|
    { group: rail.to_s, rail: rail.to_s, fired: bits.positive?, bits: bits.to_f,
      spoke_for: [rail.to_s], quantified: true }
  end
end

.required_bits(prior:, target: 0.5) ⇒ Object

How many bits it takes to get from a base rate to a target confidence.

This is the number the whole design turns on, and it is worth being able to compute rather than assert. Reaching an even-money posterior from one attack in ten thousand takes about 13.3 bits, and no rail in this gem is worth half that, which is a statement about what a single detector can honestly justify rather than about these particular rails.



295
296
297
 
# File 'lib/vangrail/evidence.rb', line 295

def required_bits(prior:, target: 0.5)
  Math.log2(to_odds(target) / to_odds(prior))
end

.speak_for(group, members, confidence = nil) ⇒ Object 



271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
 
# File 'lib/vangrail/evidence.rb', line 271

def speak_for(group, members, confidence = nil)
  fired = members.select { |m| m[:fired] }
  chosen = if fired.empty?
             members.max_by { |m| m[:entry].detection }
           else
             fired.max_by { |m| m[:entry].bits(true, confidence: confidence) }
           end

  {
    group: group,
    rail: chosen[:rail],
    fired: chosen[:fired],
    bits: chosen[:entry].bits(chosen[:fired], confidence: confidence),
    spoke_for: members.map { |m| m[:rail] },
  }
end

.to_odds(probability) ⇒ Object 



321
322
323
 
# File 'lib/vangrail/evidence.rb', line 321

def to_odds(probability)
  probability / (1 - probability)
end

.weigh(observations, evidence, confidence = nil) ⇒ Object

One term per group rather than one per rail.

Within a group, the firing rail with the most evidence speaks for the group; if none fired, the most sensitive member's silence speaks for it. Both rules pick the single most informative member, which is the conservative reading of a set of observations that are not independent.



258
259
260
261
262
263
264
265
266
267
268
269
 
# File 'lib/vangrail/evidence.rb', line 258

def weigh(observations, evidence, confidence = nil)
  seen = observations.filter_map do |rail, fired|
    next if fired.nil?

    entry = evidence[rail.to_s]
    next unless entry&.measured?

    { rail: rail.to_s, group: entry.group || rail.to_s, fired: fired, entry: entry }
  end

  seen.group_by { |o| o[:group] }.map { |group, members| speak_for(group, members, confidence) }
end