Class: Vangrail::Rails::Budget

Inherits:
Vangrail::Rail show all
Defined in:
lib/vangrail/rails/budget.rb

Overview

Refuses text too large to be a question.

The cost of answering is paid by whoever runs the endpoint, and a public desk gives anybody a way to spend it. A megabyte pasted into the box is not a question: it is a bill, and on a metered endpoint it is somebody else's bill. The same size also dilutes whatever the instructions said, which is the mechanism the many-shot work measured, so the limit is worth having twice over.

Characters rather than tokens, because counting tokens means shipping a tokeniser, and a tokeniser is a dependency and a model-specific one. The ratio is close enough for a limit: roughly four characters a token for English prose, less for code.

Rails::Budget.new(max_characters: 8_000)

A separate, larger limit applies to retrieved documents, which are legitimately longer than anything a reader types and are usually clipped by the retrieval step already. Setting it to nil turns that side off.

Blocks rather than truncating. A truncated question is a different question, and answering a different question well is worse than saying the box has a limit.

Constant Summary collapse

DEFAULT_INPUT = 
8_000
DEFAULT_CONTEXT = 
60_000

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(max_characters: DEFAULT_INPUT, max_context: DEFAULT_CONTEXT, name: 'budget', sides: %i[input context])) ⇒ Budget

Returns a new instance of Budget.



36
37
38
39
40
41
 
# File 'lib/vangrail/rails/budget.rb', line 36

def initialize(max_characters: DEFAULT_INPUT, max_context: DEFAULT_CONTEXT,
               name: 'budget', sides: %i[input context])
  super(name: name, sides: sides)
  @max_characters = max_characters
  @max_context = max_context
end

Instance Attribute Details

#max_characters ⇒ Object (readonly)

Returns the value of attribute max_characters.



34
35
36
 
# File 'lib/vangrail/rails/budget.rb', line 34

def max_characters
  @max_characters
end

#max_context ⇒ Object (readonly)

Returns the value of attribute max_context.



34
35
36
 
# File 'lib/vangrail/rails/budget.rb', line 34

def max_context
  @max_context
end

Instance Method Details

#cache_key(text, context) ⇒ Object 



51
52
53
 
# File 'lib/vangrail/rails/budget.rb', line 51

def cache_key(text, context)
  "#{context[:side]}:#{text.to_s.length}"
end

#call(text, context) ⇒ Object 



55
56
57
58
59
60
61
62
63
64
 
# File 'lib/vangrail/rails/budget.rb', line 55

def call(text, context)
  limit = context[:side] == :context ? max_context : max_characters
  return pass if limit.nil?

  size = text.to_s.length
  return pass if size <= limit

  block(categories: ['over_budget'],
        reason: "#{size} characters, over the #{limit} allowed for #{context[:side]}")
end

#language_agnostic? ⇒ Boolean

Returns:

  • (Boolean) 


47
48
49
 
# File 'lib/vangrail/rails/budget.rb', line 47

def language_agnostic?
  true
end

#offline? ⇒ Boolean

Returns:

  • (Boolean) 


43
44
45
 
# File 'lib/vangrail/rails/budget.rb', line 43

def offline?
  true
end