Hey Folks! I'm excited to share a new open-source library that can help optimize your LLM deployment costs. The adaptive-classifier library learns to route queries between your models based on complexity, continuously improving through real-world usage.

We tested it on the arena-hard-auto dataset, routing between a high-cost and low-cost model (2x cost difference). The results were impressive:

  • 32.4% cost savings with adaptation enabled
  • Same overall success rate (22%) as baseline
  • System automatically learned from 110 new examples during evaluation
  • Successfully routed 80.4% of queries to the cheaper model

Perfect for setups where you're running multiple LLama models (like Llama-3.1-70B alongside Llama-3.1-8B) and want to optimize costs without sacrificing capability. The library integrates easily with any transformer-based models and includes built-in state persistence.

Check out the repo for implementation details and benchmarks. Would love to hear your experiences if you try it out!

Repo - https://github.com/codelion/adaptive-classifier

Source: View source