Cache me if you Can: an Online Cost-aware Teacher-Student Framework to Reduce the Calls to Large Language Models

Ilias Stogiannidis^1,2 Stavros Vassos² Prodromos Malakasiotis^1,4 Ion Androutsopoulos^1,3

Abstract

Prompting Large Language Models (LLMs) performs impressively in zero- and few-shot settings. Hence, small and medium-sized enterprises (SMEs) that cannot afford the cost of creating large task-specific training datasets, but also the cost of pretraining their own LLMs, are increasingly turning to third-party services that allow them to prompt LLMs. However, such services currently require a payment per call, which becomes a significant operating expense (OpEx). Furthermore, customer inputs are often very similar over time, hence SMEs end-up prompting LLMs with very similar instances. We propose a framework that allows reducing the calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. The framework includes criteria for deciding when to trust the local model or call the LLM, and a methodology to tune the criteria and measure the tradeoff between performance and cost. For experimental purposes, we instantiate our framework with two LLMs, GPT-3.5 or GPT-4, and two inexpensive students, a k-NN classifier or a Multi-Layer Perceptron, using two common business tasks, intent recognition and sentiment analysis. Experimental results indicate that significant OpEx savings can be obtained with only slightly lower performance.

Architecture

Framework

We introduce OCaTS: an Online Cost-aware Teacher Student Framework. We draw inspiration from the general idea of a teacher-student schema to train a local inexpensive model (student) using the previous responses of the more expensive model (teacher). Unlike other teacher-student architectures, where the student is trained in all available data, OCaTS takes into account the cost of acquiring this data from the teacher making it an optimal solution for small and medium enterprises that want to use powerful and broadly available, via API, Large Language Models while cutting down their total operational expenses.

Cost-awareness

To achieve the consideration of the cost, we used a simple yet effective solution. We penalized the total performance of the framework by a fraction of the rate of the calls to the teacher model. By setting this objective as the goal of the framework, it learns to maximize the performance by letting the student respond when they are confident enough or prompt the teacher to give their response.

Performance

We evaluate the framework in an intent recognition task. We use the Banking77 dataset, simple k-NN student, and GPT-4 as the teacher. We simulate the incoming data in a streaming manner, as they would normally come to an Enterprise. The use of OCaTS resulted in having the almost the same performance as using the GPT-4 alone to detect the customer intents, while cutting down the operational cost by a remarkable 66%.

Bibtex

@article{stogiannidis_et_al2023,
  title={Cache me if you can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models},
  author={Ilias Stogiannidis and Stavros Vassos and Prodromos Malakasiotis and Ion Androutsopoulos},
  journal={Findings of 2023 Conference on Empirical Methods in Natural Language Processing},
  location={Singapore}
  year={2023},
  date={December, 6}
  publisher={Association for Computational Linguistics}
  url={https://arxiv.org/abs/2310.13395}
}

Acknoweledgements

This work was supported by Google’s TPU Research Cloud (TRC)