ai_utils

accuracy

openai

toolium.utils.ai_utils.openai.openai_request(system_message, user_message, model_name=None, azure=False, **kwargs)[source]

Make a request to OpenAI API (Azure or standard)

Parameters:

  • system_message – system message to set the behavior of the assistant

  • user_message – user message with the request

  • model_name – name of the model to use

  • azure – whether to use Azure OpenAI or standard OpenAI

  • kwargs – additional parameters to be passed to the OpenAI client (azure_endpoint, timeout, etc.)

Returns:

tuple with response from OpenAI and token usage dict

prompt_injection

text_similarity

toolium.utils.ai_utils.text_similarity.assert_text_similarity(text, expected_texts, threshold, similarity_method=None, model_name=None, **kwargs)[source]

Get similarity between one text and a list of expected texts and assert if any of the expected texts is similar.

Parameters:

  • text – string to compare

  • expected_texts – string or list of strings with the expected texts

  • threshold – minimum similarity score to consider texts similar

  • similarity_method – method to use for text comparison (‘spacy’, ‘sentence_transformers’, ‘openai’ or ‘azure_openai’)

  • model_name – model name to use for the similarity method

  • kwargs – additional parameters to be used by comparison methods

toolium.utils.ai_utils.text_similarity.get_text_similarity_with_azure_openai(text, expected_text, model_name=None, **kwargs)[source]

Return semantic similarity between two texts using Azure OpenAI LLM

Parameters:

  • text – string to compare

  • expected_text – string with the expected text

  • model_name – name of the Azure OpenAI model to use

  • kwargs – additional parameters to be used by Azure OpenAI client

Returns:

tuple with similarity score between the two texts and explanation

toolium.utils.ai_utils.text_similarity.get_text_similarity_with_openai(text, expected_text, model_name=None, azure=False, **kwargs)[source]

Return semantic similarity between two texts using OpenAI LLM

Parameters:

  • text – string to compare

  • expected_text – string with the expected text

  • model_name – name of the OpenAI model to use

  • azure – whether to use Azure OpenAI or standard OpenAI

  • kwargs – additional parameters to be used by OpenAI client

Returns:

tuple with similarity score between the two texts and explanation

toolium.utils.ai_utils.text_similarity.get_text_similarity_with_sentence_transformers(text, expected_text, model_name=None, **kwargs)[source]

Return similarity between two texts using Sentence Transformers

Parameters:

  • text – string to compare

  • expected_text – string with the expected text

  • model_name – name of the Sentence Transformers model to use

  • kwargs – additional parameters to be used by SentenceTransformer (modules, device, prompts, etc.)

Returns:

similarity score between the two texts

toolium.utils.ai_utils.text_similarity.get_text_similarity_with_spacy(text, expected_text, model_name=None, **kwargs)[source]

Return similarity between two texts using spaCy. This method normalize both texts before comparing them.

Parameters:

  • text – string to compare

  • expected_text – string with the expected text

  • model_name – name of the spaCy model to use

  • kwargs – additional parameters to be used by spaCy (disable, exclude, etc.)

Returns:

similarity score between the two texts

text_readability

toolium.utils.ai_utils.text_readability.assert_text_readability(text, threshold, technical_chars=None, readability_method=None, model_name=None, **kwargs)[source]

Get the readability of a text and assert if it is above a given threshold.

Parameters:

  • text – string to compare

  • threshold – minimum readability score to consider the text readable

  • technical_chars – list of technical characters to consider as non-linguistic content

  • readability_method – method to use for text readability (‘spacy’)

  • model_name – model name to use for the readability method

  • kwargs – additional parameters to be used by comparison methods

toolium.utils.ai_utils.text_readability.get_text_readability_with_spacy(text, technical_chars=None, model_name=None, **kwargs)[source]

Return the readability score of a text using spaCy.

Parameters:

  • text – string to assess readability

  • model_name – name of the spaCy model to use

  • kwargs – additional parameters to be used by spaCy (disable, exclude, etc.)

Returns:

readability score between 0 and 1