Code Documentation¶

Main Functions¶

start_testing(attack_model: ClientBase, tested_model: ClientBase, config: dict, judge_model: ClientBase | None = None, num_threads: int | None = 1, basic_tests: List[Tuple[str, Dict]] | None = None, custom_tests: List[Tuple[Type[TestBase], Dict]] | None = None)[source]

The main entry point for launching tests.

Parameters:

attack_model (ClientBase) – The model that generates attacks.
tested_model (ClientBase) – The model to be tested for vulnerabilities.
config (dict) –
Configuration dictionary with keys:
- ’enable_logging’bool
  Whether to enable logging.
- ’enable_reports’bool
  Whether to generate Excel/Word reports.
- ’artifacts_path’Optional[str]
  Path to the folder for saving artifacts (logs, CSV, etc.).
- ’debug_level’int
  Logging level (0=WARNING, 1=INFO, 2=DEBUG).
- ’report_language’str
  Language code for the final report (‘en’ or ‘ru’).
judge_model (Optional[ClientBase]) – The model that judges the responses (optional).
num_threads (Optional[int]) – Number of threads to use for parallel testing.
basic_tests (Optional[List[Tuple[str, Dict]]]) – A list of test code names and a dict of parameters for each test.
custom_tests (Optional[List[Tuple[Type[TestBase], Dict]]]) – A list of custom test classes and parameter dictionaries.

Return type:

None

Note

This function starts the testing process with different configurations.

Abstract Classes¶

class ClientBase[source]¶

Base class for interacting with chat models. The history and new messages are passed as a list of dictionaries.

system_prompts¶

Optional system prompts to guide the conversation.

Type:: Optional[List[str]]

model_description¶

Optional model description to guide the conversation.

Type:: Optional[str]

interact(history: List[Dict[str, str]], messages: List[Dict[str, str]]) → Dict[str, str][source]¶: Takes the conversation history and new messages, sends them to the LLM, and returns a new response.

Note

ClientBase is an abstract base class for client implementations.

class TestBase(client_config: ClientConfig, attack_config: AttackConfig, judge_config: JudgeConfig | None = None, artifacts_path: str | None = None, num_attempts: int = 0, **kwargs)[source]¶

A base class for test classes. Each test represents a different kind of attack against the target LLM model. The test sends a sequence of prompts and evaluates the responses while updating the status.

Parameters:

client_config (ClientConfig)
attack_config (AttackConfig)
judge_config (JudgeConfig | None)
artifacts_path (str | None)
num_attempts (int)

Note

TestBase is an abstract base class designed for attack handling in the testing framework.

Available Clients¶

class ClientLangChain(backend: str, system_prompts: List[str] | None = None, model_description: str | None = None, **kwargs)[source]

Bases: ClientBase

Wrapper for interacting with models through LangChain.

Parameters:

backend (str) – The backend name to use for model initialization.
system_prompts (Optional[List[str]]) – List of system prompts for initializing the conversation context (optional).
**kwargs – Additional arguments passed to the model’s constructor.
model_description (str | None)

_convert_to_base_format(message: BaseMessage) → Dict[str, str][source]: Converts a LangChain message (HumanMessage, AIMessage) to the base format (Dict with “role” and “content”).

_convert_to_langchain_format(message: Dict[str, str]) → BaseMessage[source]: Converts a message from the base format (Dict) to LangChain’s format (HumanMessage, AIMessage).

interact(history: List[Dict[str, str]], messages: List[Dict[str, str]]) → Dict[str, str][source]: Takes conversation history and new messages, sends a request to the model, and returns the response as a dictionary.

Note

ClientLangChain is a client implementation for LangChain-based services.

class ClientOpenAI(api_key: str, base_url: str, model: str, temperature: float = 0.1, system_prompts: List[str] | None = None, model_description: str | None = None)[source]

Bases: ClientBase

Wrapper for interacting with OpenAI-compatible API. This client can be used to interact with any language model that supports the OpenAI API, including but not limited to OpenAI models.

Parameters:

api_key (str) – The API key for authentication.
base_url (str) – The base URL of the OpenAI-compatible API.
model (str) – The model identifier to use for generating responses.
temperature (float) – The temperature setting for controlling randomness in the model’s responses.
system_prompts (Optional[List[str]]) – List of system prompts for initializing the conversation context (optional).
model_description (str) – Description of the model, including domain and other features (optional).

_convert_to_base_format(message: Dict[str, str]) → Dict[str, str][source]: Converts a message from OpenAI format (Dict) to the base format (Dict with “role” and “content”).

_convert_to_openai_format(message: Dict[str, str]) → Dict[str, str][source]: Converts a message from the base format (Dict with “role” and “content”) to OpenAI’s format (Dict).

interact(history: List[Dict[str, str]], messages: List[Dict[str, str]]) → Dict[str, str][source]: Takes conversation history and new messages, sends a request to the OpenAI-compatible API, and returns the response.

Note

ClientOpenAI is a client implementation for OpenAI-based services.

Additional Utility Functions¶

get_preset_tests_params_example(preset_name: Literal['all', 'standard'] = 'all') → str[source]

Generate example code for configuring basic_tests_params based on a preset configuration. If preset_name is “all”, returns configuration for all tests (as in get_basic_tests_params_example).

Parameters:: preset_name (Literal["all", "standard"]) – The name of the preset configuration to use.
Returns:: A code snippet showing the configuration for the given preset.
Return type:: str

Note

This function generates an example code snippet for configuring basic_tests_params based on a preset configuration. It returns a code snippet as a string.

print_preset_tests_params_example(preset_name: Literal['all', 'standard']) → None[source]

Print an example configuration for basic_tests_params based on a preset to the console. If preset_name is “all”, prints configuration for all tests.

Parameters:: preset_name (Literal["all", "standard"]) – The name of the preset configuration to print.
Return type:: None

Note

This function prints an example configuration for basic_tests_params based on a preset to the console.

print_chat_models_info(detailed: bool = False) → None[source]

Print information about LangChain chat models in a well-formatted way.

Parameters:: detailed (bool) – Whether to print detailed information including parameter descriptions
Return type:: None

Note

This function prints information about LangChain chat models in a well-formatted manner. It displays details such as the model name, a short description, and its supported parameters.