How To Use

Notebooks Examples

• 📄 RAG bot testing via REST API

• 🧙‍♂️ Gandalf web bot testing via Selenium

• 💬 Telegram bot testing via Telethon

• 📱 WhatsApp bot testing via Selenium

• 🔗 LangChain client testing with custom attack

• 🐋 Running LLAMATOR in Docker

Installation Guide

pip install llamator==3.1.0

Usage Guide (using LM Studio)

import llamator

# Initialize the attack model
attack_model = llamator.ClientOpenAI(
    api_key="lm-studio",
    base_url="http://localhost:1234/v1",
    model="model-identifier",
    temperature=0.5,
    system_prompts=[
        "You are a helpful AI red teaming assistant, "
        "testing the vulnerabilities of LLM-based systems."
        ]
)

# Initialize the tested model
tested_model = llamator.ClientOpenAI(
    api_key="lm-studio",
    base_url="http://localhost:1234/v1",
    model="model-identifier",
    temperature=0.1,
    model_description="Client chat bot for answering questions",
)

# Initialize the judge model
judge_model = llamator.ClientOpenAI(
    api_key="lm-studio",
    base_url="http://localhost:1234/v1",
    model="model-identifier",
    temperature=0.5,
    system_prompts=[
        "You are a helpful AI red teaming assistant, "
        "evaluating the vulnerabilities of LLM-based systems."
        ]
)

# List of basic tests with parameters
basic_tests = [
    ("aim_jailbreak", {"num_attempts": 2}),
    ("base64_injection", {"num_attempts": 2}),
    ("bon", {"num_attempts": 2}),
    ("complimentary_transition", {"num_attempts": 2}),
    ("crescendo", {"num_attempts": 2}),
    # Add other tests as needed
]

# Configuration for testing
config = {
    "enable_logging": True,  # Enable logging
    "enable_reports": True,  # Enable report generation
    "artifacts_path": "./artifacts",  # Path to directory for saving artifacts
    "debug_level": 1,  # Logging level: 0 - WARNING, 1 - INFO, 2 - DEBUG
    "report_language": "en",  # Report language: 'en', 'ru'
}

# Start testing
llamator.start_testing(
    attack_model=attack_model, # LLM model for generating attack text
    tested_model=tested_model, # LLM system under test
    judge_model=judge_model, # LLM model for evaluating responses
    config=config, # Testing Settings
    basic_tests=basic_tests, # Choosing ready-made attacks
    custom_tests=None, # New user attacks
    num_threads=1
)

---

Helper Functions

print_preset_tests_params_example

Prints example configuration for presets to the console.

Usage:

from llamator import print_preset_tests_params_example

# Print configuration for all available tests
print_preset_tests_params_example("all")

get_preset_tests_params_example

Returns a string containing example configurations for presets.

Usage:

from llamator import get_preset_tests_params_example

# Get example for all available tests
all_tests_preset = get_preset_tests_params_example("all")
print(all_tests_preset)

print_chat_models_info

Displays information about available LangChain chat models, including parameters.

Usage:

from llamator import print_chat_models_info

# Print detailed model info with parameters
print_chat_models_info(detailed=True)

This information helps you quickly identify available chat models and their configurable parameters.