How to Cut the Review of Hundreds of Tasks from a Week to Two Days with AI: A Guide to the Rosgosstrakh Case

What You Will Need

• Access to a large language model (LLM — a model trained to predict the next token; for example, via an API or web interface).
• A table with tasks: at minimum the columns "ID", "Title", "Description", "Evaluation Criteria" (for example, impact on compliance, complexity, urgency).
• A Python script for batch processing (installed via pip install openai pandas).
• An API key from the chosen LLM provider. Store it in an environment variable, not in the code.
• Examples of tasks with scores already assigned (at least 10–20) to check quality.

Step 1. Collect the Tasks into a Single Table

Export all tasks from the tracker (Jira, YouTrack, etc.) to CSV. Make sure each task has a text description and at least one numerical criterion. In the RGS case, the first cycle had about 500 tasks. Save the file as tasks.csv.

Example structure:

id,title,description,compliance_score
1,Update the password policy,Passwords must be changed every 90 days,3
2,Add access logging,Log all system logins,5

Step 2. Set Up Access to the LLM

Install the library and set the key. Get the key from the provider's personal account (for example, OpenAI, YandexGPT, GigaChat). The example uses an OpenAI-compatible API.

pip install openai pandas

import os
os.environ["OPENAI_API_KEY"] = "YOUR_KEY"  # replace with the real key

Step 3. Write a Prompt for Evaluating Tasks

The prompt (the request to the model) must contain the criteria and the response format. The clearer the criteria, the more stable the result. In the RGS case, tasks were evaluated against several criteria, including IT compliance.

PROMPT_TEMPLATE = """
You are an IT compliance expert. Rate the task on a scale from 1 to 5, where 5 is the maximum impact on legal compliance.
Reply with only a number.

Task: {title}
Description: {description}
"""

Step 4. Run Batch Processing

Read the CSV, apply the prompt to each task, and save the scores to a new file. Use a loop with a delay to avoid exceeding API limits.

import pandas as pd
from openai import OpenAI
import time

client = OpenAI()
df = pd.read_csv("tasks.csv")

scores = []
for _, row in df.iterrows():
    prompt = PROMPT_TEMPLATE.format(title=row["title"], description=row["description"])
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    score = response.choices[0].message.content.strip()
    scores.append(score)
    time.sleep(0.5)  # pause between requests

df["ai_score"] = scores
df.to_csv("tasks_scored.csv", index=False)

Step 5. Check Quality on a Control Sample

Take 10–20 tasks for which you already have an expert score. Compare them with the model's answers. If the discrepancy is greater than 1 point in 30% of cases, revise the prompt: add examples (few-shot) or clarify the criteria.

# Example check: calculate mean absolute error
control = df[df["expert_score"].notna()]
mae = (control["ai_score"].astype(float) - control["expert_score"]).abs().mean()
print(f"Mean error: {mae:.2f}")

Step 6. Sort the Tasks by Score

Sort the final table by descending AI score. Tasks with a high score require attention first. In the RGS case, this reduced review time from five working days to two.

df_sorted = df.sort_values("ai_score", ascending=False)
df_sorted.to_csv("tasks_priority.csv", index=False)

Step 7. Integrate into the Process

Show the results to the team. Explain that the model is an auxiliary tool and the final decision is made by an expert. Start with a pilot in one area, as at RGS (two "trains"). After a successful cycle, expand to other teams.

Typical Mistakes and How to Recognize Them

• The model returns non-numeric answers. Solution: add the instruction "Reply with only a number from 1 to 5" to the prompt and validate with a regular expression.
• Scores diverge greatly from expert scores. Cause: vague criteria. Refine the wording, add examples of correct answers to the prompt.
• Exceeding API limits. Symptom: 429 errors. Increase the pause between requests or use batch mode if the provider supports it.
• Key leakage. Never commit the key to the repository. Use environment variables or CI secrets.
• Too many tasks for one request. Split into batches of 50–100.

Bottom Line

The approach requires no programming skills beyond basic Python and no development budget. The main thing is a high-quality prompt and validation on a control sample. In the RGS case, review time was reduced from a week to two days, but the exact savings figures are not disclosed.