1c-ai-codegen-core

<div align="center"> # Tool List **Open-source benchmark LLM for code generation 1C:Enterprise (BSL)** Evaluation of Claude, GPT, Gemini · SMOP methodology · Metadata context via MCP · Docker [](LICENSE) [](https://python.org) [](https://github.com/4dand/1c-ai-codegen-research-paper) [](https://github.com/4dand/genlab-1c-web) [](Dockerfile) </div> --- ## About the Project Open-source computational core of the **GenLab-1C** platform — a benchmark for evaluating large language models (LLM) in generating code in the built-in language of 1C:Enterprise 8 (BSL). Developed as part of the R&D-2026: > *"Experimental evaluation of the effectiveness of artificial intelligence in generating code for domain-specific platforms (on the example of 1C:Enterprise 8)"* Supports models **Claude Opus 4.5**, **GPT-5.2 Codex**, **Gemini 3 Flash** through a unified API gateway OpenRouter. Implements a full pipeline: batch code generation → expert evaluation according to SMOP methodology → statistical analysis → visualization and reports (HTML / LaTeX / JSON). **Related scientific work repository:** [4dand/1c-ai-codegen-research-paper](https://github.com/4dand/1c-ai-codegen-research-paper) **Web platform for collective evaluation:** [4dand/genlab-1c-web](https://github.com/4dand/genlab-1c-web) ### Why a custom tool? Existing AI assistants (GitHub Copilot, Cursor IDE, 1C:Assistant) do not meet the requirements of a scientific experiment: they are "black boxes" without access to generation parameters, without an API for batch processing, and without mechanisms for formalized evaluation. GenLab-1C Core is developed specifically for research and provides: - **Reproducibility** — management of `temperature` and `seed`, hashing responses, fixing the full configuration - **Transparency** — open source code under MIT, no black boxes - **Scalability** — batch launch of hundreds of generations without manual intervention - **Formalized evaluation** — built-in SMOP methodology with TUI interface for experts - **Statistics** — t-intervals, Cohen's kappa, Pearson correlation, graphs for publications --- ## Quick Start ### Installation ```bash git clone https://github.com/4dand/genlab-1c-core.git cd genlab-1c-core pip install -r requirements.txt ``` Or as a Python package (for using `src` from other projects): ```bash pip install -e ".[dev]" ``` ### Configuration ```bash cp .env.example .env # Open .env and insert your OpenRouter key ``` ```env OPENROUTER_API_KEY=sk-or-v1-your-key ``` > Get the key: [openrouter.ai/keys](https://openrouter.ai/keys) ### First Experiment ```bash # Run task A1 with Gemini model python main.py run -c A -m gemini -t A1 # Run all tasks of category A for all models python main.py run -c A --all-models # Run category B using MCP server python main.py run -c B -m claude --no-mock ``` After execution: ```bash # Open expert evaluation interface python main.py evaluate experiment_A_YYYYMMDD_HHMMSS # Build report (HTML + LaTeX + JSON) python main.py report experiment_A_YYYYMMDD_HHMMSS # Generate charts separately python main.py charts experiment_A_YYYYMMDD_HHMMSS ``` --- ## Running in Docker (standalone mode) The core can be run as a standalone solution in Docker — without installing Python and dependencies. ### Building the image ```bash docker build -t genlab-1c-core . ``` ### Running via Docker Compose (recommended) ```bash # Create .env with API key echo "OPENROUTER_API_KEY=sk-or-v1-your-key" > .env # Run experiment docker compose run genlab run -c A -m gemini -t A1 # All models, category A docker compose run genlab run -c A --all-models # Information about models and balance docker compose run genlab info --models docker compose run genlab info --balance # Expert evaluation docker compose run genlab evaluate experiment_A_YYYYMMDD_HHMMSS # Generating reports and charts docker compose run genlab report experiment_A_YYYYMMDD_HHMMSS docker compose run genlab charts experiment_A_YYYYMMDD_HHMMSS ``` All results, reports, and logs are saved on the host via Docker volumes — data does not disappear when the container is restarted. ### Running without Compose ```bash docker run --rm --env-file .env \ -v ./raw_results:/app/raw_results \ -v ./reports:/app/reports \ -v ./evaluations:/app/evaluations \ -v ./code_outputs:/app/code_outputs \ -v ./logs:/app/logs \ genlab-1c-core run -c A -m gemini -t A1 ``` ### Category B (MCP) in Docker For tasks of category B, the MCP server must be available from the container. Replace `localhost` with `host.docker.internal` in `configs/settings.yaml`: ```yaml mcp: url: "http://host.docker.internal:8000" ``` ```bash # Run MCP server on the host docker run -d -p 8000:8000 vladimirkharin/1c_mcp # Run benchmark from the container docker compose run genlab run -c B -m claude --no-mock ``` --- ## Configuration Setup The entire experiment is controlled by YAML files in the `configs/` directory. No need to change the code — only configs. ### `configs/settings.yaml` — global paths and parameters ```yaml openrouter: base_url: "https://openrouter.ai/api/v1" timeout: 120 mcp: url: "http://localhost:8000" # URL of MCP server (vladimir-kharin/1c_mcp) timeout: 30 paths: configs_dir: "configs" raw_results_dir: "raw_results" code_outputs_dir: "code_outputs" evaluations_dir: "evaluations" reports_dir: "reports" ``` ### `configs/models.yaml` — adding models Each model is defined by a key (used in CLI via `-m <key>`): ```yaml models: gemini: id: "google/gemini-3-flash-preview" name: "Gemini 3 Flash" generation: temperature: 0.0 seed: 42 meta: price_input: 0.075 # $ / 1M tokens price_output: 0.30 ``` ### `configs/tasks_category_A.yaml` — adding tasks ```yaml category: name: "A" description: "Algorithmic tasks" requires_mcp: false system_prompt: | You are an expert in developing on 1C:Enterprise 8.3. Generate only code in the built-in 1C language. tasks: - id: "A1" name: "Bubble sort" difficulty: "easy" prompt: "Implement a function for sorting an array using bubble sort..." ``` ### Connecting MCP server (`vladimir-kharin/1c_mcp`) MCP server provides access to metadata of a real 1C configuration and is necessary for tasks of **category B**. **1. Run the server via Docker:** ```bash docker pull vladimirkharin/1c_mcp docker run -d -p 8000:8000 vladimirkharin/1c_mcp ``` Details about tools and supported configurations: [vladimir-kharin/1c_mcp](https://github.com/vladimir-kharin/1c_mcp) **2. Ensure the URL matches `configs/settings.yaml`:** ```yaml mcp: url: "http://localhost:8000" ``` **3. Run the experiment without mock:** ```bash python main.py run -c B -m gemini --no-mock ``` Without the `--no-mock` flag, the framework uses built-in mock data of UT 11.5 (`tests/mocks/mcp_mock.py`) — the server is not needed. --- ## Methodology: SMOP The quality of generated code is evaluated by experts according to four criteria: | Criterion | Description | What is evaluated | |----------|-------------|-----------------| | **S** | Syntax | Syntactic correctness of BSL code | | **M** | Meaning | Semantic correctness, correspondence of logic to the task | | **O** | Optimization | Compliance with 1C development standards, efficiency | | **P** | Platform | Correctness of work with metadata and platform objects | Each criterion is evaluated on a scale: **{0, 2, 4, 6, 8, 10}** Integral quality indicator: $$Q = \frac{S + M + O + P}{4}$$ | Level | Q | Interpretation | |---------|---|---------------| | High | ≥ 7.0 | Code is ready for use | | Acceptable | ≥ 4.0 | Requires refinement | | Low | < 4.0 | Unacceptable quality | Detailed descriptions of criteria for each level of the scale: [`configs/smop_criteria.yaml`](configs/smop_criteria.yaml) --- ## Task Categories ### Category A — Algorithmic Tasks Tasks on algorithmic logic in 1C syntax without dependence on metadata of a specific configuration. Allow evaluating the basic competence of the model in BSL language. Examples: bubble sort, parsing a string of full name, counting working days, working with JSON, grouping table values. ### Category B — Platform Tasks Tasks that require knowledge of metadata objects of a specific configuration (UT 11.5). For their execution, the model uses **Agentic Context Loader** — an agent that dynamically explores the structure of the configuration and collects the necessary context before generation. Examples: obtaining product balances, filling prices in the goods table, writing off according to FIFO, calculating cost, recalculating currency amounts. # Running with a Real MCP Server (Docker) docker run -d -p 8000:8000 vladimir-kharin/1c_mcp python main.py run -c B --all-models --no-mock # Running with Mock Data (UT 11.5, without Docker) python main.py run -c B --all-models ## Supported Models | Model | OpenRouter ID | Determinism | Context Window | |--------|--------------|-------------|----------------| | Claude Opus 4.5 | `anthropic/claude-opus-4.5` | `temperature=0` | 200K | | GPT-5.2 Codex | `openai/gpt-5.2-codex` | `seed` | 400K | | Gemini 3 Flash | `google/gemini-3-flash-preview` | `seed` | 1M | Add a new model: [`configs/models.yaml`](configs/models.yaml) ## Repository Structure ``` genlab-1c-core/ │ ├── configs/ # Experiment Configuration │ ├── models.yaml # LLM Model Parameters │ ├── tasks_category_A.yaml # Category A Tasks (algorithmic) │ ├── tasks_category_B.yaml # Category B Tasks (platform-specific) │ ├── smop_criteria.yaml # SMOP Criteria with Scale Descriptions │ ├── experiment.yaml # Experiment Metadata │ └── settings.yaml # Global Settings │ ├── src/ │ ├── clients/ # HTTP Clients for External Services │ │ ├── openrouter.py # OpenRouter API Client │ │ └── mcp.py # MCP Server Client (Streamable HTTP) │ ├── core/ # Computational Core │ │ ├── benchmark.py # BenchmarkRunner — Experiment Orchestrator │ │ └── context_loader.py # AgenticContextLoader — Context Collection Agent │ ├── evaluator/ # Evaluation and Analysis Module │ │ ├── smop.py # SMOPEvaluator, SMOPCriteria │ │ ├── statistics.py # Statistics, DI, Cohen's Kappa │ │ ├── charts.py # Charts (matplotlib, publication-ready) │ │ ├── report.py # HTML/JSON/LaTeX Report Generation │ │ ├── dashboard.py # TUI Interface for Expert Evaluation │ │ ├── parser.py # Parsing raw_results for Evaluation │ │ └── schemas.py # Pydantic Evaluation Schemas │ ├── schemas/ # Common Data Schemas │ ├── config/ # Configuration Loading and Validation │ ├── cli/ # CLI Command Handlers │ └── utils/ # Utilities (hashing, export, files) │ ├── tests/ │ └── mocks/ │ └── mcp_mock.py # Mock MCP Client with UT 11.5 Data │ ├── docs/ │ └── evaluator.md # Evaluation Module Documentation │ ├── main.py # CLI Entry Point ├── Dockerfile # Docker Image for Standalone Deployment ├── docker-compose.yml # Docker Compose with Volumes ├── requirements.txt └── .env.example ``` ## Command Reference ```bash # Information python main.py info --balance # OpenRouter API Balance python main.py info --models # List of Available Models python main.py info --tasks A # Category A Tasks with Descriptions python main.py info --tasks B # Category B Tasks with Descriptions # Running Experiments python main.py run -c A -m gemini # One Model, Category A python main.py run -c A -m claude -t A1 A3 # Specific Tasks python main.py run -c A --all-models # All Models python main.py run -c B -m gemini # Category B (mock MCP) python main.py run -c B -m gemini --no-mock # Category B (real MCP) # Expert Evaluation (TUI) python main.py evaluate --list # List of Available Experiments python main.py evaluate experiment_A_YYYYMMDD # Open TUI Interface for Evaluation # Reports and Analytics python main.py report experiment_A_YYYYMMDD # HTML + JSON + LaTeX python main.py stats experiment_A_YYYYMMDD # Statistics Output to Terminal python main.py charts experiment_A_YYYYMMDD # Charts Only ``` ## Output Artifacts ``` raw_results/ └── experiment_A_YYYYMMDD_HHMMSS.json # Raw Results with Hashes and Metrics code_outputs/ └── experiment_A_YYYYMMDD_HHMMSS/ └── A1_Сортировка/ ├── claude_run_1.bsl # Generated Code by Runs └── claude_run_2.bsl evaluations/ └── experiment_A_YYYYMMDD_expert_01.json # Expert Evaluations SMOP reports/ ├── experiment_A_YYYYMMDD_report.html # Interactive HTML Report ├── experiment_A_YYYYMMDD_report.json # Structured Metrics ├── experiment_A_YYYYMMDD_tables.tex # LaTeX Tables for Publication └── charts/ └── experiment_A_YYYYMMDD/ ├── smop_radar.png # SMOP Radar Charts ├── models_comparison.png # Model Comparison with CI ├── heatmap_tasks_models.png # Heatmap Task × Model ├── scores_distribution.png # Score Distribution ├── boxplot_by_model.png # Boxplot by Models └── determinism_vs_quality.png # Correlation Determinism / Quality ``` ## Interfaces ### CLI — Experiment Launch  *Main CLI Interface: run, info, evaluate, report, stats, charts commands* ### TUI — Expert SMOP Evaluation  *Terminal interface for assigning S, M, O, P scores for each run. Navigation through tasks, code highlighting, autosave, progress indication.* ## Documentation | File | Description | |------|----------| | [docs/evaluator.md](docs/evaluator.md) | Evaluation Module: SMOP, Statistics Formulas (CI, Cohen's Kappa, Pearson Correlation), API Reference, Code Examples | ## Contributing The project is focused on the research community. Ways to participate: - **New Tasks** — add to `configs/tasks_category_*.yaml` - **New Models** — add to `configs/models.yaml` - **New Categories** — create `tasks_category_C.yaml` and set `requires_mcp` - **Improving Statistics** — `src/evaluator/statistics.py` - **Bugs and Suggestions** — [open issue](https://github.com/4dand/genlab-1c-core/issues) ```bash git clone https://github.com/4dand/genlab-1c-core.git cd genlab-1c-core pip install -r requirements.txt cp .env.example .env # add OPENROUTER_API_KEY python main.py run -c A -m gemini -t A1 ``` ## Scientific Context This repository is part of the publication complex of R&D-2026: | Artifact | Link | |----------|--------| | Full paper, experiment data, evaluation results | [4dand/1c-ai-codegen-research-paper](https://github.com/4dand/1c-ai-codegen-research-paper) | | Core source code (this repository) | [4dand/genlab-1c-core](https://github.com/4dand/genlab-1c-core) | | Web platform for collective expert evaluation | [4dand/genlab-1c-web](https://github.com/4dand/genlab-1c-web) | | MCP server for accessing 1C:Enterprise metadata | [vladimir-kharin/1c_mcp](https://github.com/vladimir-kharin/1c_mcp) | ## English Summary **Benchmarking LLM Code Generation for 1C:Enterprise 8 (BSL Language)** GenLab-1C Core is an open-source benchmarking framework for evaluating large language models on code generation tasks targeting the **1C:Enterprise platform** — a domain-specific ERP system widely used in Russia and CIS countries with its own programming language (BSL / Built-in Scripting Language). **Key features:** - Structured benchmark pipeline: prompt → LLM generation → expert evaluation → statistical analysis → publication-ready reports - **SMOP evaluation methodology** — 4-criteria scoring system (Syntax, Meaning, Optimization, Platform) designed for domain-specific languages tied to configuration metadata - Two task categories: algorithmic (Category A) and platform-specific with MCP metadata context (Category B) - Models tested: Claude Opus 4.5, GPT-5.2 Codex, Gemini 3 Flash (via OpenRouter) - Docker-ready for standalone deployment - TUI interface for expert scoring, LaTeX/HTML/JSON report generation, publication-quality charts **Key findings** (from the [research paper](https://github.com/4dand/1c-ai-codegen-research-paper)): - Category A (algorithmic tasks): mean Q = 8.8/10 — all models perform well - Category B (platform-specific tasks with MCP context): mean Q = 6.8/10 — significant quality drop - Claude Opus 4.5 achieves the best consistency (Q = 8.4 in Category B, σ = 0.89) - Providing metadata context via MCP does not fully close the quality gap > **GitHub Topics (recommended for this repo):** `benchmark`, `llm`, `code-generation`, `1c-enterprise`, `bsl`, `ai`, `smop`, `mcp`, `model-context-protocol`, `openrouter`, `claude`, `gpt`, `gemini`, `research`, `docker`, `domain-specific-language` ## License [MIT](LICENSE) — free use, modification, and distribution with proper attribution. <div align="center"> Andreev Danila · Galtcova Yulia · R&D-2026 </div>